Evidence for the effect of depth on visual working memory

The influence of depth on object selection and manipulation in visual working memory within a 3D context

Recent studies have examined whether the internal selection mechanism functions similarly for perception and visual working memory (VWM). However, the process of how we access and manipulate object representations distributed in a 3D space remains unclear. In this study, we utilized a memory search task to investigate the effect of depth on object selection and manipulation within VWM. The memory display consisted of colored items half positioned at the near depth plane and the other half at the far plane. During memory maintenance, the participants were instructed to search for a target representation and update its color. The results showed that under object-based attention (Experiments 1, 3, and 5), the update time was faster for targets at the near plane than for those at the far plane. This effect was absent in VWM when deploying spatial attention (Experiment 2) and in visual search regardless of the type of attention deployed (Experiment 4). The differential effects of depth on spatial and object-based attention in VWM suggest that spatial attention primarily relied on 2D location information irrespective of depth, whereas object-based attention seemed to prioritize memory representations at the front plane before shifting to the back. Our findings shed light on the interaction between depth perception and the selection mechanisms within VWM in a 3D context, emphasizing the importance of ordinal, rather than metric, spatial information in guiding object-based attention in VWM.

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues - a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.

The depth of executive function: Depth information aids executive function under challenging task conditions

The present studies investigated how three core aspects of executive functioning may be influenced by the presence of depth information. Specifically, participants were assigned to one of three executive functioning tasks: working memory (i.e., a change detection task), selective attention (i.e., a visual search task), or inhibitory control (i.e., a flanker task). For all three tasks, participants completed trials where the items in the display were presented either all in one depth plane or the target item was isolated in depth. For the working memory and selective attention tasks, there was an additional condition where items were evenly distributed across two depth planes. Each task also had multiple levels of difficulty to explore if task conditions influence the effect of depth information. Results indicated that although depth information can improve both working memory and selective attention performance, this benefit is specific to the task difficulty and depth information can even hinder performance under certain circumstances. Depth information did not appear to influence inhibitory control performance. Future work is required to investigate if depth can improve inhibitory control performance, and how/what task conditions influence the benefit of depth information. Until further research is completed, researchers and designers should be cautious when implementing multidimensional (3D) displays, as it remains unclear if the performance benefits of including depth information outweigh the present costs.

Does perceptual grouping improve visuospatial working memory? Optimized processing or encoding bias

Visual working memory has been defined as a system of limited capacity that enables the maintenance and manipulation of visual information. However, some perceptual features like Gestalt grouping could improve visual working memory effectiveness. In two different experiments, we aimed to explore how the presence of elements grouped by color similarity affects the change detection performance of both, grouped and non-grouped items. We combined a change detection task with a retrocue paradigm in which a six item array had to be remembered. An always valid, variable-delay retrocue appeared in some trials during the retention interval, either after 100 ms (iconic-trace period) or 1400 ms (working memory period), signaling the location of the probe. The results indicated that similarity grouping biased the information entered into the visual working memory, improving change detection accuracy only for previously grouped probes, but hindering change detection for non-grouped probes in certain conditions (Exp. 1). However, this bottom-up automatic encoding bias was overridden when participants were explicitly instructed to ignore grouped items as they were irrelevant for the task (Exp. 2).

The Short-Term Retention of Depth

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Increasing perceptual separateness affects working memory for depth - re-allocation of attention from boundaries to the fixated center

For decades, working memory (WM) has been a heated research topic in the field of cognitive psychology. However, most studies on WM presented visual stimuli on a two-dimensional plane, rarely involving depth perception. Several previous studies have investigated how depth information is stored in WM, and found that WM for depth is even more limited in capacity and the memory performance is poor compared to visual WM. In the present study, we used a change detection task to investigate whether dissociating memory items by different visual features, thereby to increase their perceptual separateness, can improve WM performance for depth. Memory items presented at various depth planes were bound with different colors (Experiments 1 and 3) or sizes (Experiment 2). The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. The results showed a consistent pattern that although separating items with various feature values did not affect the overall memory performance, the manipulation significantly improved memory performance for the middle depth locations but impaired the performance for the boundary locations when observers fixated at the center of the whole depth volume. The memory benefits of feature separation can be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources.

Visual Search in 3D: Effects of Monoscopic and Stereoscopic Cues to Depth on the Validity of Feature Integration Theory and Perceptual Load Theory

Previous research has successfully used feature integration theory to operationalise the predictions of Perceptual Load Theory, while simultaneously testing the predictions of both models. Building on this work, we test the extent to which these models hold up in a 3D world. In two experiments, participants responded to a target stimulus within an array of shapes whose apparent depth was manipulated using a combination of monoscopic and stereoscopic cues. The search task was designed to test the predictions of (a) feature integration theory, as the target was identified by a single feature or a conjunction of features and embedded in search arrays of varying size, and (b) perceptual load theory, as the task included congruent and incongruent distractors presented alongside search tasks imposing high or low perceptual load. Findings from both experiments upheld the predictions of feature integration theory, regardless of 2D/3D condition. Longer search times in conditions with a combination of monoscopic and stereoscopic depth cues suggests that binding features into three-dimensional objects requires greater attentional effort. This additional effort should have implications for perceptual load theory, yet our findings did not uphold its predictions; the effect of incongruent distractors did not differ between conjunction search trials (conceptualised as high perceptual load) and feature search trials (low perceptual load). Individual differences in susceptibility to the effects of perceptual load were evident and likely explain the absence of load effects. Overall, our findings suggest that feature integration theory may be useful for predicting attentional performance in a 3D world.

Effect of attentional selection on working memory for depth in a retro-cueing paradigm

Recent studies have shown that the temporary storage and manipulation of depth information (working memory for depth; WMd) is largely different from that of visual information in a 2D context (visual working memory; VWM). Although there has been abundant evidence on VWM showing that cueing a memory item during retention could bias attention to its internal representation and thus improves its memory performance (a retro-cue effect), it is unknown whether such an effect differs for WMd that is nested in a 3D context compared with that in a conventional 2D context. Here, we used a change detection task to investigate the effect of attentional selection on WMd by testing several types of retro-cue. The memory array consisted of items positioned at various stereoscopic depth planes, and a cue was presented during retention. Participants needed to make judgments on whether the depth position of target (one memory item) had changed. Our study showed reliable valid retro-cue benefits but no invalid retro-cue cost, indicating that the relational information may be registered in WMd to prevent a strategical removal of the unattended item. There was also a slight improvement in memory performance for cueing depth order compared with that for cueing other feature dimensions or 2D locations. The attentional effect on memory representation in a 3D context is different from that in a 2D context, and the divergence may suggest the distinctive nature of working memory for depth.

Relation matters: relative depth order is stored in working memory for depth

Working memory is considered as a cognitive memory buffer for temporarily holding, processing, and manipulating information. Although working memory for verbal and visual information has been studied extensively in the past literature, few studies have systematically investigated how depth information is stored in working memory. Here, we show that the memory performance for detecting changes in stereoscopic depth is low when there is no change in relative depth order, and the performance is reliably better when depth order is changed. Increasing the magnitude of change only improves memory performance when depth order is kept constant. However, if depth order is changed, the performance remains high, even with a small change magnitude. Our findings suggest that relative depth order is a better indicator for working memory performance than absolute metric depth. The memory representation for individual depth is not independent, but inherently relational, revealing a fundamental organizing principle for depth information in the visual system.

Exogenous spatial attention shortens perceived depth

Although spatial attention has been found to alter the subjective appearance of visual stimuli in several perceptual dimensions, no research has explored whether exogenous spatial attention can affect depth perception, which is a fundamental dimension in perception that allows us to effectively interact with the environment. Here, we used an experimental paradigm adapted from Gobell and Carrasco (Psychological Science, 16[8], 644-651, 2005) to investigate this question. A peripheral cue preceding two line stimuli was used to direct exogenous attention to either location of the two lines. The two lines were separated by a certain relative disparity, and participants were asked to judge the perceived depth of two lines while attention was manipulated. We found that a farther stereoscopic depth at the attended location was perceived to be equally distant as a nearer depth at the unattended location. No such effect was found in a control experiment that employed a postcue paradigm, suggesting that our findings could not be attributed to response bias. Therefore, our study shows that exogenous spatial attention shortens perceived depth. The apparent change in stereoscopic depth may be regulated by a mechanism involving direct neural enhancement on those tuned to disparity, or be modulated by an attentional effect on apparent contrast. This finding shows that attention can change not only visual appearance but also the perceived spatial relation between an object and an observer.

Working memory for stereoscopic depth is limited and imprecise-evidence from a change detection task

Most studies on visual working memory (VWM) and spatial working memory (SWM) have employed visual stimuli presented at the fronto-parallel plane and few have involved depth perception. VWM is often considered as a memory buffer for temporarily holding and manipulating visual information that relates to visual features of an object, and SWM for holding and manipulating spatial information that concerns the spatial location of an object. Although previous research has investigated the effect of stereoscopic depth on VWM, the question of how depth positions are stored in working memory has not been systematically investigated, leaving gaps in the existing literature on working memory. Here, we explore working memory for depth by using a change detection task. The memory items were presented at various stereoscopic depth planes perpendicular to the line of sight, with one item per depth plane. Participants were asked to make judgments on whether the depth position of the target (one of the memory items) had changed. The results showed a conservative response bias that observers tended to make 'no change' responses when detecting changes in depth. In addition, we found that similar to VWM, the change detection accuracy degraded with the number of memory items presented, but the accuracy was much lower than that reported for VWM, suggesting that the storage for depth information is severely limited and less precise than that for visual information. The detection sensitivity was higher for the nearest and farthest depths and was better when the probe was presented along with the other items originally in the memory array, indicating that how well the to-be-stored depth can be stored in working memory depends on its relation with the other depth positions.

Saturation and brightness modulate the effect of depth on visual working memory

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.

Task-dependent effects of voluntary space-based and involuntary feature-based attention on visual working memory

Previous research has shown that visual working memory (VWM) can be modulated by space-based or feature-based attentional selection. However, it remains unclear how the two modes of attention operate jointly to affect VWM, and in particular, if involuntary feature-based attention plays a role in VWM. In this study, a pre-cued change detection paradigm was employed to investigate the concurrent effects of space- and feature-based attention on VWM. Space-based attention was manipulated by informative spatial cueing and by varying the proximity between the test item and the cued (fixated) memory item, while feature-based attention was induced in an involuntary manner by having the test item to share the same color or shape with the cued item on a fraction of trials. The results showed that: (1) the memory performance for the cued items was always better than the uncued items, suggesting a beneficial effect of voluntary spatial attention; (2) with a brief duration of the memory array (250 ms), cue-test proximity benefited VWM in the shape judgment task but not in the color judgment task, whereas with a longer duration (1200 ms), no proximity effect was found for either task; (3) VWM was improved for the same-colored items regardless of the task and duration; (4) VWM was improved for the same-shaped items only in the shape judgment task with the longer duration of the memory array. A discrimination task further showed that the proximity effect associated with VWM reflects a perceptual bottleneck in memory encoding for shape but not for color with a brief display. Our results suggest that involuntary feature-based attention could be triggered by spatial cueing to modulate VWM; involuntary color-based attention facilitates VWM independently of task, whereas shape-based facilitation is task-dependent, i.e., confined only to the shape judgment task, presumably reflecting different attention-guiding potencies of the two features.

Depth benefits now loading: Visual working memory capacity and benefits in 3-D

The present studies explored how performance in multidimensional displays varies as a function of visual working memory load, item distribution across depths, and individual capacity differences. In Experiment 1, the benefit of depth information (one depth vs. two depths) was examined across seven set sizes within a change-detection paradigm. Multiple depth planes engendered performance benefits with five items, but elicited performance decrements with three items. These effects were associated with working memory capacity, such that benefits were only observed when the working memory load exceeded an individual's max capacity. Experiment 2 evaluated how the distribution of items in depth aids working memory performance. Equal distribution of items across depths produced higher accuracy compared with when the target was isolated in depth. Lastly, Experiment 3 explored how differences in working memory capacity affect an individual's ability to use depth information to improve their performance. The results indicate that both low-capacity and high-capacity individuals can benefit from depth information, but this may vary as a function of working memory load. Overall, the results indicate that multidimensional displays can improve performance with sufficient working memory load, possibly through some sort of depth tag.

Separating memoranda in depth increases visual working memory performance

Visual working memory is the mechanism supporting the continued maintenance of information after sensory inputs are removed. Although the capacity of visual working memory is limited, memoranda that are spaced farther apart on a 2-D display are easier to remember, potentially because neural representations are more distinct within retinotopically organized areas of visual cortex during memory encoding, maintenance, or retrieval. The impact on memory of spatial separability in depth is less clear, even though depth information is essential to guiding interactions with objects in the environment. On one account, separating memoranda in depth may facilitate performance if interference between items is reduced. However, depth information must be inferred indirectly from the 2-D retinal image, and less is known about how visual cortex represents depth. Thus, an alternative possibility is that separation in depth does not attenuate between-items interference; it may even impair performance, as attention must be distributed across a larger volume of 3-D space. We tested these alternatives using a stereo display while participants remembered the colors of stimuli presented either near or far in the 2-D plane or in depth. Increasing separation in-plane and in depth both enhanced performance. Furthermore, participants who were better able to utilize stereo depth cues showed larger benefits when memoranda were separated in depth, particularly for large memory arrays. The observation that spatial separation in the inferred 3-D structure of the environment improves memory performance, as is the case in 2-D environments, suggests that separating memoranda in depth might reduce neural competition by utilizing cortically separable resources.

Evidence for the beneficial effect of perceptual grouping on visual working memory: an empirical study on illusory contour and a meta-analytic study

The capacity of visual working memory (VWM) is found to be extremely limited. Past research shows that VWM can be facilitated by Gestalt principles of grouping, however, it remains controversial whether factors like the type of Gestalt principles, the characteristics of stimuli and the nature of experimental design could affect the beneficial effect of grouping. In particular, studies have shown that perceptual grouping could improve memory performance for a feature that is relevant for grouping, but it is unclear whether the same improvement exists for a feature that is irrelevant for grouping. In this article, an empirical study and a meta-analytic study were conducted to investigate the effect of perceptual grouping on VWM. In the empirical study, we examined the grouping effect by employing a Kanizsa illusion in which memory items were grouped by illusory contour. We found that the memory performance was improved for the grouped items even though the tested feature was grouping irrelevant, and the improvement was not significantly different from the effect of grouping by physical connectedness or by solid occlusion. In the meta-analytic study, we systematically and quantitatively examined the effect of perceptual grouping on VWM by pulling the results from all eligible studies, and found that the beneficial grouping effect was robust but the magnitude of the effect can be affected by several moderators. Factors like the types of grouping methods, the duration and the layout of the memory display, and the characteristics of the tested feature moderated the grouping effect, whereas whether employing a cue or a verbal suppression task did not. Our study suggests that the underlying mechanism of the grouping benefit may be distinct with regard to grouping relevancy of the to-be-stored feature. The grouping effect on VWM may be independent of attention for a grouping relevant feature, but may rely on attentional prioritization for a grouping irrelevant feature.