Remembering "what" brings along "where" in visual working memory

Predictable object motion is extrapolated to support visual working memory for surface features

Space and time support visual working memory (VWM) by providing incidental reference frames for task-relevant information. While this has been studied with stationary items, natural scenes also contain moving objects, whose positions change over time, often in a predictable manner. We investigated if predictable item motion is leveraged to facilitate VWM for surface features. In a dynamic change-detection task, participants memorised the colours of three disks moving at constant speed in different directions before disappearing. After a retention interval, the disks reappeared (a) at the movement endpoint locations where they had disappeared (b) at positions spatiotemporally congruent with their previous motion (where they would have been had they continued their movement), (c) at positions with a temporal offset (consistent with a change in speed while out of view) or (d) at positions with a spatial offset (consistent with a change in movement direction). Performance decreased with increasing temporal or spatial offsets relative to congruent positions, indicating that the memorised items' positions were remapped to their anticipated future locations. This updating of positions in spatiotemporal reference frames, however, only occurs if motion extrapolation allows for reliable predictions of where occluded items will reappear. In a task context with unreliable motion patterns, the congruency effect diminished over time and performance instead increased at the movement endpoints. A second experiment confirmed this influence of motion reliability. Thus, predictable motion is extrapolated to update spatiotemporal reference frames in VWM, supporting memory for surface features and thereby contributing to visually guided behavior in dynamic environments.

Binding in visual working memory is task dependent

Working memory is a neurocognitive system for maintaining and manipulating information online for a short period after the source of information disappears. The information held in working memory has been shown to flexibly match current functional goals. Considering this, we revisited the question of whether information is held in working memory as separate features or as bound objects. We conjectured that, rather than having a fixed answer, the format in which information is maintained in working memory is also task dependent. In two separate experiments, we investigated the binding between features when the location was not (Experiment 1, color and orientation binding) or was (Experiment 2, color and location binding) a task-relevant feature in a delayed (yes/no) recognition task by manipulating the relative relevance of conjunctions and separate features. Each experiment included two conditions: binding dominant (BD), which emphasized the retention of binding between features, and feature dominant (FD), which emphasized the retention of individual features. In both experiments, we found that memory for conjunctions was better in the BD condition and that memory for separate features was better in the FD condition. These patterns suggested that the formats of objects in visual working memory could be shaped by the tasks they serve. Additionally, we found that the memory of location was impaired when the conjunction between location and color was task irrelevant, whereas the memory of color and the memory of orientation were relatively independent of each other. We conclude that the representation format of objects in working memory is influenced by task requirements.

Properties of iconic and visuospatial working memory in pigeons and humans using a location change-detection procedure

Tests of visuospatial memory following short (<1 s) and medium (1 to 30 s) delays have revealed characteristically different patterns of behavior in humans. These data have been interpreted as evidence for different memory systems operating during short (iconic memory) and long delays (working memory). Leising et al. (2019, Behavioural Processes, 169, Article 103957 ) found evidence for both systems in pigeons and humans completing a location change-detection task using a visual mask that disrupted accuracy following a short (100 ms), but not a long (1,000 ms) delay. Another common finding is that adding to-be-remembered items should disrupt accuracy after a long, but not short, delay. Experiments 1a and 1b reported this memory system crossover effect in pigeons and people, respectively, tested on location change detection with delays of 0, 100, and 1,000 ms and displays of two to 16 items. Experiments 2a and 2b reported that the color of the items had little (pigeons) or no (humans) effect on change-detection accuracy. Pigeons tested in Experiment 3 with longer delays (2,000, 4,000, and 8,000 ms) and large set sizes demonstrated the crossover effect with most displays but did not demonstrate an abrupt drop in accuracy characteristic of iconic memory. In Experiment 4, accuracy with novel types of change (color, shape, and size) was better after a 0-ms delay and above-chance levels on color and shape trials. These data demonstrate the memory system crossover effect in both humans and pigeons and expand our knowledge of the properties of memory systems across species.

The interaction of internal and external attention

The internal/external framework of attention characterizes attention focused to perceptual stimuli and internal representations as highly similar processes. While much research on external attention examines how attention may be broadened or narrowed (attentional zoom lens), it is unclear if internal attention functions in a similar way. In the present study, we manipulate both internal and external attentions to be either broad or narrow. Participants first encoded either a broad or narrow working memory array containing three differently colored items. This array was maintained as they performed an Eriksen flanker task that was either distributed broadly or narrowly, followed by a memory test for a random memory item. We found that regardless of whether the flanker fell inside or outside of the internal breadth of attention, flanker congruency effects did not change. The exception to this was when internal breadth was manipulated with retrocuing, which resulted in greater congruency effects when flankers aligned with the span of internal breadth rather than falling outside of it. Overall, this study shows that internal breadth information is unlikely to alter processing of external distractors until some of the information is cued internally after encoding, suggesting limitations in the internal/external attention framework.

Object-based visual working memory: an object benefit for equidistant memory items presented within simple contours

Previous research has shown that more information can be stored in visual working memory (VWM) when multiple items belong to the same object. Here, in four experiments, we investigated the object effect on memory for spatially equidistant features by manipulating simple, task-irrelevant contours that combined these features. In Experiments 1, 3, and, 4, three grating orientations, and in Experiment 2, one color and two orientations, were presented simultaneously to be memorized. Mixture modeling was applied to estimate both the precision and the guess rates of recall errors. Overall results showed that two target features were remembered more accurately when both were part of the same object. Further analysis showed that the probability of recall increased in particular when both features were extracted from the same object. In Experiment 2, we found that the object effect was greater for features from orthogonal dimensions, but this came at the cost of lower memory precision. In Experiment 3, when we kept the locations of the features perfectly consistent over trials so that the participants could attend to these locations rather than the contour, we still found object benefits. Finally, in Experiment 4 when we manipulated the temporal order of the object and the memory features presentations, it was confirmed that the object benefit is unlikely to stem from the strategical usage of object information. These results suggested that the object benefit arises automatically, likely at an early perceptual level.

Similar mechanisms of temporary bindings for identity and location of objects in healthy ageing: an eye-tracking study with naturalistic scenes

The ability to maintain visual working memory (VWM) associations about the identity and location of objects has at times been found to decrease with age. To date, however, this age-related difficulty was mostly observed in artificial visual contexts (e.g., object arrays), and so it is unclear whether it may manifest in naturalistic contexts, and in which ways. In this eye-tracking study, 26 younger and 24 healthy older adults were asked to detect changes in a critical object situated in a photographic scene (192 in total), about its identity (the object becomes a different object but maintains the same position), location (the object only changes position) or both (the object changes in location and identity). Aging was associated with a lower change detection performance. A change in identity was harder to detect than a location change, and performance was best when both features changed, especially in younger adults. Eye movements displayed minor differences between age groups (e.g., shorter saccades in older adults) but were similarly modulated by the type of change. Latencies to the first fixation were longer and the amplitude of incoming saccades was larger when the critical object changed in location. Once fixated, the target object was inspected for longer when it only changed in identity compared to location. Visually salient objects were fixated earlier, but saliency did not affect any other eye movement measures considered, nor did it interact with the type of change. Our findings suggest that even though aging results in lower performance, it does not selectively disrupt temporary bindings of object identity, location, or their association in VWM, and highlight the importance of using naturalistic contexts to discriminate the cognitive processes that undergo detriment from those that are instead spared by aging.

Remembering nothing: Encoding and memory processes involved in representing empty locations

Previous research has provided rich evidence that a set of visual objects can be encoded in isolation along with their exact coordinate positions as well as a global configuration that provides a network of interrelated spatial information. However, much less data is available on how unoccupied locations are encoded and maintained in memory. We tested this ability in adults using a novel paradigm that involved both empty and filled locations and required participants to monitor the addition or deletion of an item, which occurred 50% of the time. Crucially, a number of locations remained hidden to the participant-thus, information on the absence of an item at a location could not be inferred from the presence of items elsewhere. We used eye-tracking to measure the proportion of target looking during encoding and the amount of pupil dilation during memory retention. Participants looked significantly longer at filled compared with empty targets, and target looking during encoding only predicted accuracy in case of filled targets. Increased pupil dilation was observed in response to an increasing number of items, while pupil diameter was unaffected by the number of empty locations. In addition, participants made significantly more errors in the conditions that involved the representation of an empty location. Our findings support the view that human adults encode exact coordinates of items in memory. In contrast, we suggest that empty locations are represented as a property of the global configuration of items and empty space, and not as independent units of information.

Aging and binding in short-term memory: processes involved in conjunctive and relational binding

In visual short-term binding memory tasks, some studies suggested that aging disrupts relational binding more than conjunctive binding, whereas others report equivalent age-related differences in both types of binding. Yet, demands in controlled resources are potentially the greatest for relational short-term binding. In order to test the hypothesis that aging would affect preferentially tasks demanding in controlled processes, we assessed the contribution of controlled and automatic memory processes to relational and conjunctive short-term binding. Groups of young and older adults studied shape-colour (Exp.1 and 3) or object-colour (Exp.2) pairs in a relational condition in which items were linked to colour patches and a conjunctive condition where colour was integrated into the items. Memory for bindings was tested with a reconstruction task (Exp. 1 and 2) or with a recognition memory task (Exp. 3) under inclusion and exclusion instructions (Process Dissociation Procedure). The three experiments showed that the retrieval of both relational and conjunctive bindings relied primarily on controlled memory processes, the use of which was diminished in older participants. This study brings additional evidence that age-related differences in top-down control processes explain at least partly decreased short-term binding capacities.

Incidental encoding of visual information in temporal reference frames in working memory

Visual events are structured in space and time, yet models of visual working memory (VWM) have largely relied on tasks emphasizing spatial aspects. Here, we show that temporal properties of visual events are incidentally encoded along with spatial properties. In five experiments, participants performed change-detection tasks, in which items had unique spatial and temporal coordinates at encoding. Crucially, neither space nor time was task-relevant. The key manipulation concerned the retrieval context: The test array was identical to the memory array either in its entire spatiotemporal structure, or only its spatial or temporal structure. Removing spatial or temporal information at retrieval resulted in costs, indicating that memory relied on both spatial and temporal context in which items were initially perceived. Encoding of spatiotemporal structure occurred incidentally, not strategically, as it was robust even when the retrieval context was perfectly predictable. However, spatial and temporal inter-item spacings influenced the weighting of spatial and temporal information: It favoured the domain in which items were more widely spaced, facilitating their individuation and, likely, access to representations. Across individuals, the weighting of spatial and temporal information varied substantially, but it remained consistent across sessions, suggesting stable preferences for coding in the spatial or temporal domain. No comparable incidental encoding occurred for other task-irrelevant feature dimensions (size or colour). We propose that temporal structure serves as fundamental a function in VWM as spatial structure, scaffolding events that unfold over time.

The Role of Location-Context Binding in Nonspatial Visual Working Memory

Successful retrieval of an item from visual working memory (VWM) often requires an associated representation of the trial-unique context in which that item was presented. In experiment 1, fMRI of 16 male and female humans replicated a previous dissociation of the effects of manipulating memory load in comparison to the effects of manipulating context binding, by comparing VWM for one oriented line versus for three lines individuated by their location versus for three “heterogeneous” items drawn from different categories (orientation, color, and luminance): delay-period fMRI signal in frontal cortex and intraparietal sulcus (IPS) was sensitive to stimulus homogeneity rather than to memory load per se. Additionally, inspection of behavioral performance revealed a broad range of individual differences in the probability of responses to nontargets (also known as “swap errors”), and a post hoc comparison of high swap-error versus low swap-error groups generated several intriguing results: at recall, high swap-error subjects were seen to represent both the orientation and the location of the probed item less strongly, and with less differentiation from nonprobed items, and delay-period signal in IPS predicted behavioral and neural correlates of context binding at recall. In experiment 2, which was a preregistered replication, the 27 male and female humans were grouped into low and high swap-error groups by median split, and the results were broadly consistent with experiment 1. These results present a neural correlate of swap errors, and suggest that delay-period activity of the IPS may be more important for the operation of context binding than for representation per se of stimulus identity.

Event-based encoding of biological motion and location in visual working memory

We make use of discrete yet meaningful events to orient ourselves to the dynamic environment. Among these events, biological motion, referring to the movements of animate entities, is one of the most biologically salient. We usually encounter biological motions of multiple human beings taking place simultaneously at distinct locations. How we encode biological motions into visual working memory (VWM) to form a coherent experience of the external world and guide our social behaviour remains unclear. This study for the first time addressed the VWM encoding mechanism of biological motions and their corresponding locations. We tested an event-based encoding hypothesis for biological motion and location: When one element of an event is required to be memorised, the irrelevant element of an event will also be extracted into VWM. We presented participants with three biological motions at different locations and required them to memorise only the biological motions or their locations while ignoring the other dimension. We examined the event-based encoding by probing a distracting effect: If the event-based encoding took place, the change of irrelevant dimension in the probe would lead to a significant distraction and impair the performance of detecting target dimension. We found significant distracting effects, which lasted for 3 s but vanished at 6 s, regardless of the target dimension (biological motions vs. locations, Experiment 1) and the exposure time of memory array (1 s vs. 3 s, Experiment 2). These results together support an event-based encoding mechanism during VWM encoding of biological motions and their corresponding locations.

How does aging influence object-location and name-location binding during a visual short-term memory task?

Objective: Age-related impairments in human visual short-term memory (VSTM) may reflect a reduced ability to retain bound object representations, viz., object form, name, spatial, and temporal location (so called 'memory sources'). Our objective is to examine how healthy aging affects VSTM in a battery of memory recognition tasks in which sequentially presented objects, locations, and names (as auditory stimuli) were learned, with one component cued at test.Methods: Thirty-six young healthy adults (18-30 years) and 36 normally aging older adults (>60 years with no underlying health and vision issues) completed five VSTM tasks: 1. Object recognition for two or four objects; 2. Spatial location recognition for two or four objects; 3. Bound object-location recognition for two or four objects; 4. Object recognition with location priming for two or four objects; 5. Bound name (auditory)-location (cross-modal) recognition for four objects.Results: Significantly lower performance for older adults was found in spatial location recognition [task 2, p = 0.03], bound object-location recognition [task 3, p = 0.001], object recognition with location priming [task 4, p = 0.02], and bound name-location recognition [task 5, p = 0.001, independent samples t-test] tasks. A significant age group-task interaction was found (p = 0.02).Conclusion: Performance for all tests except test 1 was impaired in older adults. Lower performance for older adults was most significant in VSTM tasks requiring object-location (visual only) or name-location (auditory and visual) binding. The findings are compatible with the 'memory source' model, demonstrating that age-related binding performance is influenced by spatial coding and location priming deficits.

Feature-based guidance of attention by visual working memory is applied independently of remembered object location

Visual working memory (VWM) has been implicated both in the online representation of object tokens (in the object-file framework) and in the top-down guidance of attention during visual search, implementing a feature template. It is well established that object representations in VWM are structured by location, with access to the content of VWM modulated by position consistency. In the present study, we examined whether this property generalizes to the guidance of attention. Specifically, in two experiments, we probed whether the guidance of spatial attention from features in VWM is modulated by the position of the object from which these features were encoded. Participants remembered an object with an incidental color. Items in a subsequent search array could match either the color of the remembered object, the location, or both. Robust benefits of color match (when the matching item was the target) and costs (when the matching items was a distractor) were observed. Critically, the magnitude of neither effect was influenced by spatial correspondence. The results demonstrate that features in VWM influence attentional priority maps in a manner that does not necessarily inherit the spatial structure of the object representations in which those features are maintained.

Overlapping and distinct contributions of stimulus location and of spatial context to nonspatial visual short-term memory

Stimulus location is not always informative during visual short-term memory (VSTM) for nonspatial features. Nevertheless, there is considerable evidence for the automatic encoding and retention of location information, regardless of its task relevance. To explore the functional and neural bases of the representation of spatial context in VSTM for nonspatial information, functional magnetic resonance imaging was performed while subjects performed delayed recall for the orientation of individual stimuli. Stimulus location varied across trials, and although this information was irrelevant for task performance, multivariate pattern analysis decoding of stimulus location sustained across trials, and also the decoding strength, predicted the precision of the recall of orientation. The influence of spatial context on the representation of orientation was operationalized by comparing the orientation reconstructions with multivariate inverted encoding models (IEM) trained in location context-dependent vs. -independent data. Although orientation reconstructions were robust for both location-dependent and location-independent IEMs, they were markedly stronger for the former. Furthermore, the functional relevance of location context was demonstrated by the fact that only the location-dependent neural representations of stimulus orientation predicted recall precision. NEW & NOTEWORTHY Neural representation strength of stimulus location predicts the precision of visual short-term memory (VSTM) recall of nonspatial stimulus, even when this information is task irrelevant. Neural representations of nonspatial stimuli that incorporate location context are stronger than those that do not, and only the former representations are strongly linked to behavior. The contributions to nonspatial VSTM performance of the representation of location context are at least partly distinct from those of the representation of stimulus content.

Separating memoranda in depth increases visual working memory performance

Visual working memory is the mechanism supporting the continued maintenance of information after sensory inputs are removed. Although the capacity of visual working memory is limited, memoranda that are spaced farther apart on a 2-D display are easier to remember, potentially because neural representations are more distinct within retinotopically organized areas of visual cortex during memory encoding, maintenance, or retrieval. The impact on memory of spatial separability in depth is less clear, even though depth information is essential to guiding interactions with objects in the environment. On one account, separating memoranda in depth may facilitate performance if interference between items is reduced. However, depth information must be inferred indirectly from the 2-D retinal image, and less is known about how visual cortex represents depth. Thus, an alternative possibility is that separation in depth does not attenuate between-items interference; it may even impair performance, as attention must be distributed across a larger volume of 3-D space. We tested these alternatives using a stereo display while participants remembered the colors of stimuli presented either near or far in the 2-D plane or in depth. Increasing separation in-plane and in depth both enhanced performance. Furthermore, participants who were better able to utilize stereo depth cues showed larger benefits when memoranda were separated in depth, particularly for large memory arrays. The observation that spatial separation in the inferred 3-D structure of the environment improves memory performance, as is the case in 2-D environments, suggests that separating memoranda in depth might reduce neural competition by utilizing cortically separable resources.

No evidence for binding of items to task-irrelevant backgrounds in visual working memory

When representing visual features such as color and shape in visual working memory (VWM), participants also represent the locations of those features as a spatial configuration of the locations of those features in the display. In everyday life, we encounter objects against some background, yet it is unclear whether the configural representation in memory obligatorily constitutes the entire display, including that (often task-irrelevant) background information. In three experiments, participants completed a change detection task on color and shape; the memoranda were presented in front of uniform gray backgrounds, a textured background (Exp. 1), or a background containing location placeholders (Exps. 2 and 3). When whole-display probes were presented, changes to the objects' locations or feature bindings impacted memory performance-implying that the spatial configuration of the probes influenced participants' change decisions. Furthermore, when only a single item was probed, the effect of changing its location or feature bindings was either diminished or completely extinguished, implying that single probes do not necessarily elicit the entire spatial configuration. Critically, when task-irrelevant backgrounds were also presented that may have provided a spatial configuration for the single probes, the effect of location or bindings was not moderated. These findings suggest that although the spatial configuration of a display guides VWM-based recognition, this information does not necessarily always influence the decision process during change detection.

Of "what" and "where" in a natural search task: Active object handling supports object location memory beyond the object's identity

Looking for as well as actively manipulating objects that are relevant to ongoing behavioral goals are intricate parts of natural behavior. It is, however, not clear to what degree these two forms of interaction with our visual environment differ with regard to their memory representations. In a real-world paradigm, we investigated if physically engaging with objects as part of a search task influences identity and position memory differently for task-relevant versus irrelevant objects. Participants equipped with a mobile eye tracker either searched for cued objects without object interaction (Find condition) or actively collected the objects they found (Handle condition). In the following free-recall task, identity memory was assessed, demonstrating superior memory for relevant compared to irrelevant objects, but no difference between the Handle and Find conditions. Subsequently, location memory was inferred via times to first fixation in a final object search task. Active object manipulation and task-relevance interacted in that location memory for relevant objects was superior to irrelevant ones only in the Handle condition. Including previous object recall performance as a covariate in the linear mixed-model analysis of times to first fixation allowed us to explore the interaction between remembered/forgotten object identities and the execution of location memory. Identity memory performance predicted location memory in the Find but not the Handle condition, suggesting that active object handling leads to strong spatial representations independent of object identity memory. We argue that object handling facilitates the prioritization of relevant location information, but this might come at the cost of deprioritizing irrelevant information.

The effects of changes in object location on object identity detection: A simultaneous EEG-fMRI study

Object identity and location are bound together to form a unique integration that is maintained and processed in visual working memory (VWM). Changes in task-irrelevant object location have been shown to impair the retrieval of memorial representations and the detection of object identity changes. However, the neural correlates of this cognitive process remain largely unknown. In the present study, we aim to investigate the underlying brain activation during object color change detection and the modulatory effects of changes in object location and VWM load. To this end we used simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) recordings, which can reveal the neural activity with both high temporal and high spatial resolution. Subjects responded faster and with greater accuracy in the repeated compared to the changed object location condition, when a higher VWM load was utilized. These results support the spatial congruency advantage theory and suggest that it is more pronounced with higher VWM load. Furthermore, the spatial congruency effect was associated with larger posterior N1 activity, greater activation of the right inferior frontal gyrus (IFG) and less suppression of the right supramarginal gyrus (SMG), when object location was repeated compared to when it was changed. The ERP-fMRI integrative analysis demonstrated that the object location discrimination-related N1 component is generated in the right SMG.

The conjunction of non-consciously perceived object identity and spatial position can be retained during a visual short-term memory task

Although non-consciously perceived information has previously been assumed to be short-lived (< 500 ms), recent findings show that non-consciously perceived information can be maintained for at least 15 s. Such findings can be explained as working memory without a conscious experience of the information to be retained. However, whether or not working memory can operate on non-consciously perceived information remains controversial, and little is known about the nature of such non-conscious visual short-term memory (VSTM). Here we used continuous flash suppression to render stimuli non-conscious, to investigate the properties of non-consciously perceived representations in delayed match-to-sample (DMS) tasks. In Experiment I we used variable delays (5 or 15 s) and found that performance was significantly better than chance and was unaffected by delay duration, thereby replicating previous findings. In Experiment II the DMS task required participants to combine information of spatial position and object identity on a trial-by-trial basis to successfully solve the task. We found that the conjunction of spatial position and object identity was retained, thereby verifying that non-conscious, trial-specific information can be maintained for prospective use. We conclude that our results are consistent with a working memory interpretation, but that more research is needed to verify this interpretation.

The objects of visuospatial short-term memory: Perceptual organization and change detection

We used a colour change-detection paradigm where participants were required to remember colours of six equally spaced circles. Items were superimposed on a background so as to perceptually group them within (a) an intact ring-shaped object, (b) a physically segmented but perceptually completed ring-shaped object, or (c) a corresponding background segmented into three arc-shaped objects. A nonpredictive cue at the location of one of the circles was followed by the memory items, which in turn were followed by a test display containing a probe indicating the circle to be judged same/different. Reaction times for correct responses revealed a same-object advantage; correct responses were faster to probes on the same object as the cue than to equidistant probes on a segmented object. This same-object advantage was identical for physically and perceptually completed objects, but was only evident in reaction times, and not in accuracy measures. Not only, therefore, is it important to consider object-level perceptual organization of stimulus elements when assessing the influence of a range of factors (e.g., number and complexity of elements) in visuospatial short-term memory, but a more detailed picture of the structure of information in memory may be revealed by measuring speed as well as accuracy.

The privileged role of location in visual working memory

Reports have conflicted about the possible special role of location in visual working memory (WM). One important question is: Do we maintain the locations of objects in WM even when they are irrelevant to the task at hand? Here we used a continuous response scale to study the types of reporting errors that participants make when objects are presented at the same or at different locations in space. When several objects successively shared the same location, participants exhibited a higher tendency to report features of the wrong object in memory; that is, they responded with features that belonged to objects retained in memory but not probed at retrieval. On the other hand, a similar effect was not observed when objects shared a nonspatial feature, such as color. Furthermore, the effect of location on reporting errors was present even when its manipulation was orthogonal to the task at hand. These findings are consistent with the view that binding together different nonspatial features of an object in memory might be mediated through an object’s location. Hence, spatial location may have a privileged role in WM. The relevance of these findings to conceptual models, as well as to neural accounts of visual WM, is discussed.

Visual memory performance for color depends on spatiotemporal context

Performance on visual short-term memory for features has been known to depend on stimulus complexity, spatial layout, and feature context. However, with few exceptions, memory capacity has been measured for abruptly appearing, single-instance displays. In everyday life, objects often have a spatiotemporal history as they or the observer move around. In three experiments, we investigated the effect of spatiotemporal history on explicit memory for color. Observers saw a memory display emerge from behind a wall, after which it disappeared again. The test display then emerged from either the same side as the memory display or the opposite side. In the first two experiments, memory improved for intermediate set sizes when the test display emerged in the same way as the memory display. A third experiment then showed that the benefit was tied to the original motion trajectory and not to the display object per se. The results indicate that memory for color is embedded in a richer episodic context that includes the spatiotemporal history of the display.

A multisensory perspective of working memory

Although our sensory experience is mostly multisensory in nature, research on working memory representations has focused mainly on examining the senses in isolation. Results from the multisensory processing literature make it clear that the senses interact on a more intimate manner than previously assumed. These interactions raise questions regarding the manner in which multisensory information is maintained in working memory. We discuss the current status of research on multisensory processing and the implications of these findings on our theoretical understanding of working memory. To do so, we focus on reviewing working memory research conducted from a multisensory perspective, and discuss the relation between working memory, attention, and multisensory processing in the context of the predictive coding framework. We argue that a multisensory approach to the study of working memory is indispensable to achieve a realistic understanding of how working memory processes maintain and manipulate information.

The asymmetry and temporal dynamics of incidental letter-location bindings in working memory

Verbal-spatial bindings are integral to routine cognitive operations (e.g., reading), yet the processes supporting them in working memory are little understood. Campo and colleagues [Campo, P., Poch, C., Parmentier, F. B. R., Moratti, S., Elsley, J. V., Castellanos, N., … Maestú, F. (2010). Oscillatory activity in prefrontal and posterior regions during implicit letter-location binding. Neuroimage, 49, 2807-2815] recently reported data suggesting obligatory letter-location binding when participants were directed to remember the letters in a display (of letters in locations), but no evidence for binding when instructed to remember the filled locations. The present study contrasted two explanations for this binding asymmetry. First, it may result from an obligatory dependence on "where" during the representation of "what" information, while "where" information may be held independently of its contents (the strong asymmetry hypothesis). Second, it may constitute a snapshot of a dynamic feature inhibition process that had partially completed by test: the asymmetrical inhibition hypothesis. Using Campo and colleagues' task with a variable retention interval between display and test, we presented four consonants in distinct locations and contrasted performance between "remember letters" and "remember locations" instructions. Our data supported the strong asymmetry hypothesis through demonstrating binding in the verbal task, but not in the spatial task. Critically, when present, verbal-spatial bindings were remarkably stable, enduring for at least 15 seconds.

Visual short-term memory deficits associated with GBA mutation and Parkinson's disease

Individuals with mutation in the lysosomal enzyme glucocerebrosidase (GBA) gene are at significantly high risk of developing Parkinson’s disease with cognitive deficit. We examined whether visual short-term memory impairments, long associated with patients with Parkinson’s disease, are also present in GBA-positive individuals—both with and without Parkinson’s disease. Precision of visual working memory was measured using a serial order task in which participants observed four bars, each of a different colour and orientation, presented sequentially at screen centre. Afterwards, they were asked to adjust a coloured probe bar’s orientation to match the orientation of the bar of the same colour in the sequence. An additional attentional ‘filtering’ condition tested patients’ ability to selectively encode one of the four bars while ignoring the others. A sensorimotor task using the same stimuli controlled for perceptual and motor factors. There was a significant deficit in memory precision in GBA-positive individuals—with or without Parkinson’s disease—as well as GBA-negative patients with Parkinson’s disease, compared to healthy controls. Worst recall was observed in GBA-positive cases with Parkinson’s disease. Although all groups were impaired in visual short-term memory, there was a double dissociation between sources of error associated with GBA mutation and Parkinson’s disease. The deficit observed in GBA-positive individuals, regardless of whether they had Parkinson’s disease, was explained by a systematic increase in interference from features of other items in memory: misbinding errors. In contrast, impairments in patients with Parkinson’s disease, regardless of GBA status, was explained by increased random responses. Individuals who were GBA-positive and also had Parkinson’s disease suffered from both types of error, demonstrating the worst performance. These findings provide evidence for dissociable signature deficits within the domain of visual short-term memory associated with GBA mutation and with Parkinson’s disease. Identification of the specific pattern of cognitive impairment in GBA mutation versus Parkinson’s disease is potentially important as it might help to identify individuals at risk of developing Parkinson’s disease.

Asymmetric binding in serial memory for verbal and spatial information

As the number of studies showing that items can be retained as bound representations in memory increases, researchers are beginning to investigate how the different features are bound together. In the present study, we examined the relative importances of the verbal and spatial features in serial memory for visual stimuli. Participants were asked to memorize the order of series of letters presented visually in different locations on the computer screen. The results showed that manipulating the phonological similarity of the letters affected recall of their spatial locations, but that increasing the complexity of the spatial pattern had no effect on recall of the letters. This finding was observed in both order reconstruction (Exps. 1 and 2) and probe serial recall (Exps. 3 and 4), suggesting that verbal-spatial binding in serial memory for visual information is asymmetric.

The role of spatial configuration in multiple identity tracking

Background

The simultaneous tracking and identification of multiple moving objects encountered in everyday life requires one to correctly bind identities to objects. In the present study, we investigated the role of spatial configuration made by multiple targets when observers are asked to track multiple moving objects with distinct identities.

Methodology/Principal Findings

The overall spatial configuration made by the targets was manipulated: In the constant condition, the configuration remained as a virtual convex polygon throughout the tracking, and in the collapsed condition, one of the moving targets (critical target) crossed over an edge of the virtual polygon during tracking, destroying it. Identification performance was higher when the configuration remained intact than when it collapsed (Experiments 1a, 1b, and 2). Moreover, destroying the configuration affected the allocation of dynamic attention: the critical target captured more attention than did the other targets. However, observers were worse at identifying the critical target and were more likely to confuse it with the targets that formed the virtual crossed edge (Experiments 3–5). Experiment 6 further showed that the visual system constructs an overall configuration only by using the targets (and not the distractors); identification performance was not affected by whether the distractor violated the spatial configuration.

Conclusions/Significance

In sum, these results suggest that the visual system may integrate targets (but not distractors) into a spatial configuration during multiple identity tracking, which affects the distribution of dynamic attention and the updating of identity-location binding.

The strategic retention of task-relevant objects in visual working memory

The serial and spatially extended nature of many real-world visual tasks suggests the need for control over the content of visual working memory (VWM). We examined the management of VWM in a task that required participants to prioritize individual objects for retention during scene viewing. There were 5 principal findings: (a) Strategic retention of task-relevant objects was effective and was dissociable from the current locus of visual attention; (b) strategic retention was implemented by protection from interference rather than by preferential encoding; (c) this prioritization was flexibly transferred to a new object as task demands changed; (d) no-longer-relevant items were efficiently eliminated from VWM; and (e) despite this level of control, attended and fixated objects were consolidated into VWM regardless of task relevance. These results are consistent with a model of VWM control in which each fixated object is automatically encoded into VWM, replacing a portion of the content in VWM. However, task-relevant objects can be selectively protected from replacement.

Binding in haptics: integration of "what" and "where" information in working memory for active touch

Information about the identity and the location of perceptual objects can be automatically integrated in perception and working memory (WM). Contrasting results in visual and auditory WM studies indicate that the characteristics of feature-to-location binding can vary according to the sensory modality of the input. The present study provides first evidence of binding between “what” and “where” information in WM for haptic stimuli. In an old-new recognition task, blindfolded participants were presented in their peripersonal space with sequences of three haptic stimuli varying in texture and location. They were then required to judge if a single probe stimulus was previously included in the sequence. Recall was measured both in a condition in which both texture and location were relevant for the task (Experiment 1) and in two conditions where only one feature had to be recalled (Experiment 2). Results showed that when both features were task-relevant, even if the association of location and texture was neither necessary nor required to perform the task, participants exhibited a recall advantage in conditions in which the location and the texture of the target probe was kept unaltered between encoding and recall. By contrast, when only one feature was task-relevant, the concurrent feature did not influence the recall of the target feature. We conclude that attention to feature binding is not necessary for the emergence of feature integration in haptic WM. For binding to take place, however, it is necessary to encode and maintain in memory both the identity and the location of items.

The mental wormhole: internal attention shifts without regard for distance

Attention operates perceptually on items in the environment, and internally on objects in visuospatial working memory. In the present study, we investigated whether spatial and temporal constraints affecting endogenous perceptual attention extend to internal attention. A retro-cue paradigm in which a cue is presented beyond the range of iconic memory and after stimulus encoding was used to manipulate shifts of internal attention. Participants' memories were tested for colored circles (Experiments 1, 2, 3a, 4) or for novel shapes (Experiment 3b) and their locations within an array. In these experiments, the time to shift internal attention (Experiments 1 and 3) and the eccentricity of encoded objects (Experiments 2-4) were manipulated. Our data showed that, unlike endogenous perceptual attention, internal shifts of attention are not modulated by stimulus eccentricity. Across several timing parameters and stimuli, we found that shifts of internal attention require a minimum quantal amount of time regardless of the object eccentricity at encoding. Our findings are consistent with the view that internal attention operates on objects whose spatial information is represented in relative terms. Although endogenous perceptual attention abides by the laws of space and time, internal attention can shift across spatial representations without regard for physical distance.

Selective maintenance in visual working memory does not require sustained visual attention

In four experiments, we tested whether sustained visual attention is required for the selective maintenance of objects in visual working memory (VWM). Participants performed a color change-detection task. During the retention interval, a valid cue indicated the item that would be tested. Change-detection performance was higher in the valid-cue condition than in a neutral-cue control condition. To probe the role of visual attention in the cuing effect, on half of the trials, a difficult search task was inserted after the cue, precluding sustained attention on the cued item. The addition of the search task produced no observable decrement in the magnitude of the cuing effect. In a complementary test, search efficiency was not impaired by simultaneously prioritizing an object for retention in VWM. The results demonstrate that selective maintenance in VWM can be dissociated from the locus of visual attention.

O envolvimento da atenção na codificação da informação visuoespacial integrada

O objetivo deste estudo foi investigar o envolvimento de recursos da atenção na codificação e manutenção da informação visual e espacial na memória de trabalho. Utilizou-se um paradigma de tarefas duplas em que uma tarefa primária de localização espacial foi realizada simultaneamente a uma tarefa atentiva secundária de discriminação de tons. O desempenho dos participantes (n = 20) na tarefa primária foi afetado pela presença e pela similaridade entre os tons da tarefa secundária, e também, pela instrução de priorizar uma ou outra tarefa. Os resultados indicam que recursos atentivos (do executivo central) estão envolvidos na codificação e na manutenção ativa da informação integrada na memória visuoespacial, assim como na manutenção dos objetivos das tarefas a serem realizadas simultaneamente.

Encoding location and serial order in auditory working memory: evidence for separable processes

In this study, we investigated the interactions between temporal and spatial information in auditory working memory. In two experiments, participants were presented with sequences of sounds originating from different locations in space and were then asked to recall either their position or their serial order. In Experiment 1, attention during encoding was manipulated by contrasting ‘pure’ blocks (i.e., location-only or serial-order-only trials) to ‘mixed’ blocks (i.e., different percentages of spatial and serial-order trials). In Experiment 2, ‘pure’ blocks were contrasted to blocks in which spatial and serial-order trials were intermixed with a third task requiring a semantic categorization of sounds. Results from both experiments showed that, whereas serial-order recall is linearly affected by the simultaneous encoding of a concurrent feature, the recall of position is mostly unaffected by concurrent feature encoding. Contrastingly, overall performance level was lower for spatial recall than serial recall. We concluded that serial order and location of items appear to be independently encoded in auditory working memory. Serial order is easier to recall, but strongly affected by the processing of concurrent item dimensions, while item location is more difficult to recall, but relatively automatic, as shown by its strong resistance to interfering dimensions in encoding.

A review of visual memory capacity: Beyond individual items and toward structured representations

Traditional memory research has focused on identifying separate memory systems and exploring different stages of memory processing. This approach has been valuable for establishing a taxonomy of memory systems and characterizing their function but has been less informative about the nature of stored memory representations. Recent research on visual memory has shifted toward a representation-based emphasis, focusing on the contents of memory and attempting to determine the format and structure of remembered information. The main thesis of this review will be that one cannot fully understand memory systems or memory processes without also determining the nature of memory representations. Nowhere is this connection more obvious than in research that attempts to measure the capacity of visual memory. We will review research on the capacity of visual working memory and visual long-term memory, highlighting recent work that emphasizes the contents of memory. This focus impacts not only how we estimate the capacity of the system--going beyond quantifying how many items can be remembered and moving toward structured representations--but how we model memory systems and memory processes.

Object — Position Binding in Visual Short-Term Memory for Sequentially Presented Unfamiliar Stimuli

The effect of spatial position on visual short-term memory (VSTM) for sequentially presented objects has been investigated relatively little, despite the fact that vision in natural environments is characterised by frequent changes in object position and gaze location. We investigated the effect of reusing previously examined spatial positions on VSTM for object appearance. Observers performed a yes – no recognition task following a memory display comprising briefly presented 1/ f noise discs (ie possessing spectral properties akin to natural images) shown sequentially at random coordinates. At test, single stimuli were presented either at original spatial positions, new positions, or at a fixed central position. Results, interpreted in terms of appearance and position preview effects, indicate that, where original spatial positions were reused at test, memory performance was elevated by more than 25%, despite that spatial position was task-irrelevant (in the sense that it could not be used to facilitate a correct response per se). This study generalises object – spatial-position binding theory to a sequential display scenario in which the influences of extrafoveal processing, spatial context cues, and long-term memory support were minimised, thereby eliminating the hypothesis that object priming is the principal cause of the ‘same-position advantage’ in VSTM.

Binding objects to locations: the relationship between object files and visual working memory

The relationship between object files and visual working memory (VWM) was investigated in a new paradigm combining features of traditional VWM experiments (color change detection) and object-file experiments (memory for the properties of moving objects). Object-file theory was found to account for a key component of object-position binding in VWM: With motion, color memory came to be associated with the new locations of objects. However, robust binding to the original locations was found despite clear evidence that the objects had moved. This latter binding appears to constitute a scene-based component in VWM, which codes object location relative to the abstract spatial configuration of the display and is largely insensitive to the dynamic properties of objects.

The involuntary capture of attention by novel feature pairings: a study of voice-location integration in auditory sensory memory

Past researchers of the integration of information in memory have typically required participants to attend to and/or commit to memory the stimuli conveying distinct features, rendering difficult the examination of whether the maintenance of the feature pairings can occur involuntarily. To address this issue, the integration of voice and location information in auditory sensory memory was measured using a cross-modal oddball task, in which task-irrelevant auditory deviants are known to capture attention in an involuntary fashion. Participants categorized visual digits presented shortly after to-be-ignored sounds. These sounds consisted in the same phoneme played simultaneously in both ears but in different voices (female in one ear, male in the other). On most trials, the pairing of voice to location was constant (standard sound). On rare and unpredictable trials, the voices swapped locations (deviant sound). In line with past work on attention capture by auditory novelty, the participants were significantly slower to judge the visual digits following the deviant sound, indicating the involuntary encoding of the links between voice and location in auditory memory. These results suggest that voices and locations are integrated in memory and that this binding occurs in conditions in which participants do not intend to commit any information to memory.

Visual memory for fixated regions of natural images dissociates attraction and recognition

Recognition memory for fixated regions from briefly viewed full-screen natural images is examined. Low-level image statistics reveal that observers fixated, on average (pooled across images and observers), image regions that possessed greater visual saliency than non-fixated regions, a finding that is robust across multiple fixation indices. Recognition-memory performance indicates that, of the fixation loci tested, observers were adept at recognising those with a particular profile of image statistics; visual saliency was found to be attenuated for unrecognised loci, despite that all regions were freely fixated. Furthermore, although elevated luminance was the local image statistic found to discriminate least between human and random image locations, it was the greatest predictor of recognition-memory performance, demonstrating a dissociation between image features that draw fixations and those that support visual memory. An analysis of corresponding eye movements indicates that image regions fixated via short-distance saccades enjoyed better recognition-memory performance, alluding to a focal rather than ambient mode of processing. Recognised image regions were more likely to have originated from areas evaluated (a posteriori) to have higher fixation density, a numerical metric of local interest. Surprisingly, memory for image regions fixated later in the viewing period exhibited no recency advantage, despite (typically) also being longer in duration, a finding for which a number of explanations are posited.