Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information

Beyond Routine Maintenance: Current Trends in Working Memory Research

Working memory (WM) is an evolving concept. Our understanding of the neural functions that support WM develops iteratively alongside the approaches used to study it, and both can be profoundly shaped by available tools and prevailing theoretical paradigms. Here, the organizers of the 2024 Working Memory Symposium-inspired by this year's meeting-highlight current trends and looming questions in WM research. This review is organized into sections describing (1) ongoing efforts to characterize WM function across sensory modalities, (2) the growing appreciation that WM representations are malleable to context and future actions, (3) the enduring problem of how multiple WM items and features are structured and integrated, and (4) new insights about whether WM shares function with other cognitive processes that have conventionally been considered distinct. This review aims to chronicle where the field is headed and calls attention to issues that are paramount for future research.

When working memory works for our goals

When navigating environments with changing rules, human brain circuits flexibly adapt how and where we retain information to help us achieve our immediate goals.

Stimulus representation in human frontal cortex supports flexible control in working memory

When holding visual information temporarily in working memory (WM), the neural representation of the memorandum is distributed across various cortical regions, including visual and frontal cortices. However, the role of stimulus representation in visual and frontal cortices during WM has been controversial. Here, we tested the hypothesis that stimulus representation persists in the frontal cortex to facilitate flexible control demands in WM. During functional MRI, participants flexibly switched between simple WM maintenance of visual stimulus or more complex rule-based categorization of maintained stimulus on a trial-by-trial basis. Our results demonstrated enhanced stimulus representation in the frontal cortex that tracked demands for active WM control and enhanced stimulus representation in the visual cortex that tracked demands for precise WM maintenance. This differential frontal stimulus representation traded off with the newly-generated category representation with varying control demands. Simulation using multi-module recurrent neural networks replicated human neural patterns when stimulus information was preserved for network readout. Altogether, these findings help reconcile the long-standing debate in WM research, and provide empirical and computational evidence that flexible stimulus representation in the frontal cortex during WM serves as a potential neural coding scheme to accommodate the ever-changing environment.

Re-focusing visual working memory during expected and unexpected memory tests

A classic distinction from the domain of external attention is that between anticipatory orienting and subsequent re-orienting of attention to unexpected events. Whether and how humans also re-orient attention 'in mind' following expected and unexpected working-memory tests remains elusive. We leveraged spatial modulations in neural activity and gaze to isolate re-orienting within the spatial layout of visual working memory following central memory tests of certain, expected, or unexpected mnemonic content. Besides internal orienting after predictive cues, we unveil a second stage of internal attentional deployment following both expected and unexpected memory tests. Following expected tests, internal attentional deployment was not contingent on prior orienting, suggesting an additional verification - 'double checking' - in memory. Following unexpected tests, re-focusing of alternative memory content was prolonged. This brings attentional re-orienting to the domain of working memory and underscores how memory tests can invoke either a verification or a revision of our internal focus.

Dynamic categorization rules alter representations in human visual cortex

Everyday tasks often require stimuli to be categorized dynamically, such that an identical object can elicit different responses based on the current decision rule. Traditionally, sensory regions have been viewed as separate from such context-dependent processing, functioning primarily to process incoming inputs. However, an alternative view suggests sensory regions also integrate inputs with current task goals, facilitating more efficient information relay to higher-level areas. Here we test this by asking human participants to visually categorize novel shape stimuli based on different decision boundaries. Using fMRI and multivariate analyses of retinotopically-defined visual areas, we show that cortical shape representations become more distinct across relevant decision boundaries in a context-dependent manner, with the largest changes in discriminability observed for stimuli near the decision boundary. Importantly, these modulations are associated with improved task performance. These findings demonstrate that visual cortex representations are adaptively modulated to support dynamic behavior.

Neural dynamics of reselecting visual and motor contents in working memory after external interference

In everyday tasks, we must often shift our focus away from internal representations held in working memory to engage with perceptual events in the external world. Here, we investigated how our internal focus is reestablished following an interrupting task by tracking the reselection of visual representations and their associated action plans in working memory. Specifically, we ask whether reselection occurs for both visual and motor memory attributes and when this reselection occurs. We developed a visual-motor working-memory task in which participants were retrospectively cued to select one of two memory items before being interrupted by a perceptual discrimination task. To determine what information was reselected, the memory items had distinct visual and motor attributes. To determine when internal representations were reselected, the interrupting task was presented at one of three distinct time points following the retro-cue. We employed electroencephalography time-frequency analyses to track the initial selection and later reselection of visual and motor representations, as operationalized through modulations of posterior alpha (8-12 Hz) activity relative to the memorized item location (visual) and of central beta (13-30 Hz) activity relative to the required response hand (motor). Our results showed that internal visual and motor contents were concurrently reselected immediately after completing the interrupting task, rather than only when internal information was required for memory-guided behavior. Thus, following interruption, we swiftly resume our internal focus in working memory through the simultaneous reselection of memorized visual representations and their associated action plans, thereby restoring internal contents to a ready-to-use state.Significance statement A key challenge for working memory is to maintain past visual representations and their associated actions while engaging with the external environment. Our cognitive system must, therefore, often juggle multiple tasks within a common time frame. Despite the ubiquity of multi-task situations in everyday life, working memory has predominantly been studied devoid of additional perceptual, attentional, and response demands during the retention interval. Here, we investigate the neural dynamics of returning to internal contents following task-relevant interruptions. Particularly, we identify which attributes of internal representations are reselected and when this reselection occurs. Our findings demonstrate that both visual and motor contents are reselected immediately and in tandem after completion of an external, interrupting task.

Memory load influences our preparedness to act on visual representations in working memory without affecting their accessibility

It is well established that when we hold more content in working memory, we are slower to act upon part of that content when it becomes relevant for behavior. Here, we asked whether this load-related slowing is due to slower access to the sensory representations held in working memory (as predicted by serial working-memory search), or by a reduced preparedness to act upon those sensory representations once accessed. To address this, we designed a visual-motor working-memory task in which participants memorized the orientation of two or four colored bars, of which one was cued for reproduction. We independently tracked EEG markers associated with the selection of visual (cued item location) and motor (relevant manual action) information from the EEG time-frequency signal, and compared their latencies as a function of memory load. We confirm slower memory-guided behavior with higher working-memory load and show that this is associated with delayed motor selection. In contrast, we find no evidence for a concomitant delay in the latency of visual selection. Moreover, we show that variability in decision times within each memory-load condition is associated with corresponding changes in the latency of motor, but not visual selection. These results reveal how memory load affects our preparedness to act on sensory representations in working memory, while leaving sensory access itself unaffected. This posits action readiness as a key factor that shapes the speed of memory-guided behavior and that underlies delayed responding with higher working-memory load.

Coordinated social interactions are supported by integrated neural representations

Joint actions are defined as coordinated interactions of two or more agents toward a shared goal, often requiring different and complementary individual contributions. However, how humans can successfully act together without the interfering effects of observing incongruent movements is still largely unknown. It has been proposed that interpersonal predictive processes are at play to allow the formation of a Dyadic Motor Plan, encompassing both agents' shares. Yet, direct empirical support for such an integrated motor plan is still limited. In this study, we aimed at testing the properties of these anticipated representations. We collected electroencephalography data while human participants (N = 36; 27 females) drew shapes simultaneously to a virtual partner, in two social contexts: either they had to synchronize and act jointly or they performed the movements alongside, but independently. We adopted a multivariate approach to show that the social context influenced how the upcoming action of the partner is anticipated during the interval preceding the movement. We found evidence that acting jointly induces an encoding of the partner's action that is strongly intertwined with the participant's action, supporting the hypothesis of an integrative motor plan in joint but not in parallel actions.

The Hippocampus Preorders Movements for Skilled Action Sequences

Plasticity in the subcortical motor basal ganglia-thalamo-cerebellar network plays a key role in the acquisition and control of long-term memory for new procedural skills, from the formation of population trajectories controlling trained motor skills in the striatum to the adaptation of sensorimotor maps in the cerebellum. However, recent findings demonstrate the involvement of a wider cortical and subcortical brain network in the consolidation and control of well-trained actions, including a brain region traditionally associated with declarative memory-the hippocampus. Here, we probe which role these subcortical areas play in skilled motor sequence control, from sequence feature selection during planning to their integration during sequence execution. An fMRI dataset (N = 24; 14 females) collected after participants learnt to produce four finger press sequences entirely from memory with high movement and timing accuracy over several days was examined for both changes in BOLD activity and their informational content in subcortical regions of interest. Although there was a widespread activity increase in effector-related striatal, thalamic, and cerebellar regions, in particular during sequence execution, the associated activity did not contain information on the motor sequence identity. In contrast, hippocampal activity increased during planning and predicted the order of the upcoming sequence of movements. Our findings suggest that the hippocampus preorders movements for skilled action sequences, thus contributing to the higher-order control of skilled movements that require flexible retrieval. These findings challenge the traditional taxonomy of episodic and procedural memory and carry implications for the rehabilitation of individuals with neurodegenerative disorders.

A gradual transition toward categorical representations along the visual hierarchy during working memory, but not perception

The ability to stably maintain visual information over brief delays is central to healthy cognitive functioning, as is the ability to differentiate such internal representations from external inputs. One possible way to achieve both is via multiple concurrent mnemonic representations along the visual hierarchy that differ systematically from the representations of perceptual inputs. To test this possibility, we examine orientation representations along the visual hierarchy during perception and working memory. Human participants directly viewed, or held in mind, oriented grating patterns, and the similarity between fMRI activation patterns for different orientations was calculated throughout retinotopic cortex. During direct viewing of grating stimuli, similarity was relatively evenly distributed amongst all orientations, while during working memory the similarity was higher around oblique orientations. We modeled these differences in representational geometry based on the known distribution of orientation information in the natural world: The "veridical" model uses an efficient coding framework to capture hypothesized representations during visual perception. The "categorical" model assumes that different "psychological distances" between orientations result in orientation categorization relative to cardinal axes. During direct perception, the veridical model explained the data well. During working memory, the categorical model gradually gained explanatory power over the veridical model for increasingly anterior retinotopic regions. Thus, directly viewed images are represented veridically, but once visual information is no longer tethered to the sensory world there is a gradual progression to more categorical mnemonic formats along the visual hierarchy.

Action Planning Renders Objects in Working Memory More Attentionally Salient

A rapidly growing body of work suggests that visual working memory (VWM) is fundamentally action oriented. Consistent with this, we recently showed that attention is more strongly biased by VWM representations of objects when we plan to act on those objects in the future. Using EEG and eye tracking, here, we investigated neurophysiological correlates of the interactions between VWM and action. Participants (n = 36) memorized a shape for a subsequent VWM test. At test, a probe was presented along with a secondary object. In the action condition, participants gripped the actual probe if it matched the memorized shape, whereas in the control condition, they gripped the secondary object. Crucially, during the VWM delay, participants engaged in a visual selection task, in which they located a target as fast as possible. The memorized shape could either encircle the target (congruent trials) or a distractor (incongruent trials). Replicating previous findings, we found that eye gaze was biased toward the VWM-matching shape and, importantly, more so when the shape was directly associated with an action plan. Moreover, the ERP results revealed that during the selection task, future action-relevant VWM-matching shapes elicited (1) a stronger Ppc (posterior positivity contralateral), signaling greater attentional saliency; (2) an earlier PD (distractor positivity) component, suggesting faster suppression; (3) a larger inverse (i.e., positive) sustained posterior contralateral negativity in incongruent trials, consistent with stronger suppression of action-associated distractors; and (4) an enhanced response-locked positivity over left motor regions, possibly indicating enhanced inhibition of the response associated with the memorized item during the interim task. Overall, these results suggest that action planning renders objects in VWM more attentionally salient, supporting the notion of selection-for-action in working memory.

Comparing Neural Correlates of Memory Encoding and Maintenance for Foveal and Peripheral Stimuli

Visual working memory is believed to rely on top-down attentional mechanisms that sustain active sensory representations in early visual cortex, a mechanism referred to as sensory recruitment. However, both bottom-up sensory input and top-down attentional modulations thereof appear to prioritize the fovea over the periphery, such that initially peripheral percepts may even be assimilated by foveal processes. This raises the question whether and how visual working memory differs for central and peripheral input. To address this, we conducted a delayed orientation recall task in which an orientation was presented either at the center of the screen or at 15° eccentricity to the left or right. Response accuracy, EEG activity, and gaze position were recorded from 30 participants. Accuracy was slightly but significantly higher for foveal versus peripheral memories. Decoding of EEG recordings revealed a clear dissociation between early sensory and later maintenance signals. Although sensory signals were clearly decodable for foveal stimuli, they were not for peripheral input. In contrast, maintenance signals were equally decodable for both foveal and peripheral memories, suggesting comparable top-down components regardless of eccentricity. Moreover, although memory representations were initially spatially specific and reflected in voltage fluctuations, later during the maintenance period, they generalized across locations, as emerged in alpha oscillations, thus revealing a dynamic transformation within memory from separate sensory traces to what we propose are common output-related codes. Furthermore, the combined absence of reliable decoding of sensory signals and robust presence of maintenance decoding indicates that storage activity patterns as measured by EEG reflect signals beyond primary visual cortex. We discuss the implications for the sensory recruitment hypothesis.

EEG decoding reveals task-dependent recoding of sensory information in working memory

Working memory (WM) supports future behavior by retaining perceptual information obtained in the recent past. The present study tested the hypothesis that WM recodes sensory information in a format that better supports behavioral goals. We recorded EEG while participants performed color delayed-estimation tasks where the colorwheel for the response was either randomly rotated or held fixed across trials. Accordingly, observers had to remember the exact colors in the Rotation condition, whereas they could prepare for a response based on the fixed mapping between the colors and their corresponding locations on the colorwheel in the No-Rotation condition. Results showed that the color reports were faster and more precise in the No-Rotation condition even when exactly the same set of colors were tested in both conditions. To investigate how the color information was maintained in the brain, we decoded the color using a multivariate EEG classification method. The decoding was limited to the stimulus encoding period in the Rotation condition, whereas it continued to be significant during the maintenance period in the No-Rotation condition, indicating that the color information was actively maintained in the condition. Follow-up analyses suggested that the prolonged decoding was not merely driven by the covert shift of attention but rather by the recoding of sensory information into an action-oriented response format. Together, these results provide converging evidence that WM flexibly recodes sensory information depending on the specific task context to optimize subsequent behavioral performance.

Cortical and white matter substrates supporting visuospatial working memory

OBJECTIVE

In tasks involving new visuospatial information, we rely on working memory, supported by a distributed brain network. We investigated the dynamic interplay between brain regions, including cortical and white matter structures, to understand how neural interactions change with different memory loads and trials, and their subsequent impact on working memory performance.

METHODS

Patients undertook a task of immediate spatial recall during intracranial EEG monitoring. We charted the dynamics of cortical high-gamma activity and associated functional connectivity modulations in white matter tracts.

RESULTS

Elevated memory loads were linked to enhanced functional connectivity via occipital longitudinal tracts, yet decreased through arcuate, uncinate, and superior-longitudinal fasciculi. As task familiarity grew, there was increased high-gamma activity in the posterior inferior-frontal gyrus (pIFG) and diminished functional connectivity across a network encompassing frontal, parietal, and temporal lobes. Early pIFG high-gamma activity was predictive of successful recall. Including this metric in a logistic regression model yielded an accuracy of 0.76.

CONCLUSIONS

Optimizing visuospatial working memory through practice is tied to early pIFG activation and decreased dependence on irrelevant neural pathways.

SIGNIFICANCE

This study expands our knowledge of human adaptation for visuospatial working memory, showing the spatiotemporal dynamics of cortical network modulations through white matter tracts.

Visual working memories are abstractions of percepts

During perception, decoding the orientation of gratings depends on complex interactions between the orientation of the grating, aperture edges, and topographic structure of the visual map. Here, we aimed to test how aperture biases described during perception affect working memory (WM) decoding. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases for identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Neural patterns of delay period activity while maintaining the orientation of gratings with one modulator (e.g. radial) were interchangeable with patterns while maintaining gratings with the other modulator (e.g. angular) in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using models of visual field map properties. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Neural Correlates of Online Action Preparation

When performing movements in rapid succession, the brain needs to coordinate ongoing execution with the preparation of an upcoming action. Here we identify the processes and brain areas involved in this ability of online preparation. Human participants (both male and female) performed pairs of single-finger presses or three-finger chords in rapid succession, while 7T fMRI was recorded. In the overlap condition, they could prepare the second movement during the first response and in the nonoverlap condition only after the first response was completed. Despite matched perceptual and movement requirements, fMRI revealed increased brain activity in the overlap condition in regions along the intraparietal sulcus and ventral visual stream. Multivariate analyses suggested that these areas are involved in stimulus identification and action selection. In contrast, the dorsal premotor cortex, known to be involved in planning upcoming movements, showed no discernible signs of heightened activity. This observation suggests that the bottleneck during simultaneous action execution and preparation arises at the level of stimulus identification and action selection, whereas movement planning in the premotor cortex can unfold concurrently with the execution of a current action without requiring additional neural activity.

Internal attention modulates the functional state of novel stimulus-response associations in working memory

Information in working memory (WM) is crucial for guiding behavior. However, not all WM representations are equally relevant simultaneously. Current theoretical frameworks propose a functional dissociation between 'latent' and 'active' states, in which relevant representations are prioritized into an optimal (active) state to face current demands, while relevant information that is not immediately needed is maintained in a dormant (latent) state. In this context, task demands can induce rapid and flexible prioritization of information from latent to active state. Critically, these functional states have been primarily studied using simple visual memories, with attention selecting and prioritizing relevant representations to serve as templates to guide subsequent behavior. It remains unclear whether more complex WM representations, such as novel stimulus-response associations, can also be prioritized into different functional states depending on their task relevance, and if so how these different formats relate to each other. In the present study, we investigated whether novel WM-guided actions can be brought into different functional states depending on current task demands. Our results reveal that planned actions can be flexibly prioritized when needed and show how their functional state modulates their influence on ongoing behavior. Moreover, they suggest the representations of novel actions of different functional states are maintained in WM via a non-orthogonal coding scheme, thus are prone to interference.

Visual working memories are abstractions of percepts

Pioneering studies demonstrating that the contents of visual working memory (WM) can be decoded from the patterns of multivoxel activity in early visual cortex transformed not only how we study WM, but theories of how memories are stored. For instance, the ability to decode the orientation of memorized gratings is hypothesized to depend on the recruitment of the same neural encoding machinery used for perceiving orientations. However, decoding evidence cannot be used to test the so-called sensory recruitment hypothesis without understanding the underlying nature of what is being decoded. Although unknown during WM, during perception decoding the orientation of gratings does not simply depend on activities of orientation tuned neurons. Rather, it depends on complex interactions between the orientation of the grating, the aperture edges, and the topographic structure of the visual map. Here, our goals are to 1) test how these aperture biases described during perception may affect WM decoding, and 2) leverage carefully manipulated visual stimulus properties of gratings to test how sensory-like are WM codes. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases despite having identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Results indicated that fMRI patterns of delay period activity while maintaining the orientation of a grating with one modulator (eg, radial) were interchangeable with patterns while maintaining a grating with the other modulator (eg, angular). We found significant cross-classification in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using a population receptive field model of the visual field maps. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Finger motor representation supports the autonomy in arithmetic: neuroimaging evidence from abacus training

Researches have reported the close association between fingers and arithmetic. However, it remains unclear whether and how finger training can benefit arithmetic. To address this issue, we used the abacus-based mental calculation (AMC), which combines finger training and mental arithmetic learning, to explore the neural correlates underlying finger-related arithmetic training. A total of 147 Chinese children (75 M/72 F, mean age, 6.89 ± 0.46) were recruited and randomly assigned into AMC and control groups at primary school entry. The AMC group received 5 years of AMC training, and arithmetic abilities and resting-state functional magnetic resonance images data were collected from both groups at year 1/3/5. The connectome-based predictive modeling was used to find the arithmetic-related networks of each group. Compared to controls, the AMC's positively arithmetic-related network was less located in the control module, and the inter-module connections between somatomotor-default and somatomotor-control modules shifted to somatomotor-visual and somatomotor-dorsal attention modules. Furthermore, the positive network of the AMC group exhibited a segregated connectivity pattern, with more intra-module connections than the control group. Overall, our results suggested that finger motor representation with motor module involvement facilitated arithmetic-related network segregation, reflecting increased autonomy of AMC, thus reducing the dependency of arithmetic on higher-order cognitive functions.

Prospection of Potential Actions during Visual Working Memory Starts Early, Is Flexible, and Predicts Behavior

For visual working memory to serve upcoming behavior, it is crucial that we prepare for the potential use of working-memory contents ahead of time. Recent studies have demonstrated how the prospection and planning for an upcoming manual action starts early after visual encoding, and occurs alongside visual retention. Here, we address whether such "output planning" in visual working memory flexibly adapts to different visual-motor mappings, and occurs even when an upcoming action will only potentially become relevant for behavior. Human participants (female and male) performed a visual-motor working memory task in which they remembered one or two colored oriented bars for later (potential) use. We linked, and counterbalanced, the tilt of the visual items to specific manual responses. This allowed us to track planning of upcoming behavior through contralateral attenuation of β band activity, a canonical motor-cortical EEG signature of manual-action planning. The results revealed how action encoding and subsequent planning alongside visual working memory (1) reflect anticipated task demands rather than specific visual-motor mappings, (2) occur even for actions that will only potentially become relevant for behavior, and (3) are associated with faster performance for the encoded item, at the expense of performance to other working-memory content. This reveals how the potential prospective use of visual working memory content is flexibly planned early on, with consequences for the speed of memory-guided behavior.SIGNIFICANCE STATEMENT It is increasingly studied how visual working memory helps us to prepare for the future, in addition to how it helps us to hold onto the past. Recent studies have demonstrated that the planning of prospective actions occurs alongside encoding and retention in working memory. We show that such early "output planning" flexibly adapts to varying visual-motor mappings, occurs both for certain and potential actions, and predicts ensuing working-memory guided behavior. These results highlight the flexible and future-oriented nature of visual working memory, and provide insight into the neural basis of the anticipatory dynamics that translate visual representations into adaptive upcoming behavior.

Action space restructures visual working memory in prefrontal cortex

Visual working memory enables flexible behavior by decoupling sensory stimuli from behavioral actions. While previous studies have predominantly focused on the storage component of working memory, the role of future actions in shaping working memory remains unknown. To answer this question, we used two working memory tasks that allowed the dissociation of sensory and action components of working memory. We measured behavioral performance and neuronal activity in the macaque prefrontal cortex area, frontal eye fields. We show that the action space reshapes working memory, as evidenced by distinct patterns of memory tuning and attentional orienting between the two tasks. Notably, neuronal activity during the working memory period predicted future behavior and exhibited mixed selectivity in relation to the sensory space but linear selectivity relative to the action space. This linear selectivity was achieved through the rapid transformation from sensory to action space and was subsequently maintained as a stable cross-temporal population activity pattern. Combined, we provide direct physiological evidence of the action-oriented nature of frontal eye field neurons during memory tasks and demonstrate that the anticipation of behavioral outcomes plays a significant role in transforming and maintaining the contents of visual working memory.

The effect of memory load on object reconstruction: Insights from an online mouse-tracking task

Why can't we remember everything that we experience? Previous work in the domain of object memory has suggested that our ability to resolve interference between relevant and irrelevant object features may limit how much we can remember at any given moment. Here, we developed an online mouse-tracking task to study how memory load influences object reconstruction, testing participants synchronously over virtual conference calls. We first tested up to 18 participants concurrently, replicating memory findings from a condition where participants were tested individually. Next, we examined how memory load influenced mouse trajectories as participants reconstructed target objects. We found interference between the contents of working memory and what was perceived during object reconstruction, an effect that interacted with visual similarity and memory load. Furthermore, we found interference from previously studied but currently irrelevant objects, providing evidence of object-to-location binding errors. At the greatest memory load, participants were nearly three times more likely to move their mouse cursor over previously studied nontarget objects, an effect observed primarily during object reconstruction rather than in the period before the final response. As evidence of the dynamic interplay between working memory and perception, these results show that object reconstruction behavior may be altered by (i) interference between what is represented in mind and what is currently being viewed, and (ii) interference from previously studied but currently irrelevant information. Finally, we discuss how mouse tracking can provide a rich characterization of participant behavior at millisecond temporal resolution, enormously increasing power in cognitive psychology experiments.

Changes in behavioral priority influence the accessibility of working memory content

Evolving behavioral goals require the existence of selection mechanisms that prioritize task-relevant working memory (WM) content for action. Selecting an item stored in WM is known to blunt and/or reverse information loss in stimulus-specific representations of that item reconstructed from human brain activity, but extant studies have focused on all-or-none circumstances that allow or disallow an agent to select one of several items stored in WM. Conversely, behavioral studies suggest that humans can flexibly assign different levels of priority to different items stored in WM, but how doing so influences neural representations of WM content is unclear. One possibility is that assigning different levels of priority to items in WM influences the quality of those representations, resulting in more robust neural representations of high- vs. low-priority WM content. A second - and non-exclusive - possibility is that asymmetries in behavioral priority influence how rapidly neural representations of high- vs. low-priority WM content can be selected and reported. We tested these possibilities in two experiments by decoding high- and low-priority WM content from EEG recordings obtained while human volunteers performed a retrospectively cued WM task. Probabilistic changes in the behavioral relevance of a remembered item had no effect on our ability to decode it from EEG signals; instead, these changes influenced the latency at which above-chance decoding performance was reached. Thus, our results indicate that probabilistic changes in the behavioral relevance of WM content influence the ease with which memories can be selected independently of their strength.

Interference between items stored for distinct tasks in visual working memory

The action perspective on working memory suggests that memory representations are coded according to their specific temporal and behavioral task demands. This stands in contrast to theories that assume representations are stored in a task-agnostic format within a "common workspace". Here, we tested whether visual items that are memorized for different tasks are stored separately from one another or show evidence of inter-item interference during concurrent maintenance, indicating a common storage. In two experiments, we combined a framing memory task (memorize a motion direction for continuous direction report) with an embedded memory task (memorize a motion direction for a binary direction discrimination) that was placed within the retention period of the framing task. Even though the temporal and action demands were item specific, we observed two types of interference effects between the items: The embedded motion direction was (1) repulsed away and (2) degraded in precision by the motion direction of the item in the framing task. Repulsion and precision degradation increased with item similarity when both items were concurrently held in working memory. In contrast, perceptual and iconic memory control conditions revealed weaker repulsion overall and no interference effect on precision during the stimulus processing stages prior to working memory consolidation. Thus, additional inter-item interference arose uniquely within working memory. Together, our results present evidence that items that are stored for distinct tasks to be performed at distinct points in time, reside in a common workspace in working memory.

Canonical template tracking: Measuring the activation state of specific neural representations

Multivariate analyses of neural data have become increasingly influential in cognitive neuroscience since they allow to address questions about the representational signatures of neurocognitive phenomena. Here, we describe Canonical Template Tracking: a multivariate approach that employs independent localizer tasks to assess the activation state of specific representations during the execution of cognitive paradigms. We illustrate the benefits of this methodology in characterizing the particular content and format of task-induced representations, comparing it with standard (cross-)decoding and representational similarity analyses. Then, we discuss relevant design decisions for experiments using this analysis approach, focusing on the nature of the localizer tasks from which the canonical templates are derived. We further provide a step-by-step tutorial of this method, stressing the relevant analysis choices for functional magnetic resonance imaging and magneto/electroencephalography data. Importantly, we point out the potential pitfalls linked to canonical template tracking implementation and interpretation of the results, together with recommendations to mitigate them. To conclude, we provide some examples from previous literature that highlight the potential of this analysis to address relevant theoretical questions in cognitive neuroscience.

Dynamics Are the Only Constant in Working Memory

In this short perspective, we reflect upon our tendency to use oversimplified and idiosyncratic tasks in a quest to discover general mechanisms of working memory. We discuss how the work of Mark Stokes and collaborators has looked beyond localized, temporally persistent neural activity and shifted focus toward the importance of distributed, dynamic neural codes for working memory. A critical lesson from this work is that using simplified tasks does not automatically simplify the neural computations supporting behavior (even if we wish it were so). Moreover, Stokes' insights about multidimensional dynamics highlight the flexibility of the neural codes underlying cognition and have pushed the field to look beyond static measures of working memory.