Restoring Latent Visual Working Memory Representations in Human Cortex

Distinct states in visual working memory support memory benefits from extending encoding time

Visual working memory (VWM) is a fundamental cognitive mechanism with limited capacity. Recent research has highlighted the effectiveness of extending the encoding items to enhance VWM capacity. However, the mechanisms responsible for such enhancement remain unclear. Our hypothesis is that the presence of distinct states within VWM (active vs. passive) may contribute to this enhancement. Extending the encoding time allows items that surpass the active storage capacity to temporally transition into the passive state, thus freeing up resources for active storage. To test this, participants were asked to memorize two arrays sequentially, with the first array held in the passive state while they actively maintained the second. We systematically extended the encoding time for the second (active) array and observed memory benefits. These benefits came at the expense of the storage of the first (passive) array, suggesting that items exceeding the active memory capacity are temporarily stored in the passive state. Overall, the presence of distinct states in VWM allows for storing more items. The employment of the passive state contributes to the benefits of extended-encoding time in VWM.

Neural mechanisms of resource allocation in working memory

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. We decoded higher-priority memoranda with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization between and within participants. Trial-wise variability in the magnitude of delay activity in the frontal cortex predicted differences in decoded precision between low- and high-priority items in visual cortex. These results support a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in the visual cortex according to behavioral relevance, thus identifying a neural mechanism for resource allocation.

Prioritizing Working Memory Resources Depends on the Prefrontal Cortex

How the prefrontal cortex contributes to working memory remains controversial, as theories differ in their emphasis on its role in storing memories versus controlling their content. To adjudicate between these competing ideas, we tested how perturbations to the human (both sexes) lateral prefrontal cortex impact the storage and control aspects of working memory during a task that requires human subjects to allocate resources to memory items based on their behavioral priority. Our computational model made a strong prediction that disruption of this control process would counterintuitively improve memory for low-priority items. Remarkably, transcranial magnetic stimulation of retinotopically-defined superior precentral sulcus, but not intraparietal sulcus, unbalanced the prioritization of resources, improving memory for low-priority items as predicted by the model. Therefore, these results provide direct causal support for models in which the prefrontal cortex controls the allocation of resources that support working memory, rather than simply storing the features of memoranda.

Attention to memory content enhances single-unit spike sequence fidelity in the human anterior temporal lobe

Attention aids in prioritizing information relevant to our present goals. For example, attention may augment sensory processing by modulating neural activity for low-level features of the attended items. Attention can also prioritize the contents of memory, facilitating our ability to remember some pieces of information while ignoring others. Here, we examine how using attention to modulate the contents of memory affects temporally organized sequences of neuronal spiking in the human anterior temporal lobe. These spike sequences represent higher-level semantic information and occur repeatedly and consistently as participants process and encode words into memory. Our results demonstrate that attention deployed to prioritize words for memory increases the consistency of these spike sequences. Further, retroactively cueing words elicits the replay of these sequences. Our data, therefore, suggest that paying attention to prioritizing semantic content for memory may improve the temporal organization of neural spiking representations of semantic information in the anterior temporal lobe.

Neural mechanisms of learned suppression uncovered by probing the hidden attentional priority map

Attentional capture by an irrelevant salient distractor is attenuated when the distractor appears more frequently in one location, suggesting learned suppression of that location. However, it remains unclear whether suppression is proactive (before attention is directed) or reactive (after attention is allocated). Here, we investigated this using a 'pinging' technique to probe the attentional distribution before search onset. In an EEG experiment, participants searched for a shape singleton while ignoring a color singleton distractor at a high-probability location. To reveal the hidden attentional priority map, participants also performed a continuous recall spatial memory task, with a neutral placeholder display presented before search onset. Behaviorally, search was more efficient when the distractor appeared at the high-probability location. Inverted encoding analysis of EEG data showed tuning profiles that decayed during memory maintenance but were revived by the placeholder display. Notably, tuning was most pronounced at the to-be-suppressed location, suggesting initial spatial selection followed by suppression. These findings suggest that learned distractor suppression is a reactive process, providing new insights into learned spatial distractor suppression mechanisms.

The Suppression Mechanisms of Passive Memory in Visual Working Memory: The Evidence from Electroencephalography

Recent studies of visual working memory (VWM) underscore a structured hierarchy of storage states. Memories that are not immediately relevant to the task at hand but are essential for later use are transferred to a passive state, which operates independently of actively maintaining and manipulating current memories. Note that stimulating passive memory forcefully can reactivate it into an active state, resulting in a competition with active memory. Thus, to remain stable representations for both states within VWM, passive memory might involve sustained suppression during activity-silent maintenance to prevent reactivation from disrupting the current active storage. To investigate this, we analyzed lateralized EEG signals while human participants (both women and men) were engaged in a sequential presentation memory task across two experiments. The results revealed positive contralateral delayed activity components and lateralized alpha enhancement for passive memory, neural indicative of suppression on passive storage. In addition, the suppression effect was independent of the memory load in both the active and the passive states. These findings support the notion of sustained suppression during activity-silent maintenance of passive memory, facilitating the stable maintenance of distinct storage states and advancing our understanding of the dynamic coding framework in VWM.

ATTENTION MODULATES STIMULUS REPRESENTATIONS IN NEURAL FEATURE DIMENSION MAPS

Computational theories posit that attention is guided by a combination of spatial maps for individual features that can be dynamically weighted according to task goals. Consistent with this framework, when a stimulus contains several features, attending to one or another feature results in stronger fMRI responses in regions preferring the attended feature. We hypothesized that multivariate activation patterns across feature-responsive cortical regions form spatial 'feature dimension maps', which combine to guide attentional priority. We tested this prediction by reconstructing neural feature dimension maps from fMRI activation patterns across retinotopic regions of visual cortex using a spatial inverted encoding model. Participants viewed a peripheral visual stimulus at a random location which always contained moving colored dots. On each trial, participants were precued to report the predominant direction of motion or color of the stimulus, or to attend fixation. Stimulus representations in reconstructed priority maps were selectively enhanced in color-selective regions when color was attended, and in motion-selective regions when motion was attended, and this effect was primarily observed at the stimulus location. While enhancement was localized to the stimulus in color-selective regions, motion-selective regions were globally enhanced when motion was task relevant. These results suggest different cortical regions support spatial maps of different visual features, and that each map is uniquely reweighted based on task demands to guide visual behavior towards the most relevant locations based on important features.

Working Memory Is as Working Memory Does: A Pluralist Take on the Center of the Mind

Working memory is thought to be the psychological capacity that enables us to maintain or manipulate information no longer in our environment for goal-directed action. Recent work argues that working memory is not a so-called natural kind and in turn cannot explain the cognitive processes attributed to it. This paper first clarifies the scope of this earlier critique and argues for a pluralist account of working memory. Under this account, working memory is variously realized by many mechanisms that contribute to the maintenance and manipulation of information across tasks. This view in effect updates one of the earliest pluralist formulations of working memory. Juxtaposing this view against deflationary descriptions allows us to delineate two gradients that help us chart various accounts of working memory and identify their respective theoretical commitments. In turn, we can isolate those accounts that fail to accord with the evidence supporting a pluralist view, and we can begin to rehabilitate working memory as a pluralist, and ultimately more informative, construct.

Intermittent rate coding and cue-specific ensembles support working memory

Persistent, memorandum-specific neuronal spiking activity has long been hypothesized to underlie working memory1,2. However, emerging evidence suggests a potential role for 'activity-silent' synaptic mechanisms3-5. This issue remains controversial because evidence for either view has largely relied either on datasets that fail to capture single-trial population dynamics or on indirect measures of neuronal spiking. We addressed this controversy by examining the dynamics of mnemonic information on single trials obtained from large, local populations of lateral prefrontal neurons recorded simultaneously in monkeys performing a working memory task. Here we show that mnemonic information does not persist in the spiking activity of neuronal populations during memory delays, but instead alternates between coordinated 'On' and 'Off' states. At the level of single neurons, Off states are driven by both a loss of selectivity for memoranda and a return of firing rates to spontaneous levels. Further exploiting the large-scale recordings used here, we show that mnemonic information is available in the patterns of functional connections among neuronal ensembles during Off states. Our results suggest that intermittent periods of memorandum-specific spiking coexist with synaptic mechanisms to support working memory.

Prioritizing working memory resources depends on prefrontal cortex

How the prefrontal cortex contributes to working memory remains controversial, as theories differ in their emphasis on its role in storing memories versus controlling their content. To adjudicate between these competing ideas, we tested how perturbations to the human (both sexes) lateral prefrontal cortex impact the storage and control aspects of working memory during a task that requires human subjects to allocate resources to memory items based on their behavioral priority. Our computational model made a strong prediction that disruption of this control process would counterintuitively improve memory for low-priority items. Remarkably, transcranial magnetic stimulation of retinotopically-defined superior precentral sulcus, but not intraparietal sulcus, unbalanced the prioritization of resources, improving memory for low-priority items as predicted by the model. Therefore, these results provide direct causal support for models in which the prefrontal cortex controls the allocation of resources that support working memory, rather than simply storing the features of memoranda.

Electroencephalogram Decoding Reveals Distinct Processes for Directing Spatial Attention and Encoding Into Working Memory

Past work reveals a tight relationship between spatial attention and storage in visual working memory. But is spatially attending an item tantamount to working memory encoding? Here, we tracked electroencephalography (EEG) signatures of spatial attention and working memory encoding while independently manipulating the number of memory items and the spatial extent of attention in two studies of adults (N = 39; N = 33). Neural measures of spatial attention tracked the position and size of the attended area independent of the number of individuated items encoded into working memory. At the same time, multivariate decoding of the number of items stored in working memory was insensitive to variations in the breadth and position of spatial attention. Finally, representational similarity analyses provided converging evidence for a pure load signal that is insensitive to the spatial extent of the stored items. Thus, although spatial attention is a persistent partner of visual working memory, it is functionally dissociable from the selection and maintenance of individuated representations in working memory.

Maintenance and transformation of representational formats during working memory prioritization

Visual working memory depends on both material-specific brain areas in the ventral visual stream (VVS) that support the maintenance of stimulus representations and on regions in the prefrontal cortex (PFC) that control these representations. How executive control prioritizes working memory contents and whether this affects their representational formats remains an open question, however. Here, we analyzed intracranial EEG (iEEG) recordings in epilepsy patients with electrodes in VVS and PFC who performed a multi-item working memory task involving a retro-cue. We employed Representational Similarity Analysis (RSA) with various Deep Neural Network (DNN) architectures to investigate the representational format of prioritized VWM content. While recurrent DNN representations matched PFC representations in the beta band (15-29 Hz) following the retro-cue, they corresponded to VVS representations in a lower frequency range (3-14 Hz) towards the end of the maintenance period. Our findings highlight the distinct coding schemes and representational formats of prioritized content in VVS and PFC.

Manipulating attentional priority creates a trade-off between memory and sensory representations in human visual cortex

People often remember visual information over brief delays while actively engaging with ongoing inputs from the surrounding visual environment. Depending on the situation, one might prioritize mnemonic contents (i.e., remembering details of a past event), or preferentially attend sensory inputs (i.e., minding traffic while crossing a street). Previous fMRI work has shown that early sensory regions can simultaneously represent both mnemonic and passively viewed sensory information. Here we test the limits of such simultaneity by manipulating attention towards sensory distractors during a working memory task performed by human subjects during fMRI scanning. Participants remembered the orientation of a target grating while a distractor grating was shown during the middle portion of the memory delay. Critically, there were several subtle changes in the contrast and the orientation of the distractor, and participants were cued to either ignore the distractor, detect a change in contrast, or detect a change in orientation. Despite sensory stimulation being matched in all three conditions, the fidelity of memory representations in early visual cortex was highest when the distractor was ignored, intermediate when participants attended distractor contrast, and lowest when participants attended the orientation of the distractor during the delay. In contrast, the fidelity of distractor representations was lowest when ignoring the distractor, intermediate when attending distractor-contrast, and highest when attending distractor-orientation. These data suggest a trade-off in early sensory representations when engaging top-down feedback to attend both seen and remembered features and may partially explain memory failures that occur when subjects are distracted by external events.

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Neural mechanisms of resource allocation in working memory

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. Higher-priority memoranda were decoded with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization. Remarkably, trialwise variability in the magnitude of delay activity in frontal cortex predicted differences in decoded precision between low and high-priority items in visual cortex. These results suggest a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in visual cortex according to behavioral relevance, thus, identifying a neural mechanism for resource allocation.

Concurrent attention to hetero-depth surfaces in 3-D visual space is governed by theta rhythm

When simultaneously confronted with multiple attentional targets, visual system employs a time-multiplexing approach in which each target alternates for prioritized access, a mechanism broadly known as rhythmic attentional sampling. For the past decade, rhythmic attentional sampling has received mounting support from converging behavioral and neural findings. However, so compelling are these findings that a critical test ground has been long overshadowed, namely the 3-D visual space where attention is complicated by extraction of the spatial layout of surfaces extending beyond 2-D planes. It remains unknown how attentional deployment to multiple targets is accomplished in the 3-D space. Here, we provided a time-resolved portrait of the behavioral and neural dynamics when participants concurrently attended to two surfaces defined by motion-depth conjunctions. To characterize the moment-to-moment attentional modulation effects, we measured perceptual sensitivity to the hetero-depth surface motions on a fine temporal scale and reconstructed their neural representations using a time-resolved multivariate inverted encoding model. We found that the perceptual sensitivity to the two surface motions rhythmically fluctuated over time at ~4 Hz, with one's enhancement closely tracked by the other's diminishment. Moreover, the behavioral pattern was coupled with an ongoing periodic alternation in strength between the two surface motion representations in the same frequency. Together, our findings provide the first converging evidence of an attentional "pendulum" that rhythmically traverses different stereoscopic depth planes and are indicative of a ubiquitous attentional time multiplexor based on theta rhythm in the 3-D visual space.

Internal attention modulates the functional state of novel stimulus-response associations in working memory

Information in working memory (WM) is crucial for guiding behavior. However, not all WM representations are equally relevant simultaneously. Current theoretical frameworks propose a functional dissociation between 'latent' and 'active' states, in which relevant representations are prioritized into an optimal (active) state to face current demands, while relevant information that is not immediately needed is maintained in a dormant (latent) state. In this context, task demands can induce rapid and flexible prioritization of information from latent to active state. Critically, these functional states have been primarily studied using simple visual memories, with attention selecting and prioritizing relevant representations to serve as templates to guide subsequent behavior. It remains unclear whether more complex WM representations, such as novel stimulus-response associations, can also be prioritized into different functional states depending on their task relevance, and if so how these different formats relate to each other. In the present study, we investigated whether novel WM-guided actions can be brought into different functional states depending on current task demands. Our results reveal that planned actions can be flexibly prioritized when needed and show how their functional state modulates their influence on ongoing behavior. Moreover, they suggest the representations of novel actions of different functional states are maintained in WM via a non-orthogonal coding scheme, thus are prone to interference.

Dissociating the pre-activation of word meaning and form during sentence comprehension: Evidence from EEG representational similarity analysis

During language comprehension, the processing of each incoming word is facilitated in proportion to its predictability. Here, we asked whether anticipated upcoming linguistic information is actually pre-activated before new bottom-up input becomes available, and if so, whether this pre-activation is limited to the level of semantic features, or whether extends to representations of individual word-forms (orthography/phonology). We carried out Representational Similarity Analysis on EEG data while participants read highly constraining sentences. Prior to the onset of the expected target words, sentence pairs predicting semantically related words (financial "bank" - "loan") and form-related words (financial "bank" - river "bank") produced more similar neural patterns than pairs predicting unrelated words ("bank" - "lesson"). This provides direct neural evidence for item-specific semantic and form predictive pre-activation. Moreover, the semantic pre-activation effect preceded the form pre-activation effect, suggesting that top-down pre-activation is propagated from higher to lower levels of the linguistic hierarchy over time.

Age-related differences in the removal of information from working memory

Removal has been assumed to be a core mechanism in working memory. However, it remains unclear whether children can actively remove outdated information from working memory and how this ability develops as children age. The current study aimed to examine age-related differences in removal ability and its relations with cognitive control and working memory capacity. Children aged 7, 9, and 11 years performed a modified working memory updating task assessing removal efficiency. In addition, a battery of cognitive control and working memory capacity tasks was administered. Results indicated that updating response times decreased considerably when a longer time was given for removal, suggesting that children aged 7 to 11 years can actively remove outdated items from working memory prior to encoding the new ones and that removal efficiency increased with age. More important, age-related increases in removal efficiency occurred concurrently with the development of working memory capacity. Proactive control predicted removal efficiency over and beyond age and working memory capacity. The findings shed new light on the mechanisms underlying the development of working memory updating.

A construct-first approach to consciousness science

We propose a new approach to consciousness science that instead of comparing complex theoretical positions deconstructs existing theories, takes their central assumptions while disregarding their auxiliary hypotheses, and focuses its investigations on the main constructs that these central assumptions rely on (like global workspace, recurrent processing, metarepresentation). Studying how these main constructs are anchored in lower-level constructs characterizing underlying neural processing will not just offer an alternative to theory comparisons but will also take us one step closer to empirical resolutions. Moreover, exploring the compatibility and possible combinations of the lower-level constructs will allow for new theoretical syntheses. This construct-first approach will improve our ability to understand the commitments of existing theories and pave the way for moving beyond them.

Feature-Specific Salience Maps in Human Cortex

Priority map theory is a leading framework for understanding how various aspects of stimulus displays and task demands guide visual attention. Per this theory, the visual system computes a priority map, which is a representation of visual space indexing the relative importance, or priority, of locations in the environment. Priority is computed based on both salience, defined based on image-computable properties; and relevance, defined by an individual's current goals, and is used to direct attention to the highest-priority locations for further processing. Computational theories suggest that priority maps identify salient locations based on individual feature dimensions (e.g., color, motion), which are integrated into an aggregate priority map. While widely accepted, a core assumption of this framework, the existence of independent feature dimension maps in visual cortex, remains untested. Here, we tested the hypothesis that retinotopic regions selective for specific feature dimensions (color or motion) in human cortex act as neural feature dimension maps, indexing salient locations based on their preferred feature. We used fMRI activation patterns to reconstruct spatial maps while male and female human participants viewed stimuli with salient regions defined by relative color or motion direction. Activation in reconstructed spatial maps was localized to the salient stimulus position in the display. Moreover, the strength of the stimulus representation was strongest in the ROI selective for the salience-defining feature. Together, these results suggest that feature-selective extrastriate visual regions highlight salient locations based on local feature contrast within their preferred feature dimensions, supporting their role as neural feature dimension maps.SIGNIFICANCE STATEMENT Identifying salient information is important for navigating the world. For example, it is critical to detect a quickly approaching car when crossing the street. Leading models of computer vision and visual search rely on compartmentalized salience computations based on individual features; however, there has been no direct empirical demonstration identifying neural regions as responsible for performing these dissociable operations. Here, we provide evidence of a critical double dissociation that neural activation patterns from color-selective regions prioritize the location of color-defined salience while minimally representing motion-defined salience, whereas motion-selective regions show the complementary result. These findings reveal that specialized cortical regions act as neural "feature dimension maps" that are used to index salient locations based on specific features to guide attention.

Temporally Dissociable Mechanisms of Spatial, Feature, and Motor Selection during Working Memory-guided Behavior

Working memory (WM) is a capacity- and duration-limited system that forms a temporal bridge between fleeting sensory phenomena and possible actions. But how are the contents of WM used to guide behavior? A recent high-profile study reported evidence for simultaneous access to WM content and linked motor plans during WM-guided behavior, challenging serial models where task-relevant WM content is first selected and then mapped on to a task-relevant motor response. However, the task used in that study was not optimized to distinguish the selection of spatial versus nonspatial visual information stored in memory, nor to distinguish whether or how the chronometry of selecting nonspatial visual information stored in memory might differ from the selection of linked motor plans. Here, we revisited the chronometry of spatial, feature, and motor selection during WM-guided behavior using a task optimized to disentangle these processes. Concurrent EEG and eye position recordings revealed clear evidence for temporally dissociable spatial, feature, and motor selection during this task. Thus, our data reveal the existence of multiple WM selection mechanisms that belie conceptualizations of WM-guided behavior based on purely serial or parallel visuomotor processing.

Motion direction is represented as a bimodal probability distribution in the human visual cortex

Humans infer motion direction from noisy sensory signals. We hypothesize that to make these inferences more precise, the visual system computes motion direction not only from velocity but also spatial orientation signals - a 'streak' created by moving objects. We implement this hypothesis in a Bayesian model, which quantifies knowledge with probability distributions, and test its predictions using psychophysics and fMRI. Using a probabilistic pattern-based analysis, we decode probability distributions of motion direction from trial-by-trial activity in the human visual cortex. Corroborating the predictions, the decoded distributions have a bimodal shape, with peaks that predict the direction and magnitude of behavioral errors. Interestingly, we observe similar bimodality in the distribution of the observers' behavioral responses across trials. Together, these results suggest that observers use spatial orientation signals when estimating motion direction. More broadly, our findings indicate that the cortical representation of low-level visual features, such as motion direction, can reflect a combination of several qualitatively distinct signals.

Parietal-driven visual working memory representation in occipito-temporal cortex

Human fMRI studies have documented extensively that the content of visual working memory (VWM) can be reliably decoded from fMRI voxel response patterns during the delay period in both the occipito-temporal cortex (OTC), including early visual areas (EVC), and the posterior parietal cortex (PPC).1,2,3,4 Further work has revealed that VWM signal in OTC is largely sustained by feedback from associative areas such as prefrontal cortex (PFC) and PPC.4,5,6,7,8,9 It is unclear, however, if feedback during VWM simply restores sensory representations initially formed in OTC or if it can reshape the representational content of OTC during VWM delay. Taking advantage of a recent finding showing that object representational geometry differs between OTC and PPC in perception,10 here we find that, during VWM delay, the object representational geometry in OTC becomes more aligned with that of PPC during perception than with itself during perception. This finding supports the role of feedback in shaping the content of VWM in OTC, with the VWM content of OTC more determined by information retained in PPC than by the sensory information initially encoded in OTC.

Bend but don't break: Prioritization protects working memory from displacement but leaves it vulnerable to distortion from distraction

Perceptual distraction distorts visual working memory representations. Previous research has shown that memory responses are systematically biased towards passively viewed visual distractors that are similar to the memoranda. However, it remains unclear whether the prioritization of one working memory representation over another reduces the impact of perceptual distractors. We designed a study with five different types of visual distraction that varied in engagement and found evidence for both subtle distortions and catastrophic failures of memory. Importantly, prioritization protected working memories from catastrophic loss (fewer "swap errors") but rendered them more vulnerable to distortion (greater attractive "biases" towards the distractor). Our findings demonstrate that prioritization does not simply protect working memory from any and all interference, but rather it reduces the likelihood of catastrophic disruption from perceptual distraction at the cost of an increased likelihood of distortion.

A recurrent neural network model of prefrontal brain activity during a working memory task

When multiple items are held in short-term memory, cues that retrospectively prioritise one item over another (retro-cues) can facilitate subsequent recall. However, the neural and computational underpinnings of this effect are poorly understood. One recent study recorded neural signals in the macaque lateral prefrontal cortex (LPFC) during a retro-cueing task, contrasting delay-period activity before (pre-cue) and after (post-cue) retrocue onset. They reported that in the pre-cue delay, the individual stimuli were maintained in independent subspaces of neural population activity, whereas in the post-cue delay, the prioritised items were rotated into a common subspace, potentially allowing a common readout mechanism. To understand how such representational transitions can be learnt through error minimisation, we trained recurrent neural networks (RNNs) with supervision to perform an equivalent cued-recall task. RNNs were presented with two inputs denoting conjunctive colour-location stimuli, followed by a pre-cue memory delay, a location retrocue, and a post-cue delay. We found that the orthogonal-to-parallel geometry transformation observed in the macaque LPFC emerged naturally in RNNs trained to perform the task. Interestingly, the parallel geometry only developed when the cued information was required to be maintained in short-term memory for several cycles before readout, suggesting that it might confer robustness during maintenance. We extend these findings by analysing the learning dynamics and connectivity patterns of the RNNs, as well as the behaviour of models trained with probabilistic cues, allowing us to make predictions for future studies. Overall, our findings are consistent with recent theoretical accounts which propose that retrocues transform the prioritised memory items into a prospective, action-oriented format.

Ultra-High Field Imaging of Human Visual Cognition

Functional magnetic resonance imaging (fMRI), the key methodology for mapping the functions of the human brain in a noninvasive manner, is limited by low temporal and spatial resolution. Recent advances in ultra-high field (UHF) fMRI provide a mesoscopic (i.e., submillimeter resolution) tool that allows us to probe laminar and columnar circuits, distinguish bottom-up versus top-down pathways, and map small subcortical areas. We review recent work demonstrating that UHF fMRI provides a robust methodology for imaging the brain across cortical depths and columns that provides insights into the brain's organization and functions at unprecedented spatial resolution, advancing our understanding of the fine-scale computations and interareal communication that support visual cognition.

Neural population dynamics of human working memory

The activity of neurons in macaque prefrontal cortex (PFC) persists during working memory (WM) delays, providing a mechanism for memory.1,2,3,4,5,6,7,8,9,10,11 Although theory,11,12 including formal network models,13,14 assumes that WM codes are stable over time, PFC neurons exhibit dynamics inconsistent with these assumptions.15,16,17,18,19 Recently, multivariate reanalyses revealed the coexistence of both stable and dynamic WM codes in macaque PFC.20,21,22,23 Human EEG studies also suggest that WM might contain dynamics.24,25 Nonetheless, how WM dynamics vary across the cortical hierarchy and which factors drive dynamics remain unknown. To elucidate WM dynamics in humans, we decoded WM content from fMRI responses across multiple cortical visual field maps.26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 We found coexisting stable and dynamic neural representations of WM during a memory-guided saccade task. Geometric analyses of neural subspaces revealed that early visual cortex exhibited stronger dynamics than high-level visual and frontoparietal cortex. Leveraging models of population receptive fields, we visualized and made the neural dynamics interpretable. We found that during WM delays, V1 population initially encoded a narrowly tuned bump of activation centered on the peripheral memory target. Remarkably, this bump then spread inward toward foveal locations, forming a vector along the trajectory of the forthcoming memory-guided saccade. In other words, the neural code transformed into an abstraction of the stimulus more proximal to memory-guided behavior. Therefore, theories of WM must consider both sensory features and their task-relevant abstractions because changes in the format of memoranda naturally drive neural dynamics.

Cycles of goal silencing and reactivation underlie complex problem-solving in primate frontal and parietal cortex

While classic views proposed that working memory (WM) is mediated by sustained firing, recent evidence suggests a contribution of activity-silent states. Within WM, human neuroimaging studies suggest a switch between attentional foreground and background, with only the foregrounded item represented in active neural firing. To address this process at the cellular level, we recorded prefrontal (PFC) and posterior parietal (PPC) neurons in a complex problem-solving task, with monkeys searching for one or two target locations in a first cycle of trials, and retaining them for memory-guided revisits on subsequent cycles. When target locations were discovered, neither frontal nor parietal neurons showed sustained goal-location codes continuing into subsequent trials and cycles. Instead there were sequences of timely goal silencing and reactivation, and following reactivation, sustained states until behavioral response. With two target locations, goal representations in both regions showed evidence of transitions between foreground and background, but the PFC representation was more complete, extending beyond the current trial to include both past and future selections. In the absence of unbroken sustained codes, different neuronal states interact to support maintenance and retrieval of WM representations across successive trials.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Internal selective attention is delayed by competition between endogenous and exogenous factors

External attention is mediated by competition between endogenous (goal-driven) and exogenous (stimulus-driven) factors, with the balance of competition determining which stimuli are selected. Occasionally, exogenous factors "win" this competition and drive the selection of task-irrelevant stimuli. Endogenous and exogenous selection mechanisms may also compete to control the selection of internal representations (e.g., those stored in working memory), but whether this competition is resolved in the same way as external attention is unknown. Here, we leveraged the high temporal resolution of human EEG to determine how competition between endogenous and exogenous factors influences the selection of internal representations. Unlike external attention, competition did not prompt the selection of task-irrelevant working memory content. Instead, it delayed the endogenous selection of task-relevant working memory content by several hundred milliseconds. Thus, competition between endogenous and exogenous factors influences internal selective attention, but in a different way than external selective attention.

Dynamic neural reconstructions of attended object location and features using EEG

Attention allows us to select relevant and ignore irrelevant information from our complex environments. What happens when attention shifts from one item to another? To answer this question, it is critical to have tools that accurately recover neural representations of both feature and location information with high temporal resolution. In the present study, we used human electroencephalography (EEG) and machine learning to explore how neural representations of object features and locations update across dynamic shifts of attention. We demonstrate that EEG can be used to create simultaneous time courses of neural representations of attended features (time point-by-time point inverted encoding model reconstructions) and attended location (time point-by-time point decoding) during both stable periods and across dynamic shifts of attention. Each trial presented two oriented gratings that flickered at the same frequency but had different orientations; participants were cued to attend one of them and on half of trials received a shift cue midtrial. We trained models on a stable period from Hold attention trials and then reconstructed/decoded the attended orientation/location at each time point on Shift attention trials. Our results showed that both feature reconstruction and location decoding dynamically track the shift of attention and that there may be time points during the shifting of attention when 1) feature and location representations become uncoupled and 2) both the previously attended and currently attended orientations are represented with roughly equal strength. The results offer insight into our understanding of attentional shifts, and the noninvasive techniques developed in the present study lend themselves well to a wide variety of future applications.NEW & NOTEWORTHY We used human EEG and machine learning to reconstruct neural response profiles during dynamic shifts of attention. Specifically, we demonstrated that we could simultaneously read out both location and feature information from an attended item in a multistimulus display. Moreover, we examined how that readout evolves over time during the dynamic process of attentional shifts. These results provide insight into our understanding of attention, and this technique carries substantial potential for versatile extensions and applications.

Mnemonic representations in human lateral geniculate nucleus

There is a growing appreciation for the role of the thalamus in high-level cognition. Motivated by findings that internal cognitive state drives activity in feedback layers of primary visual cortex (V1) that target the lateral geniculate nucleus (LGN), we investigated the role of LGN in working memory (WM). Specifically, we leveraged model-based neuroimaging approaches to test the hypothesis that human LGN encodes information about spatial locations temporarily encoded in WM. First, we localized and derived a detailed topographic organization in LGN that accords well with previous findings in humans and non-human primates. Next, we used models constructed on the spatial preferences of LGN populations in order to reconstruct spatial locations stored in WM as subjects performed modified memory-guided saccade tasks. We found that population LGN activity faithfully encoded the spatial locations held in memory in all subjects. Importantly, our tasks and models allowed us to dissociate the locations of retinal stimulation and the motor metrics of memory-guided saccades from the maintained spatial locations, thus confirming that human LGN represents true WM information. These findings add LGN to the growing list of subcortical regions involved in WM, and suggest a key pathway by which memories may influence incoming processing at the earliest levels of the visual hierarchy.

Prioritizing flexible working memory representations through retrospective attentional strengthening

Previous work has proposed two potential benefits of retrospective attention on working memory (WM): target strengthening and non-target inhibition. It remains unknown which hypothesis contributes to the improved WM performance, yet the neural mechanisms responsible for this attentional benefit are unclear. Here, we recorded electroencephalography (EEG) signals while 33 participants performed a retrospective-cue WM task. Multivariate pattern classification analysis revealed that only representations of target features were enhanced by valid retrospective attention during retention, supporting the target strengthening hypothesis. Further univariate analysis found that mid-frontal theta inter-trial phase coherence (ITPC) and ERP components were modulated by valid retrospective attention and correlated with individual differences and moment-to-moment fluctuations on behavioral outcomes, suggesting that both trait- and state-level variability in attentional preparatory processes influence goal-directed behavior. Furthermore, task-irrelevant target spatial location could be decoded from EEG signals, indicating that enhanced spatial binding of target representation is vital to high WM precision. Importantly, frontoparietal theta-alpha phase-amplitude coupling was increased by valid retrospective attention and predicted the reduced random guessing rates. This long-range connection supported top-down information flow in the engagement of frontoparietal networks, which might organize attentional states to integrate target features. Altogether, these results provide neurophysiological bases that retrospective attention improves WM precision by enhancing flexible target representation and emphasize the critical role of the frontoparietal attentional network in the control of WM representations.

Spectrotemporal content of human auditory working memory represented in functional connectivity patterns

Recent research suggests that working memory (WM), the mental sketchpad underlying thinking and communication, is maintained by multiple regions throughout the brain. Whether parts of a stable WM representation could be distributed across these brain regions is, however, an open question. We addressed this question by examining the content-specificity of connectivity-pattern matrices between subparts of cortical regions-of-interest (ROI). These connectivity patterns were calculated from functional MRI obtained during a ripple-sound auditory WM task. Statistical significance was assessed by comparing the decoding results to a null distribution derived from a permutation test considering all comparable two- to four-ROI connectivity patterns. Maintained WM items could be decoded from connectivity patterns across ROIs in frontal, parietal, and superior temporal cortices. All functional connectivity patterns that were specific to maintained sound content extended from early auditory to frontoparietal cortices. Our results demonstrate that WM maintenance is supported by content-specific patterns of functional connectivity across different levels of cortical hierarchy.

Neural representations of competing stimuli along the dorsal and ventral visual pathways during binocular rivalry

Binocular rivalry arises when two discrepant stimuli are simultaneously presented to different eyes, during which observers consciously experience vivid perceptual alternations without physical changes in visual inputs. Neural dynamics tracking such perceptual alternations have been identified at both early and late visual areas, leading to the fundamental debate concerning the primary neural substrate underlying binocular rivalry. One promising hypothesis that might reconcile these seemingly paradoxical findings is a gradual shift from interocular competition between monocular neurons to pattern competition among binocular neurons. Here, we examined this hypothesis by investigating how neural representations of rivalrous stimuli evolved along the visual pathway. We found that representations of the dominant and the suppressed stimuli initially co-existed in V1, which were enhanced and attenuated respectively in extrastriate visual areas. Notably, neural activity in V4 was dictated by the representation of the dominant stimulus, while the representation of the suppressed stimulus was only partially inhibited in dorsal areas V3A and MT+. Our findings revealed a progressive transition from the co-existing representations of the rivalrous inputs to the dictatorial representation of the dominant stimulus in the ventral pathway, and advocated different cortical evolutionary patterns of visual representations between the dorsal and the ventral pathways.

The entorhinal-DG/CA3 pathway in the medial temporal lobe retains visual working memory of a simple surface feature

Classic models consider working memory (WM) and long-term memory as distinct mental faculties that are supported by different neural mechanisms. Yet, there are significant parallels in the computation that both types of memory require. For instance, the representation of precise item-specific memory requires the separation of overlapping neural representations of similar information. This computation has been referred to as pattern separation, which can be mediated by the entorhinal-DG/CA3 pathway of the medial temporal lobe (MTL) in service of long-term episodic memory. However, although recent evidence has suggested that the MTL is involved in WM, the extent to which the entorhinal-DG/CA3 pathway supports precise item-specific WM has remained elusive. Here, we combine an established orientation WM task with high-resolution fMRI to test the hypothesis that the entorhinal-DG/CA3 pathway retains visual WM of a simple surface feature. Participants were retrospectively cued to retain one of the two studied orientation gratings during a brief delay period and then tried to reproduce the cued orientation as precisely as possible. By modeling the delay-period activity to reconstruct the retained WM content, we found that the anterior-lateral entorhinal cortex (aLEC) and the hippocampal DG/CA3 subfield both contain item-specific WM information that is associated with subsequent recall fidelity. Together, these results highlight the contribution of MTL circuitry to item-specific WM representation.

Changes in behavioral priority influence the accessibility of working memory content

Evolving behavioral goals require the existence of selection mechanisms that prioritize task-relevant working memory (WM) content for action. Selecting an item stored in WM is known to blunt and/or reverse information loss in stimulus-specific representations of that item reconstructed from human brain activity, but extant studies have focused on all-or-none circumstances that allow or disallow an agent to select one of several items stored in WM. Conversely, behavioral studies suggest that humans can flexibly assign different levels of priority to different items stored in WM, but how doing so influences neural representations of WM content is unclear. One possibility is that assigning different levels of priority to items in WM influences the quality of those representations, resulting in more robust neural representations of high- vs. low-priority WM content. A second - and non-exclusive - possibility is that asymmetries in behavioral priority influence how rapidly neural representations of high- vs. low-priority WM content can be selected and reported. We tested these possibilities in two experiments by decoding high- and low-priority WM content from EEG recordings obtained while human volunteers performed a retrospectively cued WM task. Probabilistic changes in the behavioral relevance of a remembered item had no effect on our ability to decode it from EEG signals; instead, these changes influenced the latency at which above-chance decoding performance was reached. Thus, our results indicate that probabilistic changes in the behavioral relevance of WM content influence the ease with which memories can be selected independently of their strength.

Predicted utility modulates working memory fidelity in the brain

The predicted utility of information stored in working memory (WM) is hypothesized to influence the strategic allocation of WM resources. Prior work has shown that when information is prioritized, it is remembered with greater precision relative to other remembered items. However, these paradigms often complicate interpretation of the effects of predicted utility on item fidelity due to a concurrent memory load. Likewise, no fMRI studies have examined whether the predicted utility of an item modulates fidelity in the neural representation of items during the memory delay without a concurrent load. In the current study, we used fMRI to investigate whether predicted utility influences fidelity of WM representations in the brain. Using a generative model multivoxel analysis approach to estimate the quality of remembered representations across predicted utility conditions, we observed that items with greater predicted utility are maintained in memory with greater fidelity, even when they are the only item being maintained. Further, we found that this pattern follows a parametric relationship where more predicted utility corresponded to greater fidelity. These precision differences could not be accounted for based on a redistribution of resources among already-remembered items. Rather, we interpret these results in terms of a gating mechanism that allows for pre-allocation of resources based on predicted value alone. This evidence supports a theoretical distinction between resource allocation that occurs as a result of load and resource pre-allocation that occurs as a result of predicted utility.

Retrospective cue benefits in visual working memory are limited to a single location at a time

Working memory (WM) performance can be improved by an informative cue presented during storage. This effect, termed a retrocue benefit, can be used to study limits on how human observers prioritize information stored in WM for behavioral output. There is disagreement about whether retrocue benefits extend to multiple WM locations. Here, we hypothesized that multiple retrocues may improve some aspects of memory performance (e.g., a reduction in random guessing) while worsening others (e.g., an increase in the probability of reporting a feature presented at a non-probed location). We tested this possibility in three experiments. Participants remembered arrays of four orientations or colors over a brief delay, and spatial retrocues instructed participants to prioritize zero, one, two, or all four remembered orientations for possible report. At the end of the trial, participants recalled the orientation that appeared at one location. The results of this study revealed that participants' recall errors were lower during cue-one relative to cue-two and cue-four trials, and this benefit was driven primarily by a reduction in random guessing during cue-one trials. We found no evidence suggesting that multiple spatial cues (i.e., during cue-two trials) induced a trade-off between memory precision, random guessing, and non-target reports compared to neutral trials (i.e., cue-zero or cue-four). Thus, cuing participants to prioritize information appearing at multiple unique spatial positions led to no improvement in memory performance compared to neutral or no-cue trials, providing additional support for the view that retrocue benefits on WM performance are limited to a single spatial location at a time.

Awareness of the relative quality of spatial working memory representations

Working memory (WM) is the ability to maintain and manipulate information no longer accessible in the environment. The brain maintains WM representations over delay periods in noisy population-level activation patterns, resulting in variability in WM representations across items and trials. It is established that participants can introspect aspects of the quality of WM representations, and that they can accurately compare which of several WM representations of stimulus features like orientation or color is better on each trial. However, whether this ability to evaluate and compare the quality of multiple WM representations extends to spatial WM tasks remains unknown. Here, we employed a memory-guided saccade task to test recall errors for remembered spatial locations when participants were allowed to choose the most precise representation to report. Participants remembered either one or two spatial locations over a delay and reported one item's location with a saccade. On trials with two spatial locations, participants reported either the spatial location of a randomly cued item, or the location of the stimulus they remembered best. We found a significant improvement in recall error and increase in response time (RT) when participants reported their best-remembered item compared with trials in which they were randomly cued. These results demonstrate that participants can accurately introspect the relative quality of neural WM representations for spatial position, consistent with previous observations for other stimulus features, and support a model of WM coding involving noisy representations across items and trials.

Turning Attention Inside Out: How Working Memory Serves Behavior

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Temporal expectation based on the duration variability modulates alpha oscillations during working memory retention

While maintaining information over a delay of time, working memory (WM) also allows individuals to prepare the mnemonic contents for prospective utilisation. However, it remains unclear whether the expectation of the time of WM test could modulate neural responses during the retention interval of WM and subsequent performance. Here, we investigated whether temporal expectations based on the variability of delay duration can modulate 9-13 Hz alpha oscillations during WM retention and whether the expectation-induced alpha activity was associated with WM performance. Participants performed a retro-cueing WM task with magnetoencephalography (MEG) (Experiment 1) and a standard WM task with electroencephalography (EEG) (Experiment 2). The expectation of the timing of the WM test was manipulated by the temporal structure of the tasks with small or large variability in the delay durations. We showed that alpha oscillations during retention interval and WM performance varied with duration variability in both of the MEG and EEG experiments. The novel finding was greater alpha-power attenuation over the left frontal and parietal regions during WM retention when the duration variability was small and the test onset was predictable, compared to when the duration variability was large and the test onset was less predictable. Importantly, we observed a positive relationship in variability difference between the response benefit and alpha-power attenuation in the left posterior parietal regions at both MEG-source and EEG-electrode levels. Finally, we confirmed the behavioural benefit when a condition with a fixed delay-duration was included in a behavioural experiment (Experiment 3). When conjoined, the delay duration enables individuals to anticipate when the relevant information would be put to work, and alpha oscillations track the anticipatory states during WM maintenance.

Reactivating ordinal position information from auditory sequence memory in human brains

Retaining a sequence of events in their order is a core ability of many cognitive functions, such as speech recognition, movement control, and episodic memory. Although content representations have been widely studied in working memory (WM), little is known about how ordinal position information of an auditory sequence is retained in the human brain as well as its coding characteristics. In fact, there is still a lack of an efficient approach to directly accessing the stored ordinal position code during WM retention. Here, 31 participants performed an auditory sequence WM task with their brain activities recorded using electroencephalography (EEG). We developed new triggering events that could successfully reactivate neural representations of ordinal position during the delay period. Importantly, the ordinal position reactivation is further related to recognition behavior, confirming its indexing of WM storage. Furthermore, the ordinal position code displays an intriguing “stable-dynamic” format, i.e. undergoing the same dynamic neutral trajectory in the multivariate neural space during both encoding and retention (whenever reactivated). Overall, our results provide an effective approach to accessing the behaviorally-relevant ordinal position information in auditory sequence WM and reveal its new temporal characteristics.

Common Neural Mechanisms Control Attention and Working Memory

Although previous studies point to qualitative similarities between working memory (WM) and attention, the degree to which these two constructs rely on shared neural mechanisms remains unknown. Focusing on one such potentially shared mechanism, we tested the hypothesis that selecting an item within WM utilizes similar neural mechanisms as selecting a visible item via a shift of attention. We used fMRI and machine learning to decode both the selection among items visually available and the selection among items stored in WM in human subjects (both sexes). Patterns of activity in visual, parietal, and to a lesser extent frontal cortex predicted the locations of the selected items. Critically, these patterns were strikingly interchangeable; classifiers trained on data during attentional selection predicted selection from WM, and classifiers trained on data during selection from memory predicted attentional selection. Using models of voxel receptive fields, we visualized topographic population activity that revealed gain enhancements at the locations of the externally and internally selected items. Our results suggest that selecting among perceived items and selecting among items in WM share a common mechanism. This common mechanism, analogous to a shift of spatial attention, controls the relative gains of neural populations that encode behaviorally relevant information.SIGNIFICANCE STATEMENT How we allocate our attention to external stimuli that we see and to internal representations of stimuli stored in memory might rely on a common mechanism. Supporting this hypothesis, we demonstrated that not only could patterns of human brain activity predict which items were selected during perception and memory, but that these patterns were interchangeable during external and internal selection. Additionally, this generalized selection mechanism operates by changes in the gains of the neural populations both encoding attended sensory representations and storing relevant memory representations.

Reduced variability of bursting activity during working memory

Working memories have long been thought to be maintained by persistent spiking. However, mounting evidence from multiple-electrode recording (and single-trial analyses) shows that the underlying spiking is better characterized by intermittent bursts of activity. A counterargument suggested this intermittent activity is at odds with observations that spike-time variability reduces during task performance. However, this counterargument rests on assumptions, such as randomness in the timing of the bursts, which may not be correct. Thus, we analyzed spiking and LFPs from monkeys' prefrontal cortex (PFC) to determine if task-related reductions in variability can co-exist with intermittent spiking. We found that it does because both spiking and associated gamma bursts were task-modulated, not random. In fact, the task-related reduction in spike variability could largely be explained by a related reduction in gamma burst variability. Our results provide further support for the intermittent activity models of working memory as well as novel mechanistic insights into how spike variability is reduced during cognitive tasks.

Neural Reinstatement Tracks Spread of Attention between Object Features in Working Memory

Attention can be allocated in working memory (WM) to select and privilege relevant content. It is unclear whether attention selects individual features or whole objects in WM. Here, we used behavioral measures, eye-tracking, and EEG to test the hypothesis that attention spreads between an object's features in WM. Twenty-six participants completed a WM task that asked them to recall the angle of one of two oriented, colored bars after a delay while EEG and eye-tracking data were collected. During the delay, an orthogonal "incidental task" cued the color of one item for a match/mismatch judgment. On congruent trials (50%), the cued item was probed for subsequent orientation recall; on incongruent trials (50%), the other memory item was probed. As predicted, selecting the color of an object in WM brought other features of the cued object into an attended state as revealed by EEG decoding, oscillatory α-power, gaze bias, and improved orientation recall performance. Together, the results show that attentional selection spreads between an object's features in WM, consistent with object-based attentional selection. Analyses of neural processing at recall revealed that the selected object was automatically compared with the probe, whether it was the target for recall or not. This provides a potential mechanism for the observed benefits of nonpredictive cueing in WM, where a selected item is prioritized for subsequent decision-making.

Priority-based transformations of stimulus representation in visual working memory

How does the brain prioritize among the contents of working memory (WM) to appropriately guide behavior? Previous work, employing inverted encoding modeling (IEM) of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) datasets, has shown that unprioritized memory items (UMI) are actively represented in the brain, but in a "flipped", or opposite, format compared to prioritized memory items (PMI). To acquire independent evidence for such a priority-based representational transformation, and to explore underlying mechanisms, we trained recurrent neural networks (RNNs) with a long short-term memory (LSTM) architecture to perform a 2-back WM task. Visualization of LSTM hidden layer activity using Principal Component Analysis (PCA) confirmed that stimulus representations undergo a representational transformation-consistent with a flip-while transitioning from the functional status of UMI to PMI. Demixed (d)PCA of the same data identified two representational trajectories, one each within a UMI subspace and a PMI subspace, both undergoing a reversal of stimulus coding axes. dPCA of data from an EEG dataset also provided evidence for priority-based transformations of the representational code, albeit with some differences. This type of transformation could allow for retention of unprioritized information in WM while preventing it from interfering with concurrent behavior. The results from this initial exploration suggest that the algorithmic details of how this transformation is carried out by RNNs, versus by the human brain, may differ.

The Influence of Active Removal from Working Memory on Serial Dependence

Flexible control of the contents of working memory (WM) includes removing no-longer-relevant information. Although simply withdrawing attention offers a "passive" mechanism, empirical findings suggest that it is also possible to actively remove information from WM. In this Registered Report we tested evidence that the bias (serial dependence) that an item exerts on the subsequent trial will be opposite in sign-attraction vs. repulsion - depending on whether it was passively or actively removed, respectively. A repulsive bias would be consistent with a specific mechanism for active removal: a rapid adaptation-like modification of perceptual circuitry. In a preliminary study, trials of two types were administered in pairs, multi-item WM followed by 1-item delayed recall, and we evaluated serial dependence of the latter on items from the former. In the first trial of each pair, two memoranda were presented, then one was designated irrelevant, then a third memorandum was presented. The critical manipulation was whether the third item was presented at the same location as the now "irrelevant memory item" (IMI). Overlap between the two should prompt the active removal of the IMI, whereas nonoverlap might prompt just the withdrawal of attention. Whereas the IMI exerted the expected attractive bias on 1-item recall in the no-overlap condition, we found an (unexpected) repulsive bias in the overlap condition. Because repulsive biases have been attributed to the adaptation-like modification of perceptual circuitry, replication of this result in this Registered Report would provide independent evidence for this mechanism for active removal from WM. Interpretation of the Stage 2 results are complicated by the fact that the approved Registered Report, carried out online, generated data that failed to meet a basic sanity check, and were therefore uninterpretable. Consequently, a follow-up lab-based experiment using procedures similar to the Registered Report generated results consistent with the hypothesis of principal theoretical interest: The IMI in the overlap condition exerted a repulsive bias on the subsequent trial.

Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information

Working memory provides flexible storage of information in service of upcoming behavioral goals. Some models propose specific fixed loci and mechanisms for the storage of visual information in working memory, such as sustained spiking in parietal and prefrontal cortex during working memory maintenance. An alternative view is that information can be remembered in a flexible format that best suits current behavioral goals. For example, remembered visual information might be stored in sensory areas for easier comparison to future sensory inputs, or might be re-coded into a more abstract action-oriented format and stored in motor areas. Here, we tested this hypothesis using a visuo-spatial working memory task where the required behavioral response was either known or unknown during the memory delay period. Using functional magnetic resonance imaging (fMRI) and multivariate decoding, we found that there was less information about remembered spatial position in early visual and parietal regions when the required response was known versus unknown. Furthermore, a representation of the planned motor action emerged in primary somatosensory, primary motor, and premotor cortex during the same task condition where spatial information was reduced in early visual cortex. These results suggest that the neural networks supporting working memory can be strategically reconfigured depending on specific behavioral requirements during a canonical visual working memory paradigm.