Brain networks for exploration decisions utilizing distinct modeled information types during contextual learning

Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex, Anterior Insula, and the Dorsal Anterior Cingulate Cortex

Explore-exploit research faces challenges in generalizability due to a limited theoretical basis for exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions involve an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) finding activation in the dorsal lateral prefrontal cortex, anterior insula, and anterior cingulate cortex during exploration versus exploitation, which provides some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex and dorsal medial prefrontal cortex, suggesting that these brain regions do not engage in opponent processing. Furthermore, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation during exploration and exploitation decisions is generally more similar than it is different suggests that there remain significant challenges in characterizing explore-exploit decision making. Nonetheless, dorsal lateral prefrontal cortex, anterior insula, and dorsal anterior cingulate cortex activation differentiate explore and exploit decisions and identifying these responses can aid in targeted interventions aimed at manipulating these decisions.

Meta-Analysis Reveals That Explore-Exploit Decisions Are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex, Anterior Insula, and Dorsal Anterior Cingulate Cortex

Explore-exploit research faces challenges in generalizability due to a limited theoretical basis for exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions involve an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N = 23 studies) finding activation in the dorsal lateral prefrontal cortex, anterior insula, and dorsal anterior cingulate cortex during exploration versus exploitation, which provides some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the anterior cingulate cortex and dorsal medial prefrontal cortex, suggesting that these brain regions do not engage in opponent processing. Furthermore, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation during exploration and exploitation decisions is generally more similar than it is different suggests that there remain significant challenges in characterizing explore-exploit decision-making. Nonetheless, dorsal lateral prefrontal cortex, anterior insula, and dorsal anterior cingulate cortex activation differentiate explore and exploit decisions, and identifying these responses can aid in targeted interventions aimed at manipulating these decisions.

An fMRI study of finger movements in children with and without dyslexia

Introduction

Developmental dyslexia is a language-based reading disability, yet some have reported motor impairments, usually attributed to cerebellar dysfunction.

Methods

Using fMRI, we compared children with and without dyslexia during irregularly paced, left or right-hand finger tapping. Next, we examined seed-to-voxel intrinsic functional connectivity (iFC) using six seed regions of the motor system (left and right anterior lobe of the cerebellum, SM1 and SMA).

Results

A whole-brain task-evoked analysis revealed relatively less activation in the group with dyslexia in right anterior cerebellum during right hand tapping. For iFC, we found the group with dyslexia to have greater iFC between the right SM1 seed and a medial aspect of right postcentral gyrus for left hand tapping; and greater iFC between the left SM1 seed and left thalamus, as well as weaker local iFC around the left SM1 seed region for right hand tapping. Lastly, extracted activity and connectivity values that had been identified in these between-group comparisons were not correlated with measures of reading.

Discussion

We conclude that there are some aberrations in motor system function in children with dyslexia, but these are not tied to reading ability.

Working memory and active sampling of the environment: Medial temporal contributions

Working memory (WM) refers to the ability to maintain and actively process information-either derived from perception or long-term memory (LTM)-for intelligent thought and action. This chapter focuses on the contributions of the temporal lobe, particularly medial temporal lobe (MTL) to WM. First, neuropsychological evidence for the involvement of MTL in WM maintenance is reviewed, arguing for a crucial role in the case of retaining complex relational bindings between memorized features. Next, MTL contributions at the level of neural mechanisms are covered-with a focus on WM encoding and maintenance, including interactions with ventral temporal cortex. Among WM use processes, we focus on active sampling of environmental information, a key input source to capacity-limited WM. MTL contributions to the bidirectional relationship between active sampling and memory are highlighted-WM control of active sampling and sampling as a way of selecting input to WM. Memory-based sampling studies relying on scene and object inspection, visual-based exploration behavior (e.g., vicarious behavior), and memory-guided visual search are reviewed. The conclusion is that MTL serves an important function in the selection of information from perception and transfer from LTM to capacity-limited WM.

To learn or to gain: neural signatures of exploration in human decision-making

Individuals not only take actions to obtain immediate rewards but also to gain more information to guide future choices. An ideal exploration-exploitation balance is crucial for maximizing reward over the long run. However, the neural signatures of exploration in humans remain unclear. Using quantitative meta-analyses of functional magnetic resonance imaging experiments on exploratory behaviors, we sought to identify the concordant activity pertaining to exploration over a range of experiments. The results revealed that exploration activates concordant brain activity associated with risk (e.g., dorsal medial prefrontal cortex and anterior insula), cognitive control (e.g., dorsolateral prefrontal cortex and inferior frontal gyrus), and motor processing (e.g., premotor cortex). These stereotaxic maps of exploration may indicate that exploration is highly linked to risk processing, but is also specifically associated with regions involved in executive control processes. Although this explanation should be treated as exploratory, these findings support theories positing an important role for the prefrontal-insular-motor cortical network in exploration.

Competitive and cooperative interactions between medial temporal and striatal learning systems

The striatum and medial temporal lobes (MTL) exhibit dissociable roles during learning. Whereas the striatum and its network of thalamic relays and cortical nodes are necessary for nondeclarative learning, the MTL and associated network are required for declarative learning. Several studies have suggested that these networks are functionally competitive during learning. Since these discoveries, however, evidence has accumulated that they can operate in a cooperative fashion. In this review, we discuss evidence for both competition and cooperation between these systems during learning, with the aim of reconciling these seemingly contradictory findings. Examples of cooperation between the striatum and MTL have been provided, especially during consolidation and generalization of knowledge, and do not appear to be precluded by differences in functional specialization. However, whether these systems cooperate or compete does seem to depend on the phase of learning and cognitive or motor aspects of the task. The involvement of other regions, such as midbrain dopaminergic nuclei and the prefrontal cortex, may promote and mediate cooperation between the striatum and the MTL during learning. Building on this body of research, we propose a model for striatum-MTL interactions in learning and memory and attempt to predict, in general terms, when cooperation or competition will occur.

Pure correlates of exploration and exploitation in the human brain

Balancing exploration and exploitation is a fundamental problem in reinforcement learning. Previous neuroimaging studies of the exploration-exploitation dilemma could not completely disentangle these two processes, making it difficult to unambiguously identify their neural signatures. We overcome this problem using a task in which subjects can either observe (pure exploration) or bet (pure exploitation). Insula and dorsal anterior cingulate cortex showed significantly greater activity on observe trials compared to bet trials, suggesting that these regions play a role in driving exploration. A model-based analysis of task performance suggested that subjects chose to observe until a critical evidence threshold was reached. We observed a neural signature of this evidence accumulation process in the ventromedial prefrontal cortex. These findings support theories positing an important role for anterior cingulate cortex in exploration, while also providing a new perspective on the roles of insula and ventromedial prefrontal cortex.

Parsing the Role of the Hippocampus in Approach-Avoidance Conflict

The hippocampus plays a central role in the approach-avoidance conflict that is central to the genesis of anxiety. However, its exact functional contribution has yet to be identified. We designed a novel gambling task that generated approach-avoidance conflict while controlling for spatial processing. We fit subjects' behavior using a model that quantified the subjective values of choice options, and recorded neural signals using functional magnetic resonance imaging (fMRI). Distinct functional signals were observed in anterior hippocampus, with inferior hippocampus selectively recruited when subjects rejected a gamble, to a degree that covaried with individual differences in anxiety. The superior anterior hippocampus, in contrast, uniquely demonstrated value signals that were potentiated in the context of approach-avoidance conflict. These results implicate the anterior hippocampus in behavioral avoidance and choice monitoring, in a manner relevant to understanding its role in anxiety. Our findings highlight interactions between subregions of the hippocampus as an important focus for future study.

Dynamic Hippocampal and Prefrontal Contributions to Memory Processes and Representations Blur the Boundaries of Traditional Cognitive Domains

The hippocampus has long been known to be a critical component of the memory system involved in the formation and use of long-term declarative memory. However, recent findings have revealed that the reach of hippocampal contributions extends to a variety of domains and tasks that require the flexible use of cognitive and social behavior, including domains traditionally linked to prefrontal cortex (PFC), such as decision-making. In addition, the prefrontal cortex (PFC) has gained traction as a necessary part of the memory system. These findings challenge the conventional characterizations of hippocampus and PFC as being circumscribed to traditional cognitive domains. Here, we emphasize that the ability to parsimoniously account for the breadth of hippocampal and PFC contributions to behavior, in terms of memory function and beyond, requires theoretical advances in our understanding of their characteristic processing features and mental representations. Notably, several literatures exist that touch upon this issue, but have remained disjointed because of methodological differences that necessarily limit the scope of inquiry, as well as the somewhat artificial boundaries that have been historically imposed between domains of cognition. In particular, this article focuses on the contribution of relational memory theory as an example of a framework that describes both the representations and processes supported by the hippocampus, and further elucidates the role of the hippocampal–PFC network to a variety of behaviors.

A Closer Look at the Hippocampus and Memory

Current interpretations of hippocampal memory function are blind to the fact that viewing behaviors are pervasive and complicate the relationships among perception, behavior, memory, and brain activity. For example, hippocampal activity and associative memory demands increase with stimulus complexity. Stimulus complexity also strongly modulates viewing. Associative processing and viewing thus are often confounded, rendering interpretation of hippocampal activity ambiguous. Similar considerations challenge many accounts of hippocampal function. To explain relationships between memory and viewing, we propose that the hippocampus supports the online memory demands necessary to guide visual exploration. The hippocampus thus orchestrates memory-guided exploration that unfolds over time to build coherent memories. This new perspective on hippocampal function harmonizes with the fact that memory formation and exploratory viewing are tightly intertwined.

Distinct Hippocampal versus Frontoparietal Network Contributions to Retrieval and Memory-guided Exploration

Memory can profoundly influence new learning, presumably because memory optimizes exploration of to-be-learned material. Although hippocampus and frontoparietal networks have been implicated in memory-guided exploration, their specific and interactive roles have not been identified. We examined eye movements during fMRI scanning to identify neural correlates of the influences of memory retrieval on exploration and learning. After retrieval of one object in a multiobject array, viewing was strategically directed away from the retrieved object toward nonretrieved objects, such that exploration was directed toward to-be-learned content. Retrieved objects later served as optimal reminder cues, indicating that exploration caused memory to become structured around the retrieved content. Hippocampal activity was associated with memory retrieval, whereas frontoparietal activity varied with strategic viewing patterns deployed after retrieval, thus providing spatiotemporal dissociation of memory retrieval from memory-guided learning strategies. Time-lagged fMRI connectivity analyses indicated that hippocampal activity predicted frontoparietal activity to a greater extent for a condition in which retrieval guided exploration occurred than for a passive control condition in which exploration was not influenced by retrieval. This demonstrates network-level interaction effects specific to influences of memory on strategic exploration. These findings show how memory guides behavior during learning and demonstrate distinct yet interactive hippocampal-frontoparietal roles in implementing strategic exploration behaviors that determine the fate of evolving memory representations.

Ketamine Treatment and Global Brain Connectivity in Major Depression

Capitalizing on recent advances in resting-state functional connectivity magnetic resonance imaging (rs-fcMRI) and the distinctive paradigm of rapid mood normalization following ketamine treatment, the current study investigated intrinsic brain networks in major depressive disorder (MDD) during a depressive episode and following treatment with ketamine. Medication-free patients with MDD and healthy control subjects (HC) completed baseline rs-fcMRI. MDD patients received a single infusion of ketamine and underwent repeated rs-fcMRI at 24 h posttreatment. Global brain connectivity with global signal regression (GBCr) values were computed as the average of correlations of each voxel with all other gray matter voxels in the brain. MDD group showed reduced GBCr in the prefrontal cortex (PFC) but increased GBCr in the posterior cingulate, precuneus, lingual gyrus, and cerebellum. Ketamine significantly increased GBCr in the PFC and reduced GBCr in the cerebellum. At baseline, 2174 voxels of altered GBCr were identified, but only 310 voxels significantly differed relative to controls following treatment (corrected α<0.05). Responders to ketamine showed increased GBCr in the lateral PFC, caudate, and insula. Follow-up seed-based analyses illustrated a pattern of dysconnectivity between the PFC/subcortex and the rest of the brain in MDD, which appeared to normalize postketamine. The extent of the functional dysconnectivity identified in MDD and the swift and robust normalization following treatment suggest that GBCr may serve as a treatment response biomarker for the development of rapid acting antidepressants. The data also identified unique prefrontal and striatal circuitry as a putative marker of successful treatment and a target for antidepressants' development.

Hippocampal-cortical contributions to strategic exploration during perceptual discrimination

The hippocampus is crucial for long-term memory; its involvement in short-term or immediate expressions of memory is more controversial. Rodent hippocampus has been implicated in an expression of memory that occurs on-line during exploration termed "vicarious trial-and-error" (VTE) behavior. VTE occurs when rodents iteratively explore options during perceptual discrimination or at choice points. It is strategic in that it accelerates learning and improves later memory. VTE has been associated with activity of rodent hippocampal neurons, and lesions of hippocampus disrupt VTE and associated learning and memory advantages. Analogous findings of VTE in humans would support the role of hippocampus in active use of short-term memory to guide strategic behavior. We therefore measured VTE using eye-movement tracking during perceptual discrimination and identified relevant neural correlates with functional magnetic resonance imaging. A difficult perceptual-discrimination task was used that required visual information to be maintained during a several second trial, but with no long-term memory component. VTE accelerated discrimination. Neural correlates of VTE included robust activity of hippocampus and activity of a network of medial prefrontal and lateral parietal regions involved in memory-guided behavior. This VTE-related activity was distinct from activity associated with simply viewing visual stimuli and making eye movements during the discrimination task, which occurred in regions frequently associated with visual processing and eye-movement control. Subjects were mostly unaware of performing VTE, thus further distancing VTE from explicit long-term memory processing. These findings bridge the rodent and human literatures on neural substrates of memory-guided behavior, and provide further support for the role of hippocampus and a hippocampal-centered network of cortical regions in the immediate use of memory in on-line processing and the guidance of behavior.

Neural and neurochemical basis of reinforcement-guided decision making

Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcement-guided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decision-related information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making.

Associative Recognition Memory Awareness Improved by Theta-Burst Stimulation of Frontopolar Cortex

Neuroimaging and lesion studies have implicated specific prefrontal cortex locations in subjective memory awareness. Based on this evidence, a rostrocaudal organization has been proposed whereby increasingly anterior prefrontal regions are increasingly involved in memory awareness. We used theta-burst transcranial magnetic stimulation (TBS) to temporarily modulate dorsolateral versus frontopolar prefrontal cortex to test for distinct causal roles in memory awareness. In three sessions, participants received TBS bilaterally to frontopolar cortex, dorsolateral prefrontal cortex, or a control location prior to performing an associative-recognition task involving judgments of memory awareness. Objective memory performance (i.e., accuracy) did not differ based on stimulation location. In contrast, frontopolar stimulation significantly influenced several measures of memory awareness. During study, judgments of learning were more accurate such that lower ratings were given to items that were subsequently forgotten selectively following frontopolar TBS. Confidence ratings during test were also higher for correct trials following frontopolar TBS. Finally, trial-by-trial correspondence between overt performance and subjective awareness during study demonstrated a linear increase across control, dorsolateral, and frontopolar TBS locations, supporting a rostrocaudal hierarchy of prefrontal contributions to memory awareness. These findings indicate that frontopolar cortex contributes causally to memory awareness, which was improved selectively by anatomically targeted TBS.

Adolescent development of context-dependent stimulus-reward association memory and its neural correlates

Expression of learned stimulus-reward associations based on context is essential for regulation of behavior to meet situational demands. Contextual regulation improves during development, although the developmental progression of relevant neural and cognitive processes is not fully specified. We therefore measured neural correlates of flexible, contextual expression of stimulus-reward associations in pre/early-adolescent children (ages 9-13 years) and young adults (ages 19-22 years). After reinforcement learning using standard parameters, a contextual reversal manipulation was used whereby contextual cues indicated that stimulus-reward associations were the same as previously reinforced for some trials (consistent trials) or were reversed on other trials (inconsistent trials). Subjects were thus required to respond according to original stimulus-reward associations vs. reversed associations based on trial-specific contextual cues. Children and young adults did not differ in reinforcement learning or in relevant functional magnetic resonance imaging (fMRI) correlates. In contrast, adults outperformed children during contextual reversal, with better performance specifically for inconsistent trials. fMRI signals corresponding to this selective advantage included greater activity in lateral prefrontal cortex (LPFC), hippocampus, and dorsal striatum for young adults relative to children. Flexible expression of stimulus-reward associations based on context thus improves via adolescent development, as does recruitment of brain regions involved in reward learning and contextual expression of memory. HighlightsEarly-adolescent children and young adults were equivalent in reinforcement learning.Adults outperformed children in contextual expression of stimulus-reward associations.Adult advantages correlated with increased activity of relevant brain regions.Specific neurocognitive developmental changes support better contextual regulation.

Age-related striatal BOLD changes without changes in behavioral loss aversion

Loss aversion (LA), the idea that negative valuations have a higher psychological impact than positive ones, is considered an important variable in consumer research. The literature on aging and behavior suggests older individuals may show more LA, although it is not clear if this is an effect of aging in general (as in the continuum from age 20 and 50 years), or of the state of older age (e.g., past age 65 years). We also have not yet identified the potential biological effects of aging on the neural processing of LA. In the current study we used a cohort of subjects with a 30 year range of ages, and performed whole brain functional MRI (fMRI) to examine the ventral striatum/nucleus accumbens (VS/NAc) response during a passive viewing of affective faces with model-based fMRI analysis incorporating behavioral data from a validated approach/avoidance task with the same stimuli. Our a priori focus on the VS/NAc was based on (1) the VS/NAc being a central region for reward/aversion processing; (2) its activation to both positive and negative stimuli; (3) its reported involvement with tracking LA. LA from approach/avoidance to affective faces showed excellent fidelity to published measures of LA. Imaging results were then compared to the behavioral measure of LA using the same affective faces. Although there was no relationship between age and LA, we observed increasing neural differential sensitivity (NDS) of the VS/NAc to avoidance responses (negative valuations) relative to approach responses (positive valuations) with increasing age. These findings suggest that a central region for reward/aversion processing changes with age, and may require more activation to produce the same LA behavior as in younger individuals, consistent with the idea of neural efficiency observed with high IQ individuals showing less brain activation to complete the same task.