Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma

Reward-based option competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space

Primates exploring and exploiting a continuous sensorimotor space rely on dynamic maps in the dorsal stream. Two complementary perspectives exist on how these maps encode rewards. Reinforcement learning models integrate rewards incrementally over time, efficiently resolving the exploration/exploitation dilemma. Working memory buffer models explain rapid plasticity of parietal maps but lack a plausible exploration/exploitation policy. The reinforcement learning model presented here unifies both accounts, enabling rapid, information-compressing map updates and efficient transition from exploration to exploitation. As predicted by our model, activity in human frontoparietal dorsal stream regions, but not in MT+, tracks the number of competing options, as preferred options are selectively maintained on the map, while spatiotemporally distant alternatives are compressed out. When valuable new options are uncovered, posterior β1/α oscillations desynchronize within 0.4 to 0.7 s, consistent with option encoding by competing β1-stabilized subpopulations. Together, outcomes matching locally cached reward representations rapidly update parietal maps, biasing choices toward often-sampled, rewarded options.

Effects of childhood maltreatment and major depressive disorder on functional connectivity in hippocampal subregions

Major Depressive Disorder (MDD) with childhood maltreatment is a prevalent clinical phenotype. Prior studies have observed abnormal hippocampal activity in MDD patients, considering the hippocampus as a single nucleus. However, there is limited research investigating the static and dynamic changes in hippocampal subregion functional connectivity (FC) in MDD patients with childhood maltreatment. Therefore, we employed static and dynamic FC analyses using hippocampal subregions, including the anterior hippocampus and posterior hippocampus, as seed regions to investigate the neurobiological alterations associated with MDD resulting from childhood maltreatment. This study involved four groups: MDD with (n = 48) and without childhood maltreatment (n = 30), as well as healthy controls with (n = 57) and without (n = 46) childhood maltreatment. Compared to MDD patients without childhood maltreatment, those with childhood maltreatment exhibit altered FC between the hippocampal subregion and multiple brain regions, including the anterior cingulate gyrus, superior frontal gyrus, putamen, calcarine gyrus, superior temporal gyrus, angular gyrus, and supplementary motor area. Additionally, dynamic FC between the right medial-2 hippocampal head and the right calcarine gyrus shows a positive correlation with childhood maltreatment across all its subtypes. Moreover, dFC between the right hippocampal tail and the left angular gyrus moderates the relationship between childhood maltreatment and the depression severity. Our findings of distinct FC patterns within hippocampal subregions provide new clues for understanding the neurobiological basis of MDD with childhood maltreatment.

Exploration versus exploitation decisions in the human brain: A systematic review of functional neuroimaging and neuropsychological studies

Thoughts and actions are often driven by a decision to either explore new avenues with unknown outcomes, or to exploit known options with predictable outcomes. Yet, the neural mechanisms underlying this exploration-exploitation trade-off in humans remain poorly understood. This is attributable to variability in the operationalization of exploration and exploitation as psychological constructs, as well as the heterogeneity of experimental protocols and paradigms used to study these choice behaviours. To address this gap, here we present a comprehensive review of the literature to investigate the neural basis of explore-exploit decision-making in humans. We first conducted a systematic review of functional magnetic resonance imaging (fMRI) studies of exploration-versus exploitation-based decision-making in healthy adult humans during foraging, reinforcement learning, and information search. Eleven fMRI studies met inclusion criterion for this review. Adopting a network neuroscience framework, synthesis of the findings across these studies revealed that exploration-based choice was associated with the engagement of attentional, control, and salience networks. In contrast, exploitation-based choice was associated with engagement of default network brain regions. We interpret these results in the context of a network architecture that supports the flexible switching between externally and internally directed cognitive processes, necessary for adaptive, goal-directed behaviour. To further investigate potential neural mechanisms underlying the exploration-exploitation trade-off we next surveyed studies involving neurodevelopmental, neuropsychological, and neuropsychiatric disorders, as well as lifespan development, and neurodegenerative diseases. We observed striking differences in patterns of explore-exploit decision-making across these populations, again suggesting that these two decision-making modes are supported by independent neural circuits. Taken together, our review highlights the need for precision-mapping of the neural circuitry and behavioural correlates associated with exploration and exploitation in humans. Characterizing exploration versus exploitation decision-making biases may offer a novel, trans-diagnostic approach to assessment, surveillance, and intervention for cognitive decline and dysfunction in normal development and clinical populations.

Maladaptive explore/exploit trade-offs in schizophrenia

Schizophrenia is a complex disorder that remains poorly understood, particularly at the systems level. In this opinion article we argue that the explore/exploit trade-off concept provides a holistic and ecologically valid framework to resolve some of the apparent paradoxes that have emerged within schizophrenia research. We review recent evidence suggesting that fundamental explore/exploit behaviors may be maladaptive in schizophrenia during physical, visual, and cognitive foraging. We also describe how theories from the broader optimal foraging literature, such as the marginal value theorem (MVT), could provide valuable insight into how aberrant processing of reward, context, and cost/effort evaluations interact to produce maladaptive responses.

From exploration to exploitation: a shifting mental mode in late life development

Changes in cognition, affect, and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control, and a shift toward affective goals. These are accompanied by changes in cortical networks, as well as attention and reward circuits. By incorporating these factors into a unified account, the exploration-to-exploitation shift offers an integrative model of cognitive, affective, and brain aging. Here, we review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

A New Paradigm for the Study of Cognitive Flexibility in Children and Adolescents: The "Virtual House Locomotor Maze" (VHLM)

Classical neuropsychological assessments are designed to explore cognitive brain functions using paper-and-pencil or digital tests. The purpose of this study was to design and to test a new protocol named the "Virtual House Locomotor Maze" (VHLM) for studying inhibitory control as well as mental flexibility using a visuo-spatial locomotor memory test. The VHLM is a simple maze including six houses using the technology of the Virtual Carpet Paradigm™. Ten typical development children (TD) were enrolled in this study. The participants were instructed to reach a target house as quickly as possible and to bear in mind the experimental instructions. We examined their planning and replanning abilities to take the shortest path to reach a target house. In order to study the cognitive processes during navigation, we implemented a spatio-temporal index based on the measure of kinematics behaviors (i.e., trajectories, tangential velocity and head direction). Replanning was tested by first repeating a path chosen by the subject to reach a given house. After learning this path, it was blocked imposing that the subject inhibited the learned trajectory and designed a new trajectory to reach the same house. We measured the latency of the departure after the presentation of each house and the initial direction of the trajectory. The results suggest that several strategies are used by the subjects for replanning and our measures could be used as an index of impulsivity.

Search for solutions, learning, simulation, and choice processes in suicidal behavior

Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person's capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value‐based choices. Can individual‐level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case–control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision‐making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood‐congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one's model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case–control studies using gambling tasks, armed bandits, and delay discounting support this claim. Future experimental studies will need to uncover real‐time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis.

This article is categorized under:

Economics > Individual Decision‐Making

Psychology > Emotion and Motivation

Psychology > Learning

Neuroscience > Behavior