The exploration-exploitation trade-off in a foraging task is affected by mood-related arousal and valence

A review on exploration-exploitation trade-off in psychiatric disorders

Balancing exploration and exploitation is a crucial aspect of adaptive decision-making, but psychiatric disorders can disrupt this balance in various ways, shedding light on their neurocognitive roots and guiding targeted interventions. In this systematic review, we aimed to delineate potential exploration-exploitation impairments across psychiatric disorders. Through a thorough search on PubMed, we identified forty-six relevant studies employing tasks probing exploration-exploitation balances, which we synthesized to reveal distinct patterns. These disorders are clustered into three categories: addictive patterns, emotional/cognitive disturbances, and neurological (neurodevelopmental and neurodegenerative) disorders. Our findings show that anxiety and mood disorders often enhance exploratory behaviors, while depression impact decision stability and reward sensitivity. In contrast, schizophrenia, OCD (Obsessive-Compulsive Disorder), and ADHD (Attention-Deficit/Hyperactivity Disorder) are characterized by excessive switching and difficulties in balancing exploration and exploitation, leading to impaired learning and adaptability. Additionally, disorders with addictive-like features disrupt optimal decision-making strategies by either heightening exploration or causing maladaptive persistence, thus skewing the balance away from effective decision-making. Individuals exhibiting addiction-like or compulsive behaviors often demonstrate imbalances in the explore-exploit trade-off, resulting in suboptimal decision-making characterized by reduced exploration, flawed foraging strategies, and impulsive or perseverative choices despite adverse outcomes. This suggests that such disorders may originate from dysfunctional foraging processes applied to decision-making. In sum, different patterns of exploration-exploitation balance in different disorders are crucial in understanding the difficulties in learning and decision making of neuropsychiatric disorders. This suggests that such disorders may stem from dysregulated decision-making processes, where uncertainty plays a central role. Dysfunctions in dopaminergic and noradrenergic pathways appear to disrupt the brain's representation of uncertainty, thereby altering exploratory behavior. In sum, the varying patterns of exploration-exploitation balance across different disorders are critical for understanding the challenges in learning and decision-making associated with neuropsychiatric conditions.

Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex, Anterior Insula, and the Dorsal Anterior Cingulate Cortex

Explore-exploit research faces challenges in generalizability due to a limited theoretical basis for exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions involve an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) finding activation in the dorsal lateral prefrontal cortex, anterior insula, and anterior cingulate cortex during exploration versus exploitation, which provides some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex and dorsal medial prefrontal cortex, suggesting that these brain regions do not engage in opponent processing. Furthermore, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation during exploration and exploitation decisions is generally more similar than it is different suggests that there remain significant challenges in characterizing explore-exploit decision making. Nonetheless, dorsal lateral prefrontal cortex, anterior insula, and dorsal anterior cingulate cortex activation differentiate explore and exploit decisions and identifying these responses can aid in targeted interventions aimed at manipulating these decisions.

Self-guided transitions and creative idea search: adaptively choosing flexibility versus persistence in divergent and convergent thinking tasks

Adaptivity allows individuals to flexibly execute cognitive control to meet dynamically changing task demands while adhering to task goals. Such adaptivity is crucial for navigating complex problem spaces such as creative problem-solving. Recent theoretical and empirical investigations of individuals' tendencies towards flexibility versus persistence have begun to address the questions of how adaptivity may be jointly shaped by general (across-situational) predispositions and by task requirements. However, such investigation is hampered by the lack of comparable ways to quantify trait-like tendencies across different task contexts. Using a Self-Guided Transitions paradigm, in which participants are allowed to autonomously choose whether to continue, to switch or to return to working on either of two concurrently presented problem-solving items, this preregistered study provides evidence for both clear within-individual consistency in the proclivity towards flexibility versus persistence, and adaptive modulation of flexibility versus persistence for tasks that predominantly call on divergent versus convergent idea search. Both shifting and dwelling were associated with the generation of more numerous and more original ideas on divergent-thinking tasks-underscoring the creative and ideational rewards to be found both by sometimes staying the course (persistence/exploitation) and sometimes choosing to shift our efforts in a different direction (flexibility/exploration).

Metacognitive confidence and affect - two sides of the same coin?

Decision confidence is a prototypical metacognitive representation that is thought to approximate the probability that a decision is correct. The perception of being correct has also been associated with affective valence such that being correct feels more positive and being mistaken more negative. This suggests that, similarly to confidence, affective valence reflects the probability that a decision is correct. However, both fields of research have seen very little interaction. Here, we test if affect, similarly to confidence reflects probability that a decision is correct in two perceptual decision-making experiments where we compare the relationships of theoretically relevant variables (e.g. evidence, accuracy, and expectancy) with both confidence and affect ratings. The findings indicate that confidence and affect ratings are similarly sensitive to changes in accuracy, evidence, and expectancy, indicating that both track the subjective probability that a decision is correct. We identify various mechanisms that can explain these results. We also envision future research for clarifying the role of cognitive and affective aspects of metacognition relying on deeper integration of the respective research fields.

Navigating Uncertainty: The Role of Mood and Confidence in Decision-Making Flexibility and Performance

Dealing with uncertainty is a pivotal skill for adaptive decision-making across various real-life contexts. Cognitive models suggest that individuals continuously update their knowledge based on past choices and outcomes. Traditionally, uncertainty has been linked to negative states such as fear and anxiety. Recent evidence, however, highlights that uncertainty can also evoke positive emotions, such as surprise, interest, excitement, and enthusiasm, depending on one's task expectations. Despite this, the interplay between mood, confidence, and learning remains underexplored. Some studies indicate that self-reported mood does not always align with confidence, as these constructs evolve on different timescales. We propose that mood influences confidence, thereby enhancing decision flexibility-defined as the ability to switch effectively between exploration and exploitation. This increased flexibility is expected to improve task performance by increasing accuracy. Our findings support this hypothesis, revealing that confidence modulates exploration/exploitation strategies and learning rates, while mood affects reward perception and confidence levels. These findings indicate that metacognition entails a dynamic balance between exploration and exploitation, integrating mood states with high-level cognitive processes.

Test-retest reliability of behavioral and computational measures of advice taking under volatility

The development of computational models for studying mental disorders is on the rise. However, their psychometric properties remain understudied, posing a risk of undermining their use in empirical research and clinical translation. Here we investigated test-retest reliability (with a 2-week interval) of a computational assay probing advice-taking under volatility with a Hierarchical Gaussian Filter (HGF) model. In a sample of 39 healthy participants, we found the computational measures to have largely poor reliability (intra-class correlation coefficient or ICC < 0.5), on par with the behavioral measures of task performance. Further analysis revealed that reliability was substantially impacted by intrinsic measurement noise (indicated by parameter recovery analysis) and to a smaller extent by practice effects. However, a large portion of within-subject variance remained unexplained and may be attributable to state-like fluctuations. Despite the poor test-retest reliability, we found the assay to have face validity at the group level. Overall, our work highlights that the different sources of variance affecting test-retest reliability need to be studied in greater detail. A better understanding of these sources would facilitate the design of more psychometrically sound assays, which would improve the quality of future research and increase the probability of clinical translation.

You are How You Eat: Foraging Behavior as a Potential Novel Marker of Rat Affective State

Effective and safe foraging requires animals to behave according to the expectations they have about the rewards, threats, and costs in their environment. Since these factors are thought to be reflected in the animals' affective states, we can use foraging behavior as a window into those states. In this study, rats completed a foraging task in which they had repeatedly to decide whether to continue to harvest a food source despite increasing time costs, or to forgo food to switch to a different food source. Rats completed this task across two experiments using manipulations designed to induce both positive and negative, and shorter- and longer- term changes in affective state: removal and return of enrichment (Experiment 1), implementation and reversal of an unpredictable housing treatment (Experiment 1), and delivery of rewards (tickling or sucrose) and punishers (air-puff or back-handling) immediately prior to testing (Experiment 2). In Experiment 1, rats completed fewer trials and were more prone to switching between troughs when housed in standard, compared to enriched, housing conditions. In Experiment 2, rats completed more trials following pre-test tickling compared to pre-test sucrose delivery. However, we also found that they were prone to disengaging from the task, suggesting they were really choosing between three options: 'harvest', 'switch', or 'not work'. This limits the straightforward interpretation of the results. At present, foraging behavior within the context of this task cannot reliably be used as an indicator of an affective state in animals.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-024-00242-4.

Locus coeruleus integrity is related to an exploitation-based decision-making bias in older adulthood

Optimal decision-making balances exploration for new information against exploitation of known rewards, a process mediated by the locus coeruleus and its norepinephrine projections. We predicted that an exploitation-bias that emerges in older adulthood would be associated with lower microstructural integrity of the locus coeruleus. Leveraging in vivo histological methods from quantitative MRI-magnetic transfer saturation-we provide evidence that older age is associated with lower locus coeruleus integrity. Critically, we demonstrate that an exploitation bias in older adulthood, assessed with a foraging task, is sensitive and specific to lower locus coeruleus integrity. Because the locus coeruleus is uniquely vulnerable to Alzheimer's disease pathology, our findings suggest that aging, and a presymptomatic trajectory of Alzheimer's related decline, may fundamentally alter decision-making abilities in later life.

Models need mechanisms, but not labels

The target article proposes a model involving the important but not well-investigated topics of curiosity and creativity. The model, however, falls short of providing convincing explanations of the basic mechanisms underlying these phenomena. We outline the importance of mechanistic thinking in dealing with the concepts outlined in this article specifically and within psychology and cognitive neuroscience in general.

Dynamic computational phenotyping of human cognition

Computational phenotyping has emerged as a powerful tool for characterizing individual variability across a variety of cognitive domains. An individual's computational phenotype is defined as a set of mechanistically interpretable parameters obtained from fitting computational models to behavioural data. However, the interpretation of these parameters hinges critically on their psychometric properties, which are rarely studied. To identify the sources governing the temporal variability of the computational phenotype, we carried out a 12-week longitudinal study using a battery of seven tasks that measure aspects of human learning, memory, perception and decision making. To examine the influence of state effects, each week, participants provided reports tracking their mood, habits and daily activities. We developed a dynamic computational phenotyping framework, which allowed us to tease apart the time-varying effects of practice and internal states such as affective valence and arousal. Our results show that many phenotype dimensions covary with practice and affective factors, indicating that what appears to be unreliability may reflect previously unmeasured structure. These results support a fundamentally dynamic understanding of cognitive variability within an individual.

Exploration versus exploitation decisions in the human brain: A systematic review of functional neuroimaging and neuropsychological studies

Thoughts and actions are often driven by a decision to either explore new avenues with unknown outcomes, or to exploit known options with predictable outcomes. Yet, the neural mechanisms underlying this exploration-exploitation trade-off in humans remain poorly understood. This is attributable to variability in the operationalization of exploration and exploitation as psychological constructs, as well as the heterogeneity of experimental protocols and paradigms used to study these choice behaviours. To address this gap, here we present a comprehensive review of the literature to investigate the neural basis of explore-exploit decision-making in humans. We first conducted a systematic review of functional magnetic resonance imaging (fMRI) studies of exploration-versus exploitation-based decision-making in healthy adult humans during foraging, reinforcement learning, and information search. Eleven fMRI studies met inclusion criterion for this review. Adopting a network neuroscience framework, synthesis of the findings across these studies revealed that exploration-based choice was associated with the engagement of attentional, control, and salience networks. In contrast, exploitation-based choice was associated with engagement of default network brain regions. We interpret these results in the context of a network architecture that supports the flexible switching between externally and internally directed cognitive processes, necessary for adaptive, goal-directed behaviour. To further investigate potential neural mechanisms underlying the exploration-exploitation trade-off we next surveyed studies involving neurodevelopmental, neuropsychological, and neuropsychiatric disorders, as well as lifespan development, and neurodegenerative diseases. We observed striking differences in patterns of explore-exploit decision-making across these populations, again suggesting that these two decision-making modes are supported by independent neural circuits. Taken together, our review highlights the need for precision-mapping of the neural circuitry and behavioural correlates associated with exploration and exploitation in humans. Characterizing exploration versus exploitation decision-making biases may offer a novel, trans-diagnostic approach to assessment, surveillance, and intervention for cognitive decline and dysfunction in normal development and clinical populations.

Individuals who see the good in the bad engage distinctive default network coordination during post-encoding rest

Focusing on the upside of negative events often promotes resilience. Yet, the underlying mechanisms that allow some people to spontaneously see the good in the bad remain unclear. The broaden-and-build theory of positive emotion has long suggested that positive affect, including positivity in the face of negative events, is linked to idiosyncratic thought patterns (i.e., atypical cognitive responses). Yet, evidence in support of this view has been limited, in part, due to difficulty in measuring idiosyncratic cognitive processes as they unfold. To overcome this barrier, we applied Inter-Subject Representational Similarity Analysis to test whether and how idiosyncratic neural responding supports positive reactions to negative experience. We found that idiosyncratic functional connectivity patterns in the brain's default network while resting after a negative experience predicts more positive descriptions of the event. This effect persisted when controlling for connectivity 1) before and during the negative experience, 2) before, during, and after a neutral experience, and 3) between other relevant brain regions (i.e., the limbic system). The relationship between idiosyncratic default network responding and positive affect was largely driven by functional connectivity patterns between the ventromedial prefrontal cortex and the rest of the default network and occurred relatively quickly during rest. We identified post-encoding rest as a key moment and the default network as a key brain system in which idiosyncratic responses correspond with seeing the good in the bad.

Prolonged Physiological Stress Is Associated With a Lower Rate of Exploratory Learning That Is Compounded by Depression

BACKGROUND

Stress is a major risk factor for depression, and both are associated with important changes in decision-making patterns. However, decades of research have only weakly connected physiological measurements of stress to the subjective experience of depression. Here, we examined the relationship between prolonged physiological stress, mood, and explore-exploit decision making in a population navigating a dynamic environment under stress: health care workers during the COVID-19 pandemic.

METHODS

We measured hair cortisol levels in health care workers who completed symptom surveys and performed an explore-exploit restless-bandit decision-making task; 32 participants were included in the final analysis. Hidden Markov and reinforcement learning models assessed task behavior.

RESULTS

Participants with higher hair cortisol exhibited less exploration (r = -0.36, p = .046). Higher cortisol levels predicted less learning during exploration (β = -0.42, false discovery rate [FDR]-corrected p [pFDR] = .022). Importantly, mood did not independently correlate with cortisol concentration, but rather explained additional variance (β = 0.46, pFDR = .022) and strengthened the relationship between higher cortisol and lower levels of exploratory learning (β = -0.47, pFDR = .022) in a joint model. These results were corroborated by a reinforcement learning model, which revealed less learning with higher hair cortisol and low mood (β = -0.67, pFDR = .002).

CONCLUSIONS

These results imply that prolonged physiological stress may limit learning from new information and lead to cognitive rigidity, potentially contributing to burnout. Decision-making measures link subjective mood states to measured physiological stress, suggesting that they should be incorporated into future biomarker studies of mood and stress conditions.

The costs and benefits of psychedelics on cognition and mood

Anecdotal evidence has indicated that psychedelic substances may acutely enhance creative task performance, although empirical support for this claim is mixed at best. Clinical research has shown that psychedelics might have enduring effects on mood and well-being. However, there is no neurocognitive framework that ties acute changes in cognition to long-term effects in mood. In this review, we operationalize creativity within an emerging cognitive control framework and assess the current empirical evidence of the effects of psychedelics on creativity. Next, we leverage insights about the mechanisms and computations by which other psychoactive drugs act to enhance versus impair cognition, in particular to those that act on catecholamines, the neurophysiological consequences of which are relatively well understood. Finally, we use the same framework to link the suggested psychedelic-induced improvements in creativity with enduring psychedelic-induced improvements in mood.

Meta-control: From psychology to computational neuroscience

Research in the past decades shed light on the different mechanisms that underlie our capacity for cognitive control. However, the meta-level processes that regulate cognitive control itself remain poorly understood. Following the terminology from artificial intelligence, meta-control can be defined as a collection of mechanisms that (a) monitor the progress of controlled processing and (b) regulate the underlying control parameters in the service of current task goals and in response to internal or external constraints. From a psychological perspective, meta-control is an important concept because it may help explain and predict how and when human agents select different types of behavioral strategies. From a cognitive neuroscience viewpoint, meta-control is a useful concept for understanding the complex networks in the prefrontal cortex that guide higher-level behavior as well as their interactions with neuromodulatory systems (such as the dopamine or norepinephrine system). The purpose of the special issue is to integrate hitherto segregated strands of research across three different perspectives: 1) a psychological perspective that specifies meta-control processes on a functional level and aims to operationalize them in experimental tasks; 2) a computational perspective that builds on ideas from artificial intelligence to formalize normative solutions to meta-control problems; and 3) a cognitive neuroscience perspective that identifies neural correlates of and mechanisms underlying meta-control.