A Formal Valuation Framework for Emotions and Their Control

The computational psychopathology of emotion

Mood and anxiety disorders involve recurring, maladaptive patterns of distinct emotions and moods. Here, we argue that understanding these maladaptive patterns first requires understanding how emotions and moods guide adaptive behavior. We thus review recent progress in computational accounts of emotion that aims to explain the adaptive role of distinct emotions and mood. We then highlight how this emerging approach could be used to explain maladaptive emotions in various psychopathologies. In particular, we identify three computational factors that may be responsible for excessive emotions and moods of different types: self-intensifying affective biases, misestimations of predictability, and misestimations of controllability. Finally, we outline how the psychopathological roles of these factors can be tested, and how they may be used to improve psychotherapeutic and psychopharmacological interventions.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Personality (Dys)Function and General Instability

Humans adapt to a dynamic environment while maintaining psychological equilibrium. Systems theories of personality hold that generalized processes control stability by regulating how strongly a person reacts to various situations. Research shows there are higher-order traits of general personality function (Stability) and dysfunction (general personality pathology; GPP), but whether or not they capture individual differences in reactivity is largely theoretical. We tested this hypothesis by examining how general personality functioning manifests in everyday life in two samples (Ns=205; 342 participants and 24,920; 17,761 observations) that completed an ambulatory assessment protocol. Consistent with systems theories, we found (1) there is a general factor reflecting reactivity across major domains of functioning, and (2) reactivity is strongly associated with Stability and GPP. Results provide insight into how people fundamentally adapt (or not) to their environments, and lays the foundation for more practical, empirical models of human functioning.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

Model-based learning retrospectively updates model-free values

Reinforcement learning (RL) is widely regarded as divisible into two distinct computational strategies. Model-free learning is a simple RL process in which a value is associated with actions, whereas model-based learning relies on the formation of internal models of the environment to maximise reward. Recently, theoretical and animal work has suggested that such models might be used to train model-free behaviour, reducing the burden of costly forward planning. Here we devised a way to probe this possibility in human behaviour. We adapted a two-stage decision task and found evidence that model-based processes at the time of learning can alter model-free valuation in healthy individuals. We asked people to rate subjective value of an irrelevant feature that was seen at the time a model-based decision would have been made. These irrelevant feature value ratings were updated by rewards, but in a way that accounted for whether the selected action retrospectively ought to have been taken. This model-based influence on model-free value ratings was best accounted for by a reward prediction error that was calculated relative to the decision path that would most likely have led to the reward. This effect occurred independently of attention and was not present when participants were not explicitly told about the structure of the environment. These findings suggest that current conceptions of model-based and model-free learning require updating in favour of a more integrated approach. Our task provides an empirical handle for further study of the dialogue between these two learning systems in the future.

A Computational View on the Nature of Reward and Value in Anhedonia

Anhedonia - a common feature of depression and other neuropsychiatric disorders - encompasses a reduction in the subjective experience and anticipation of rewarding events, and a reduction in the motivation to seek out such events. The presence of anhedonia often predicts or accompanies treatment resistance, and as such better interventions and treatments are important. Yet the mechanisms giving rise to anhedonia are not well understood. In this chapter, we briefly review existing computational conceptualisations of anhedonia. We argue that they are mostly descriptive and fail to provide an explanatory account of why anhedonia may occur. Working within the framework of reinforcement learning, we examine two potential computational mechanisms that could give rise to anhedonic phenomena. First, we show how anhedonia can arise in multi-dimensional drive-reduction settings through a trade-off between different rewards or needs. We then generalise this in terms of model-based value inference and identify a key role for associational belief structure. We close with a brief discussion of treatment implications of both of these conceptualisations. In summary, computational accounts of anhedonia have provided a useful descriptive framework. Recent advances in reinforcement learning suggest promising avenues by which the mechanisms underlying anhedonia may be teased apart, potentially motivating novel approaches to treatment.

Value-based cognition and drug dependency

Value-based decision-making is thought to play an important role in drug dependency. Achieving elevated levels of euphoria or ameliorating dysphoria/pain may motivate goal-directed drug consumption in both drug-naïve and long-time users. In other words, drugs become viewed as the preferred means of attaining a desired internal state. The bias towards choosing drugs may affect one's cognition. Observed biases in learning, attention and memory systems within the brain gradually focus one's cognitive functions towards drugs and related cues to the exclusion of other stimuli. In this narrative review, the effects of drug use on learning, attention and memory are discussed with a particular focus on changes across brain-wide functional networks and the subsequent impact on behaviour. These cognitive changes are then incorporated into the cycle of addiction, an established model outlining the transition from casual drug use to chronic dependency. If drug use results in the elevated salience of drugs and their cues, the studies highlighted in this review strongly suggest that this salience biases cognitive systems towards the motivated pursuit of addictive drugs. This bias is observed throughout the cycle of addiction, possibly contributing to the persistent hold that addictive drugs have over the dependent. Taken together, the excessive valuation of drugs as the preferred means of achieving a desired internal state affects more than just decision-making, but also learning, attentional and mnemonic systems. This eventually narrows the focus of one's thoughts towards the pursuit and consumption of addictive drugs.

Linear reinforcement learning in planning, grid fields, and cognitive control

It is thought that the brain’s judicious reuse of previous computation underlies our ability to plan flexibly, but also that inappropriate reuse gives rise to inflexibilities like habits and compulsion. Yet we lack a complete, realistic account of either. Building on control engineering, here we introduce a model for decision making in the brain that reuses a temporally abstracted map of future events to enable biologically-realistic, flexible choice at the expense of specific, quantifiable biases. It replaces the classic nonlinear, model-based optimization with a linear approximation that softly maximizes around (and is weakly biased toward) a default policy. This solution demonstrates connections between seemingly disparate phenomena across behavioral neuroscience, notably flexible replanning with biases and cognitive control. It also provides insight into how the brain can represent maps of long-distance contingencies stably and componentially, as in entorhinal response fields, and exploit them to guide choice even under changing goals.

Models of decision making have so far been unable to account for how humans’ choices can be flexible yet efficient. Here the authors present a linear reinforcement learning model which explains both flexibility, and rare limitations such as habits, as arising from efficient approximate computation

Search for solutions, learning, simulation, and choice processes in suicidal behavior

Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person's capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value‐based choices. Can individual‐level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case–control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision‐making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood‐congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one's model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case–control studies using gambling tasks, armed bandits, and delay discounting support this claim. Future experimental studies will need to uncover real‐time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis.

This article is categorized under:

Economics > Individual Decision‐Making

Psychology > Emotion and Motivation

Psychology > Learning

Neuroscience > Behavior

Value, drug addiction and the brain

Over the years, various models have been proposed to explain the psychology and biology of drug addiction, built primarily around the habit and compulsion models. Recent research indicates drug addiction may be goal-directed, motivated by excessive valuation of drugs. Drug consumption may initially occur for the sake of pleasure but may transition to a means of escaping withdrawal, stress and negative emotions. In this hypothetical paper, we propose a value-based neurobiological model for drug addiction. We posit that during dependency, the value-based decision-making system in the brain is not inactive but has instead prioritized drugs as the reward of choice. In support of this model, we consider the role of valuation in choice, its influence on pleasure and punishment, and how valuation is contrasted in impulsive and compulsive behaviours. We then discuss the neurobiology of value, beginning with the dopaminergic system and its relationship with incentive salience before moving to brain-wide networks involved in valuation, control and prospection. These value-based neurobiological components are then integrated into the cycle of addiction as we consider the development of drug dependency from a valuation perspective. We conclude with a discussion of cognitive interventions utilizing value-based decision-making, highlighting not just advances in recalibrating the valuation system to focus on non-drug rewards, but also areas for improvement in refining this approach.

Abnormal reward valuation and event-related connectivity in unmedicated major depressive disorder

BACKGROUND

Experience of emotion is closely linked to valuation. Mood can be viewed as a bias to experience positive or negative emotions and abnormally biased subjective reward valuation and cognitions are core characteristics of major depression.

METHODS

Thirty-four unmedicated subjects with major depressive disorder and controls estimated the probability that fractal stimuli were associated with reward, based on passive observations, so they could subsequently choose the higher of either their estimated fractal value or an explicitly presented reward probability. Using model-based functional magnetic resonance imaging, we estimated each subject's internal value estimation, with psychophysiological interaction analysis used to examine event-related connectivity, testing hypotheses of abnormal reward valuation and cingulate connectivity in depression.

RESULTS

Reward value encoding in the hippocampus and rostral anterior cingulate was abnormal in depression. In addition, abnormal decision-making in depression was associated with increased anterior mid-cingulate activity and a signal in this region encoded the difference between the values of the two options. This localised decision-making and its impairment to the anterior mid-cingulate cortex (aMCC) consistent with theories of cognitive control. Notably, subjects with depression had significantly decreased event-related connectivity between the aMCC and rostral cingulate regions during decision-making, implying impaired communication between the neural substrates of expected value estimation and decision-making in depression.

CONCLUSIONS

Our findings support the theory that abnormal neural reward valuation plays a central role in major depressive disorder (MDD). To the extent that emotion reflects valuation, abnormal valuation could explain abnormal emotional experience in MDD, reflect a core pathophysiological process and be a target of treatment.

Advances in the computational understanding of mental illness

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

Opportunities for emotion and mental health research in the resource-rationality framework

We discuss opportunities in applying the resource-rationality framework toward answering questions in emotion and mental health research. These opportunities rely on characterization of individual differences in cognitive strategies; an endeavor that may be at odds with the normative approach outlined in the target article. We consider ways individual differences might enter the framework and the translational opportunities offered by each.

Neurocognitive Adaptation and Mental Health Vulnerability Following Maltreatment: The Role of Social Functioning

Childhood maltreatment is associated with a lifetime increase in risk of mental health disorder. We propose that such vulnerability may stem in large part from altered patterns of social functioning. Here, we highlight key findings from the psychological and epidemiological literature indicating that early maltreatment experience compromises social functioning and attenuates social support in ways that increase mental health vulnerability. We then review the extant neuroimaging studies of children and adolescents, focusing on three domains implicated in social functioning: threat processing, reward processing, and emotion regulation. We discuss how adaptations in these domains may increase latent vulnerability to mental health problems by impacting on social functioning via increased stress susceptibility as well as increased stress generation. Finally, we explore how computational psychiatry approaches, alongside systematically reported measures of social functioning, can complement studies of neural function in the creation of a mechanistic framework aimed at informing approaches to prevention and intervention.

The posited effect of positive affect in anorexia nervosa: Advocating for a forgotten piece of a puzzling disease

Anorexia nervosa (AN) is a complex and life-threatening eating disorder. Current models of AN onset and maintenance have largely focused on the role of negative affect, while fewer models have described the role of positive affect (PA). Given that these theoretical models have informed current treatment approaches, and that treatment remains minimally effective for adults with AN, we advocate that targeting PA is one avenue for advancing maintenance models and by extension, treatment. We specifically propose that AN may arise and be chronically and pervasively maintained as a function of dysregulated PA in response to weight loss and weight loss behaviors (e.g., restriction, excessive exercise), to a degree that is not accounted for in existing models of AN. We present evidence from multiple domains, including biological, behavioral, and self-report, supporting the hypothesis that PA dysregulation in AN contributes to the maintenance of the disorder. We conclude with several specific avenues for treatment development research as well as a call for future work elucidating the biological correlates of PA.

A Pavlovian account for paradoxical effects of motivation on controlling response vigour

In high stakes situations, people sometimes choke under pressure, performing below their abilities. Here, we suggest a novel mechanism to account for this paradoxical effect of motivation: the automatic adjustment of action vigour to potential reward. Although adaptive on average, this mechanism may impede fine motor control. Such detrimental effect was observed in three studies (n = 74 in total), using behavioural tasks where payoff depended on the precision of handgrip squeezing or golf putting. Participants produced more force for higher incentives, which aggravated their systematic overshooting of low-force targets. This reward bias was specific to action vigour, as reward did not alter action timing, direction or variability across trials. Although participants could report their reward bias, they somehow failed to limit their produced force. Such an automatic link between incentive and force level might correspond to a Pavlovian response that is counterproductive when action vigour is not instrumental for maximizing reward.

Interpersonal dysfunction in borderline personality: a decision neuroscience perspective

Borderline personality disorder (BPD) is characterized by disadvantageous decisions that are often expressed in close relationships and associated with intense negative emotions. Although functional neuroimaging studies of BPD have described regions associated with altered social cognition and emotion processing, these correlates do not inform an understanding of how brain activity leads to maladaptive choices. Drawing on recent research, we argue that formal models of decision-making are crucial to elaborating theories of BPD that bridge psychological constructs, behavior, and neural systems. We propose that maladaptive interactions between Pavlovian and instrumental influences play a crucial role in the expression of interpersonal problems. Finally, we articulate specific hypotheses about how clinical features of BPD may map onto neural systems that implement separable decision processes.