Lateral prefrontal model-based signatures are reduced in healthy individuals with high trait impulsivity

Dorsal-Ventral Reinforcement Learning Network Connectivity and Incentive-Driven Changes in Exploration

Probabilistic reinforcement learning (RL) tasks assay how individuals make decisions under uncertainty. The use of internal models (model-based) or direct learning from experiences (model-free), and the degree of choice stochasticity across alternatives (i.e., exploration), can all be influenced by the state space of the decision-making task. There is considerable individual variation in the balance between model-based and model-free control during decision-making, and this balance is affected by incentive motivation. The effect of variable reward incentives on the arbitration between model-based and model-free learning remains understudied, and individual differences in neural signatures and cognitive traits that moderate the effect of reward on model-free/model-based control are unknown. Here we combined a two-stage decision-making task utilizing differing reward incentives with computational modeling, neuropsychological tests, and neuroimaging to address these questions. Results showed the prospect of greater reward decreased exploration of alternative options and increased the balance toward model-based learning. These behavioral effects were replicated across two independent datasets including both sexes. Individual differences in processing speed and analytical thinking style affected how reward altered the dependence on both systems. Using a systems neuroscience-inspired approach to resting-state functional connectivity, we found reduced exploration of the options during the first stage of our task under high relative to low incentives was predicted by increased cross-network coupling between ventral and dorsal RL circuitry. These findings suggest that integrity of functional connections between stimulus valuation (ventral) and action valuation (dorsal) RL networks is associated with changes in the balance between explore-exploit decisions under changing reward incentives.

On the effects of impulsivity and compulsivity on neural correlates of model-based performance

Impaired goal-directed behavior is associated with a range of mental disorders, implicating underlying transdiagnostic factors. While compulsivity has been linked to reduced model-based (MB) control, impulsivity has rarely been studied in the context of reinforcement learning despite its links to reward processing and cognitive control. This study investigated the neural mechanisms underlying MB control and the influence of impulsivity and compulsivity, using EEG data from 238 individuals during a two-step decision making task. Single-trial analyses revealed a modulation of the feedback-related negativity (FRN), where amplitudes were higher after common transitions and positive reward prediction error (RPE), indicating a valence effect. Meanwhile, enhanced P3 amplitudes after rare transitions and both positive and negative RPE possibly reflect surprise. In a second step, we regressed the mean b values of the effect of RPE on the EEG signals onto self-reported impulsivity and compulsivity and behavioral MB control (w). The effect of RPE on FRN-related activity was mainly associated with higher w scores, linking the FRN to MB control. Crucially, the modulation of the P3 by RPE was negatively associated with compulsivity, pointing to a deficient mental model in highly compulsive individuals.

Heuristics in risky decision-making relate to preferential representation of information

When making choices, individuals differ from one another, as well as from normativity, in how they weigh different types of information. One explanation for this relates to idiosyncratic preferences in what information individuals represent when evaluating choice options. Here, we test this explanation with a simple risky-decision making task, combined with magnetoencephalography (MEG). We examine the relationship between individual differences in behavioral markers of information weighting and neural representation of stimuli pertinent to incorporating that information. We find that the extent to which individuals (N = 19) behaviorally weight probability versus reward information is related to how preferentially they neurally represent stimuli most informative for making probability and reward comparisons. These results are further validated in an additional behavioral experiment (N = 88) that measures stimulus representation as the latency of perceptual detection following priming. Overall, the results suggest that differences in the information individuals consider during choice relate to their risk-taking tendencies.

Erotic cue exposure increases physiological arousal, biases choices toward immediate rewards, and attenuates model-based reinforcement learning

Computational psychiatry focuses on identifying core cognitive processes that appear altered across distinct psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting may be at least partly under contextual control. Highly arousing cues were shown to increase discounting, although evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues remains unclear. Here, we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n = 39 healthy heterosexual male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic versus neutral cues both on the subjective and autonomic level. Erotic cue exposure increased discounting as reflected by more impatient choices. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation toward immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning in a heterosexual male sample. This highlights how environmental cues can impact core human decision processes and reveal that comprehensive modeling approaches can yield novel insights in reward-based decision processes.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Insula Connectivity Abnormalities Predict Impulsivity in Chronic Heroin Use Disorder: A Cross-Sectional Resting-State fMRI Study

Patients with heroin use disorder (HUD) often exhibit trait impulsivity, which may be an important factor in and a good predictor of addiction. However, the factor structure of HUD trait impulsivity (motor, attentional, and nonplanning) and its neural correlates are not yet known. A total of 24 male volunteers with HUD and 16 healthy control volunteers were recruited for this cross-sectional study. The Barratt Impulsiveness Scale (BIS-11) and resting-state functional magnetic resonance imaging (rs-fMRI) were employed using the insula as a seed point in an effort to understand the association between trait impulsivity and its intrinsic factors and functional connectivity (FC) between the insula and the whole brain. The HUD group in this study exhibited higher total trait impulsivity scores, motor impulsivity, and nonplanning impulsivity than the control group. Changes in FC between the right insula and the lateral occipital cortex and the right angular gyrus were significantly positively correlated with total trait impulsivity scores, motor impulsivity, and nonplanning impulsivity, whereas changes in the FC between the left insula and the left superior frontal gyrus and left frontopolar brain region were significantly negatively correlated with trait impulsivity. Thus, the insula may serve as an important biomarker for identifying trait impulsivity and its intrinsic factor structure in patients with HUDs.

Cortical Grey Matter Mediates Increases in Model-Based Control and Learning from Positive Feedback from Adolescence to Adulthood

Cognition and brain structure undergo significant maturation from adolescence into adulthood. Model-based (MB) control is known to increase across development, which is mediated by cognitive abilities. Here, we asked two questions unaddressed in previous developmental studies. First, what are the brain structural correlates of age-related increases in MB control? Second, how are age-related increases in MB control from adolescence to adulthood influenced by motivational context? A human developmental sample (n = 103; age, 12-50, male/female, 55:48) completed structural MRI and an established task to capture MB control. The task was modified with respect to outcome valence by including (1) reward and punishment blocks to manipulate the motivational context and (2) an additional choice test to assess learning from positive versus negative feedback. After replicating that an age-dependent increase in MB control is mediated by cognitive abilities, we demonstrate first-time evidence that gray matter density (GMD) in the parietal cortex mediates the increase of MB control with age. Although motivational context did not relate to age-related changes in MB control, learning from positive feedback improved with age. Meanwhile, negative feedback learning showed no age effects. We present a first report that an age-related increase in positive feedback learning was mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. Our findings indicate that brain maturation, putatively reflected in lower GMD, in distinct and partially overlapping brain regions could lead to a more efficient brain organization and might thus be a key developmental step toward age-related increases in planning and value-based choice.SIGNIFICANCE STATEMENT Changes in model-based decision-making are paralleled by extensive maturation in cognition and brain structure across development. Still, to date the neuroanatomical underpinnings of these changes remain unclear. Here, we demonstrate for the first time that parietal GMD mediates age-dependent increases in model-based control. Age-related increases in positive feedback learning were mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. A manipulation of motivational context did not have an impact on age-related changes in model-based control. These findings highlight that brain maturation in distinct and overlapping cortical regions constitutes a key developmental step toward improved value-based choices.

Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task

Humans form complex societies in which we routinely engage in social decision-making regarding the allocation of resources among ourselves and others. One dimension that characterizes social decision-making in particular is whether to prioritize self-interest or respect for others-proself or prosocial. What causes this individual difference in social value orientation? Recent developments in the social dual-process theory argue that social decision-making is characterized by its underlying domain-general learning systems: the model-free and model-based systems. In line with this "learning" approach, we propose and experimentally test the hypothesis that differences in social preferences stem from which learning system is dominant in an individual. Here, we used a non-social state transition task that allowed us to assess the balance between model-free/model-based learning and investigate its relation to the social value orientations. The results showed that proselfs depended more on model-based learning, whereas prosocials depended more on model-free learning. Reward amount and reaction time analyses showed that proselfs learned the task structure earlier in the session than prosocials, reflecting their difference in model-based/model-free learning dependence. These findings support the learning hypothesis on what makes differences in social preferences and have implications for understanding the mechanisms of prosocial behavior.

Stronger Prejudices Are Associated With Decreased Model-Based Control

Background: Prejudices against minorities can be understood as habitually negative evaluations that are kept in spite of evidence to the contrary. Therefore, individuals with strong prejudices might be dominated by habitual or "automatic" reactions at the expense of more controlled reactions. Computational theories suggest individual differences in the balance between habitual/model-free and deliberative/model-based decision-making. Methods: 127 subjects performed the two Step task and completed the blatant and subtle prejudice scale. Results: By using analyses of choices and reaction times in combination with computational modeling, subjects with stronger blatant prejudices showed a shift away from model-based control. There was no association between these decision-making processes and subtle prejudices. Conclusion: These results support the idea that blatant prejudices toward minorities are related to a relative dominance of habitual decision-making. This finding has important implications for developing interventions that target to change prejudices across societies.

Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference

Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlight a novel DA influence on MB-MF cooperative interactions.

Model-Based and Model-Free Control Predicts Alcohol Consumption Developmental Trajectory in Young Adults: A 3-Year Prospective Study

BACKGROUND

A shift from goal-directed toward habitual control has been associated with alcohol dependence. Whether such a shift predisposes to risky drinking is not yet clear. We investigated how goal-directed and habitual control at age 18 predict alcohol use trajectories over the course of 3 years.

METHODS

Goal-directed and habitual control, as informed by model-based (MB) and model-free (MF) learning, were assessed with a two-step sequential decision-making task during functional magnetic resonance imaging in 146 healthy 18-year-old men. Three-year alcohol use developmental trajectories were based on either a consumption score from the self-reported Alcohol Use Disorders Identification Test (assessed every 6 months) or an interview-based binge drinking score (grams of alcohol/occasion; assessed every year). We applied a latent growth curve model to examine how MB and MF control predicted the drinking trajectory.

RESULTS

Drinking behavior was best characterized by a linear trajectory. MB behavioral control was negatively associated with the development of the binge drinking score; MF reward prediction error blood oxygen level-dependent signals in the ventromedial prefrontal cortex and the ventral striatum predicted a higher starting point and steeper increase of the Alcohol Use Disorders Identification Test consumption score over time, respectively.

CONCLUSIONS

We found that MB behavioral control was associated with the binge drinking trajectory, while the MF reward prediction error signal was closely linked to the consumption score development. These findings support the idea that unbalanced MB and MF control might be an important individual vulnerability in predisposing to risky drinking behavior.

Neural representation of prediction error signals in substance users

Abnormal decision making can result in detrimental outcomes of clinical importance, and decision making is strongly linked to neural prediction error signalling. Activation likelihood estimation (ALE) meta-analyses were used to examine the neural correlates of prediction error signals of individuals taking different types of substances and healthy controls with contrast and conjunction analyses. Twenty-eight studies were included in the meta-analysis, representing 424 substance users' individuals and 834 healthy control individuals. Robust brain activity associated with prediction error signals in substance users was found for the bilateral striatum and insula. Healthy control subjects also activated bilateral striatum, midbrain, right insula and right medial-inferior frontal gyrus. Compared with healthy controls, substance users showed blunted activity in the bilateral putamen, right medial-inferior frontal gyrus and insula. The current meta-analysis of cross-sectional findings investigated neural prediction error signals in substance users. PE abnormalities in substance users might be related to poor decision making. In conclusion, the present study helps identify the pathophysiological underpinnings of maladaptive decision making in substance users.

Brain gray matter structures associated with trait impulsivity: A systematic review and voxel-based meta-analysis

Trait impulsivity is a multifaceted personality characteristic that contributes to maladaptive life outcomes. Although a growing body of neuroimaging studies have investigated the structural correlates of trait impulsivity, the findings remain highly inconsistent and heterogeneous. Herein, we performed a systematic review to depict an integrated delineation of gray matter (GM) substrates of trait impulsivity and a meta-analysis to examine concurrence across previous whole-brain voxel-based morphometry studies. The systematic review summarized the diverse findings in GM morphometry in the past literature, and the quantitative meta-analysis revealed impulsivity-related volumetric GM alterations in prefrontal, temporal, and parietal cortices. In addition, we identified the modulatory effects of age and gender in impulsivity-GM volume associations. The present study advances understanding of brain GM morphometry features underlying trait impulsivity. The findings may have practical implications in the clinical diagnosis of and intervention for impulsivity-related disorders.

Gray matter correlates of impulsivity in psychopathy and in the general population differ by kind, not by degree: a comparison of systematic reviews

A fundamental question in neuropsychiatry is whether a neurobiological continuum accompanies the behavioral continuum between subclinical and clinical traits. Impulsivity is a trait that varies in the general population and manifests severely in disorders like psychopathy. Is the neural profile of severe impulsivity in psychopathy an extreme but continuous manifestation of that associated with impulsivity in the general population (different by degree)? Or is it discontinuous and unique (different by kind)? Here, we compare systematic reviews of the relationship between impulsivity and gray matter in psychopathy and in the general population. The findings suggest that the neural profile associated with extreme impulsivity in psychopathy (increased gray matter in rostral and ventral striatum and prefrontal cortexes) is distinct from that associated with impulsivity in the general population (decreased gray matter in rostral and ventral prefrontal cortexes). Severe impulsivity in psychopathy may therefore arise from a pathophysiological mechanism that is unique to the disorder. These findings prompt the need for future studies to directly test the effect of group on the impulsivity–gray matter relationship in samples comprised of healthy individuals and individuals with psychopathy. The results caution against the use of community samples to examine impulsive psychopathic traits in relation to neurobiology.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

Combined model-free and model-sensitive reinforcement learning in non-human primates

Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates.

We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.

Alcohol Hangover Does Not Alter the Application of Model-Based and Model-Free Learning Strategies

Frequent alcohol binges shift behavior from goal-directed to habitual processing modes. This shift in reward-associated learning strategies plays a key role in the development and maintenance of alcohol use disorders and seems to persist during (early stages of) sobriety in at-risk drinkers. Yet still, it has remained unclear whether this phenomenon might be associated with alcohol hangover and thus also be found in social drinkers. In an experimental crossover design, n = 25 healthy young male participants performed a two-step decision-making task once sober and once hungover (i.e., when reaching sobriety after consuming 2.6 g of alcohol per estimated liter of total body water). This task allows the separation of effortful model-based and computationally less demanding model-free learning strategies. The experimental induction of alcohol hangover was successful, but we found no significant hangover effects on model-based and model-free learning scores, the balance between model-free and model-based valuation (ω), or perseveration tendencies (π). Bayesian analyses provided positive evidence for the null hypothesis for all measures except π (anecdotal evidence for the null hypothesis). Taken together, alcohol hangover, which results from a single binge drinking episode, does not impair the application of effortful and computationally costly model-based learning strategies and/or increase model-free learning strategies. This supports the notion that the behavioral deficits observed in at-risk drinkers are most likely not caused by the immediate aftereffects of individual binge drinking events.

Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning

Human decision-making is mainly driven by two fundamental learning processes: a slow, deliberative, goal-directed model-based process that maps out the potential outcomes of all options and a rapid habitual model-free process that enables reflexive repetition of previously successful choices. Although many model-informed neuroimaging studies have examined the neural correlates of model-based and model-free learning, the concordant activity among these two processes remains unclear. We used quantitative meta-analyses of functional magnetic resonance imaging experiments to identify the concordant activity pertaining to model-based and model-free learning over a range of reward-related paradigms. We found that: 1) both processes yielded concordant ventral striatum activity, 2) model-based learning activated the medial prefrontal cortex and orbital frontal cortex, and 3) model-free learning specifically activated the left globus pallidus and right caudate head. Our findings suggest that model-free and model-based decision making engage overlapping yet distinct neural regions. These stereotaxic maps improve our understanding of how deliberative goal-directed and reflexive habitual learning are implemented in the brain.

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Comparison of Behavioral Changes and Brain Activity between Adolescents with Internet Gaming Disorder and Student Pro-Gamers

While pro-gamers play according to defined living habits and planned schedules, adolescents with internet gaming disorder (IGD) exhibit irregular lifestyles and unregulated impulsive gaming behavior. Fourteen IGD adolescents and 12 pro-gaming students participated in this study. At baseline and after one year, demographic data, the Child Behavior Check List (CBCL), depressed mood, anxiety, and resting-state functional magnetic resonance imaging were assessed. Over the year, IGD adolescents played games as per their usual schedule, while pro-gamer students played according to their school’s team schedule. After one year, the pro-gamers’ scores had decreased in the CBCL-total (total problematic behaviors), CBCL-externalizing (under-controlled behavior, like impulsivity and aggression), and CBCL-internalizing (over-controlled behavior like depression and anxiety) compared to those of the IGD adolescents. Both groups displayed increased brain activity in the parietal lobe (a component of the attention network) over the years. Compared to pro-gamers, IGD adolescents showed higher brain activity within the left orbitofrontal cortex. Brain activity within the orbitofrontal cortex was associated with CBCL-externalizing scores. These results suggest that gaming had increased the attention network’s brain activity, but a well-organized support system could lead to different results, in terms of improved behaviors and suppressing brain activity within the orbitofrontal cortex.

No substantial change in the balance between model-free and model-based control via training on the two-step task

Human decisions can be habitual or goal-directed, also known as model-free (MF) or model-based (MB) control. Previous work suggests that the balance between the two decision systems is impaired in psychiatric disorders such as compulsion and addiction, via overreliance on MF control. However, little is known whether the balance can be altered through task training. Here, 20 healthy participants performed a well-established two-step task that differentiates MB from MF control, across five training sessions. We used computational modelling and functional near-infrared spectroscopy to assess changes in decision-making and brain hemodynamic over time. Mixed-effects modelling revealed overall no substantial changes in MF and MB behavior across training. Although our behavioral and brain findings show task-induced changes in learning rates, these parameters have no direct relation to either MF or MB control or the balance between the two systems, and thus do not support the assumption of training effects on MF or MB strategies. Our findings indicate that training on the two-step paradigm in its current form does not support a shift in the balance between MF and MB control. We discuss these results with respect to implications for restoring the balance between MF and MB control in psychiatric conditions.

Psychiatric conditions such as compulsion or addiction are associated with an overreliance on habitual, or model-free, decision-making. Goal-directed, or model-based, decision-making may protect against such overreliance. We therefore asked whether model-free control could be reduced, and model-based control strengthened, via task training. We used the well-characterized two-step task that differentiates model-based from model-free actions. Our results suggest that training on the current form of the two-step task does not support a shift in the balance between model-free and model-based strategies. Factors such as devaluation, demotivation or automatization during training may play a role in the missing emergence of a training effect. Future studies could adapt the two-step task so as to separate such factors from decision-making strategies.

Impulsivity and compulsivity are differentially associated with automaticity and routine on the Creature of Habit Scale

Habits may develop when meaningful action patterns are frequently repeated in a stable environment. We measured the differing tendencies of people to form habits in a population sample of n = 533 using the Creature of Habit Scale (COHS). We confirmed the high reliability of the two latent factors measured by the COHS, automaticity and routines. Whilst automatic behaviours are triggered by context and do not serve a particular purpose or goal, routines often have purpose, and because they have been performed so often in a given context, they become automatic only after their action sequence has been activated. We found that both types of habitual behaviours are influenced by the frequency of their occurrence and they are differentially influenced by personality traits. Compulsive personality is associated with an increase in both aspects of habitual tendency, whereas impulsivity is linked with increased automaticity, but reduced routine behaviours. Our findings provide further evidence that the COHS is a useful tool for understanding habitual tendencies in the general population and may inform the development of therapeutic strategies that capitalise on functional habits and help to treat dysfunctional ones.

Impulsivity and Active Inference

This paper characterizes impulsive behavior using a patch-leaving paradigm and active inference-a framework for describing Bayes optimal behavior. This paradigm comprises different environments (patches) with limited resources that decline over time at different rates. The challenge is to decide when to leave the current patch for another to maximize reward. We chose this task because it offers an operational characterization of impulsive behavior, namely, maximizing proximal reward at the expense of future gain. We use a Markov decision process formulation of active inference to simulate behavioral and electrophysiological responses under different models and prior beliefs. Our main finding is that there are at least three distinct causes of impulsive behavior, which we demonstrate by manipulating three different components of the Markov decision process model. These components comprise (i) the depth of planning, (ii) the capacity to maintain and process information, and (iii) the perceived value of immediate (relative to delayed) rewards. We show how these manipulations change beliefs and subsequent choices through variational message passing. Furthermore, we appeal to the process theories associated with this message passing to simulate neuronal correlates. In future work, we will use this scheme to identify the prior beliefs that underlie different sorts of impulsive behavior-and ask whether different causes of impulsivity can be inferred from the electrophysiological correlates of choice behavior.

L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action

Dopamine is a key neurotransmitter in action control. However, influential theories of dopamine function make conflicting predictions about the effect of boosting dopamine neurotransmission. Here, we tested if increases in dopamine tone by administration of L-DOPA upregulate reward learning as predicted by reinforcement learning theories, and if increases are specific for deliberative "model-based" control or reflexive "model-free" control. Alternatively, L-DOPA may impair learning as suggested by "value" or "thrift" theories of dopamine. To this end, we employed a two-stage Markov decision-task to investigate the effect of L-DOPA (randomized cross-over) on behavioral control while brain activation was measured using fMRI. L-DOPA led to attenuated model-free control of behavior as indicated by the reduced impact of reward on choice. Increased model-based control was only observed in participants with high working memory capacity. Furthermore, L-DOPA facilitated exploratory behavior, particularly after a stream of wins in the task. Correspondingly, in the brain, L-DOPA decreased the effect of reward at the outcome stage and when the next decision had to be made. Critically, reward-learning rates and prediction error signals were unaffected by L-DOPA, indicating that differences in behavior and brain response to reward were not driven by differences in learning. Taken together, our results suggest that L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action. These findings provide support for the value and thrift accounts of dopamine and call for a refined integration of valuation and action signals in reinforcement learning models.

Fractal Analysis of BOLD Time Series in a Network Associated With Waiting Impulsivity

Fractal phenomena can be found in numerous scientific areas including neuroscience. Fractals are structures, in which the whole has the same shape as its parts. A specific structure known as pink noise (also called fractal or 1/f noise) is one key fractal manifestation, exhibits both stability and adaptability, and can be addressed via the Hurst exponent (H). FMRI studies using H on regional fMRI time courses used fractality as an important characteristic to unravel neural networks from artificial noise. In this fMRI-study, we examined 103 healthy male students at rest and while performing the 5-choice serial reaction time task. We addressed fractality in a network associated with waiting impulsivity using the adaptive fractal analysis (AFA) approach to determine H. We revealed the fractal nature of the impulsivity network. Furthermore, fractality was influenced by individual impulsivity in terms of decreasing fractality with higher impulsivity in regions of top-down control (left middle frontal gyrus) as well as reward processing (nucleus accumbens and anterior cingulate cortex). We conclude that fractality as determined via H is a promising marker to quantify deviations in network functions at an early stage and, thus, to be able to inform preventive interventions before the manifestation of a disorder.

Model-based learning and individual differences in depression: The moderating role of stress

Inflexible decision-making has been proposed as a transdiagnostic risk factor for mood disorders. Evidence suggests that inflexible decision-making may emerge only when individuals are experiencing increased negative affect or stress. 151 participants completed symptom measures of depression and anxiety, followed by a two-stage decision-making task that distinguishes between habitual and goal-directed choice. An experimental manipulation to induce stress was introduced halfway through the task. Individuals with higher depression levels became less model-based after the manipulation than those with lower depression levels. There was no relationship between trait anxiety and the impact of the manipulation on decision-making. Controlling for main effects of anxiety did not attenuate the association between depression and impact of stress. Anhedonia was associated with the impact of the manipulation on model-based decision-making. These results suggest that risk for depression is associated with reflexive decision-making, but these effects may only emerge under conditions of stress.

The link between schizophrenia and substance use disorder: A unifying hypothesis

Substance use disorders occur commonly in patients with schizophrenia and dramatically worsen their overall clinical course. While the exact mechanisms contributing to substance use in schizophrenia are not known, a number of theories have been put forward to explain the basis of the co-occurrence of these disorders. We propose here a unifying hypothesis that combines recent evidence from epidemiological and genetic association studies with brain imaging and pre-clinical studies to provide an updated formulation regarding the basis of substance use in patients with schizophrenia. We suggest that the genetic determinants of risk for schizophrenia (especially within neural systems that contribute to the risk for both psychosis and addiction) make patients vulnerable to substance use. Since this vulnerability may arise prior to the appearance of psychotic symptoms, an increased use of substances in adolescence may both enhance the risk for developing a later substance use disorder, and also serve as an additional risk factor for the appearance of psychotic symptoms. Future studies that assess brain circuitry in a prospective longitudinal manner during adolescence prior to the appearance of psychotic symptoms could shed further light on the mechanistic underpinnings of these co-occurring disorders while identifying potential points of intervention for these difficult-to-treat co-occurring disorders.

No association of goal-directed and habitual control with alcohol consumption in young adults

Alcohol dependence is a mental disorder that has been associated with an imbalance in behavioral control favoring model-free habitual over model-based goal-directed strategies. It is as yet unknown, however, whether such an imbalance reflects a predisposing vulnerability or results as a consequence of repeated and/or excessive alcohol exposure. We, therefore, examined the association of alcohol consumption with model-based goal-directed and model-free habitual control in 188 18-year-old social drinkers in a two-step sequential decision-making task while undergoing functional magnetic resonance imaging before prolonged alcohol misuse could have led to severe neurobiological adaptations. Behaviorally, participants showed a mixture of model-free and model-based decision-making as observed previously. Measures of impulsivity were positively related to alcohol consumption. In contrast, neither model-free nor model-based decision weights nor the trade-off between them were associated with alcohol consumption. There were also no significant associations between alcohol consumption and neural correlates of model-free or model-based decision quantities in either ventral striatum or ventromedial prefrontal cortex. Exploratory whole-brain functional magnetic resonance imaging analyses with a lenient threshold revealed early onset of drinking to be associated with an enhanced representation of model-free reward prediction errors in the posterior putamen. These results suggest that an imbalance between model-based goal-directed and model-free habitual control might rather not be a trait marker of alcohol intake per se.

When Habits Are Dangerous: Alcohol Expectancies and Habitual Decision Making Predict Relapse in Alcohol Dependence

BACKGROUND

Addiction is supposedly characterized by a shift from goal-directed to habitual decision making, thus facilitating automatic drug intake. The two-step task allows distinguishing between these mechanisms by computationally modeling goal-directed and habitual behavior as model-based and model-free control. In addicted patients, decision making may also strongly depend upon drug-associated expectations. Therefore, we investigated model-based versus model-free decision making and its neural correlates as well as alcohol expectancies in alcohol-dependent patients and healthy controls and assessed treatment outcome in patients.

METHODS

Ninety detoxified, medication-free, alcohol-dependent patients and 96 age- and gender-matched control subjects underwent functional magnetic resonance imaging during the two-step task. Alcohol expectancies were measured with the Alcohol Expectancy Questionnaire. Over a follow-up period of 48 weeks, 37 patients remained abstinent and 53 patients relapsed as indicated by the Alcohol Timeline Followback method.

RESULTS

Patients who relapsed displayed reduced medial prefrontal cortex activation during model-based decision making. Furthermore, high alcohol expectancies were associated with low model-based control in relapsers, while the opposite was observed in abstainers and healthy control subjects. However, reduced model-based control per se was not associated with subsequent relapse.

CONCLUSIONS

These findings suggest that poor treatment outcome in alcohol dependence does not simply result from a shift from model-based to model-free control but is instead dependent on the interaction between high drug expectancies and low model-based decision making. Reduced model-based medial prefrontal cortex signatures in those who relapse point to a neural correlate of relapse risk. These observations suggest that therapeutic interventions should target subjective alcohol expectancies.

Computational approaches to schizophrenia: A perspective on negative symptoms

Schizophrenia is a heterogeneous spectrum disorder often associated with detrimental negative symptoms. In recent years, computational approaches to psychiatry have attracted growing attention. Negative symptoms have shown some overlap with general cognitive impairments and were also linked to impaired motivational processing in brain circuits implementing reward prediction. In this review, we outline how computational approaches may help to provide a better understanding of negative symptoms in terms of the potentially underlying behavioural and biological mechanisms. First, we describe the idea that negative symptoms could arise from a failure to represent reward expectations to enable flexible behavioural adaptation. It has been proposed that these impairments arise from a failure to use prediction errors to update expectations. Important previous studies focused on processing of so-called model-free prediction errors where learning is determined by past rewards only. However, learning and decision-making arise from multiple cognitive mechanisms functioning simultaneously, and dissecting them via well-designed tasks in conjunction with computational modelling is a promising avenue. Second, we move on to a proof-of-concept example on how generative models of functional imaging data from a cognitive task enable the identification of subgroups of patients mapping on different levels of negative symptoms. Combining the latter approach with behavioural studies regarding learning and decision-making may allow the identification of key behavioural and biological parameters distinctive for different dimensions of negative symptoms versus a general cognitive impairment. We conclude with an outlook on how this computational framework could, at some point, enrich future clinical studies.

Behavioral and Neural Signatures of Reduced Updating of Alternative Options in Alcohol-Dependent Patients during Flexible Decision-Making

Addicted individuals continue substance use despite the knowledge of harmful consequences and often report having no choice but to consume. Computational psychiatry accounts have linked this clinical observation to difficulties in making flexible and goal-directed decisions in dynamic environments via consideration of potential alternative choices. To probe this in alcohol-dependent patients (n = 43) versus healthy volunteers (n = 35), human participants performed an anticorrelated decision-making task during functional neuroimaging. Via computational modeling, we investigated behavioral and neural signatures of inference regarding the alternative option. While healthy control subjects exploited the anticorrelated structure of the task to guide decision-making, alcohol-dependent patients were relatively better explained by a model-free strategy due to reduced inference on the alternative option after punishment. Whereas model-free prediction error signals were preserved, alcohol-dependent patients exhibited blunted medial prefrontal signatures of inference on the alternative option. This reduction was associated with patients' behavioral deficit in updating the alternative choice option and their obsessive-compulsive drinking habits. All results remained significant when adjusting for potential confounders (e.g., neuropsychological measures and gray matter density). A disturbed integration of alternative choice options implemented by the medial prefrontal cortex appears to be one important explanation for the puzzling question of why addicted individuals continue drug consumption despite negative consequences.

SIGNIFICANCE STATEMENT

In addiction, patients maintain substance use despite devastating consequences and often report having no choice but to consume. These clinical observations have been theoretically linked to disturbed mechanisms of inference, for example, to difficulties when learning statistical regularities of the environmental structure to guide decisions. Using computational modeling, we demonstrate disturbed inference on alternative choice options in alcohol addiction. Patients neglecting "what might have happened" was accompanied by blunted coding of inference regarding alternative choice options in the medial prefrontal cortex. An impaired integration of alternative choice options implemented by the medial prefrontal cortex might contribute to ongoing drug consumption in the face of evident negative consequences.

Fractionating impulsivity: neuropsychiatric implications

The ability to make decisions and act quickly without hesitation can be advantageous in many settings. However, when persistently expressed, impulsive decisions and actions are considered risky, maladaptive and symptomatic of such diverse brain disorders as attention-deficit hyperactivity disorder, drug addiction and affective disorders. Over the past decade, rapid progress has been made in the identification of discrete neural networks that underlie different forms of impulsivity - from impaired response inhibition and risky decision making to a profound intolerance of delayed rewards. Herein, we review what is currently known about the neural and psychological mechanisms of impulsivity, and discuss the relevance and application of these new insights to various neuropsychiatric disorders.

A pathway linking reward circuitry, impulsive sensation-seeking and risky decision-making in young adults: identifying neural markers for new interventions

High trait impulsive sensation seeking (ISS) is common in 18-25-year olds, and is associated with risky decision-making and deleterious outcomes. We examined relationships among: activity in reward regions previously associated with ISS during an ISS-relevant context, uncertain reward expectancy (RE), using fMRI; ISS impulsivity and sensation-seeking subcomponents; and risky decision-making in 100, transdiagnostically recruited 18-25-year olds. ISS, anhedonia, anxiety, depression and mania were measured using self-report scales; clinician-administered scales also assessed the latter four. A post-scan risky decision-making task measured 'risky' (possible win/loss/mixed/neutral) fMRI-task versus 'sure thing' stimuli. 'Bias' reflected risky over safe choices. Uncertain RE-related activity in left ventrolateral prefrontal cortex and bilateral ventral striatum was positively associated with an ISS composite score, comprising impulsivity and sensation-seeking-fun-seeking subcomponents (ISSc; P⩽0.001). Bias positively associated with sensation seeking-experience seeking (ES; P=0.003). This relationship was moderated by ISSc (P=0.009): it was evident only in high ISSc individuals. Whole-brain analyses showed a positive relationship between: uncertain RE-related left ventrolateral prefrontal cortical activity and ISSc; uncertain RE-related visual attention and motor preparation neural network activity and ES; and uncertain RE-related dorsal anterior cingulate cortical activity and bias, specifically in high ISSc participants (all ps<0.05, peak-level, family-wise error corrected). We identify an indirect pathway linking greater levels of uncertain RE-related activity in reward, visual attention and motor networks with greater risky decision-making, via positive relationships with impulsivity, fun seeking and ES. These objective neural markers of high ISS can guide new treatment developments for young adults with high levels of this debilitating personality trait.

Impaired Flexible Reward-Based Decision-Making in Binge Eating Disorder: Evidence from Computational Modeling and Functional Neuroimaging

Despite its clinical relevance and the recent recognition as a diagnostic category in the DSM-5, binge eating disorder (BED) has rarely been investigated from a cognitive neuroscientific perspective targeting a more precise neurocognitive profiling of the disorder. BED patients suffer from a lack of behavioral control during recurrent binge eating episodes and thus fail to adapt their behavior in the face of negative consequences, eg, high risk for obesity. To examine impairments in flexible reward-based decision-making, we exposed BED patients (n=22) and matched healthy individuals (n=22) to a reward-guided decision-making task during functional resonance imaging (fMRI). Performing fMRI analysis informed via computational modeling of choice behavior, we were able to identify specific signatures of altered decision-making in BED. On the behavioral level, we observed impaired behavioral adaptation in BED, which was due to enhanced switching behavior, a putative deficit in striking a balance between exploration and exploitation appropriately. This was accompanied by diminished activation related to exploratory decisions in the anterior insula/ventro-lateral prefrontal cortex. Moreover, although so-called model-free reward prediction errors remained intact, representation of ventro-medial prefrontal learning signatures, incorporating inference on unchosen options, was reduced in BED, which was associated with successful decision-making in the task. On the basis of a computational psychiatry account, the presented findings contribute to defining a neurocognitive phenotype of BED.

Striatal dopamine, reward, and decision making in schizophrenia

Elevated striatal dopamine function is one of the best-established findings in schizophrenia. In this review, we discuss causes and consequences of this striata! dopamine alteration. We first summarize earlier findings regarding striatal reward processing and anticipation using functional neuroimaging. Secondly, we present a series of recent studies that are exemplary for a particular research approach: a combination of theory-driven reinforcement learning and decision-making tasks in combination with computational modeling and functional neuroimaging. We discuss why this approach represents a promising tool to understand underlying mechanisms of symptom dimensions by dissecting the contribution of multiple behavioral control systems working in parallel. We also discuss how it can advance our understanding of the neurobiological implementation of such functions. Thirdly, we review evidence regarding the topography of dopamine dysfunction within the striatum. Finally, we present conclusions and outline important aspects to be considered in future studies.

Subanesthetic ketamine decreases the incentive-motivational value of reward-related cues

The attribution of incentive-motivational value to reward-related cues contributes to cue-induced craving and relapse in addicted patients. Recently, it was demonstrated that subanesthetic ketamine increases motivation to quit and decreases cue-induced craving in cocaine-dependent individuals. Although the underlying mechanism of this effect is currently unknown, one possibility is that subanesthetic ketamine decreases the incentive-motivational value of reward-related cues. In the present study, we used a Pavlovian conditioned approach procedure to identify sign-trackers, rats that attribute incentive-motivational value to reward-related cues, and goal-trackers, rats that assign only predictive value to reward-related cues. This model is of interest because sign-trackers are more vulnerable to cue-induced reinstatement of drug-seeking behavior and will persist in this drug-seeking behavior despite adverse consequences. We tested the effect of subanesthetic ketamine on the expression of Pavlovian conditioned approach behavior and the conditioned reinforcing properties of a reward-related cue in sign- and goal-trackers. We found that subanesthetic ketamine decreased sign-tracking and increased goal-tracking behavior in sign-trackers, though it had no effect on conditioned reinforcement. These results suggest that subanesthetic ketamine may be a promising pharmacotherapy for addiction that acts by decreasing the incentive-motivational value of reward-related cues.

Slips of Action and Sequential Decisions: A Cross-Validation Study of Tasks Assessing Habitual and Goal-Directed Action Control

Instrumental learning and decision-making rely on two parallel systems: a goal-directed and a habitual system. In the past decade, several paradigms have been developed to study these systems in animals and humans by means of e.g., overtraining, devaluation procedures and sequential decision-making. These different paradigms are thought to measure the same constructs, but cross-validation has rarely been investigated. In this study we compared two widely used paradigms that assess aspects of goal-directed and habitual behavior. We correlated parameters from a two-step sequential decision-making task that assesses model-based (MB) and model-free (MF) learning with a slips-of-action paradigm that assesses the ability to suppress cue-triggered, learnt responses when the outcome has been devalued and is therefore no longer desirable. MB control during the two-step task showed a very moderately positive correlation with goal-directed devaluation sensitivity, whereas MF control did not show any associations. Interestingly, parameter estimates of MB and goal-directed behavior in the two tasks were positively correlated with higher-order cognitive measures (e.g., visual short-term memory). These cognitive measures seemed to (at least partly) mediate the association between MB control during sequential decision-making and goal-directed behavior after instructed devaluation. This study provides moderate support for a common framework to describe the propensity towards goal-directed behavior as measured with two frequently used tasks. However, we have to caution that the amount of shared variance between the goal-directed and MB system in both tasks was rather low, suggesting that each task does also pick up distinct aspects of goal-directed behavior. Further investigation of the commonalities and differences between the MF and habit systems as measured with these, and other, tasks is needed. Also, a follow-up cross-validation on the neural systems driving these constructs across different paradigms would promote the definition and operationalization of measures of instrumental learning and decision-making in humans.

Reversal learning strategy in adolescence is associated with prefrontal cortex activation

Adolescence is a critical maturation period for human cognitive control and executive function. In this study, a large sample of adolescents (n = 85) performed a reversal learning task during functional magnetic resonance imaging. We analyzed behavioral data using a reinforcement learning model to provide individually fitted parameters and imaging data with regard to reward prediction errors (PE). Following a model-based approach, we formed two groups depending on whether individuals tended to update expectations predominantly for the chosen stimulus or also for the unchosen one. These groups significantly differed in their problem behavior score obtained using the child behavior checklist (CBCL) and in a measure of their developmental stage. Imaging results showed that dorsolateral striatal areas covaried with PE. Participants who relied less on learning based on task structure showed less prefrontal activation compared with participants who relied more on task structure. An exploratory analysis revealed that PE-related activity was associated with pubertal development in prefrontal areas, insula and anterior cingulate. These findings support the hypothesis that the prefrontal cortex is implicated in mediating flexible goal-directed behavioral control.

When Does Model-Based Control Pay Off?

Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

When you make a choice about what groceries to get for dinner, you can rely on two different strategies. You can make your choice by relying on habit, simply buying the items you need to make a meal that is second nature to you. However, you can also plan your actions in a more deliberative way, realizing that the friend who will join you is a vegetarian, and therefore you should not make the burgers that have become a staple in your cooking. These two strategies differ in how computationally demanding and accurate they are. While the habitual strategy is less computationally demanding (costs less effort and time), the deliberative strategy is more accurate. Scientists have been able to study the distinction between these strategies using a task that allows them to measure how much people rely on habit and planning strategies. Interestingly, we have discovered that in this task, the deliberative strategy does not increase performance accuracy, and hence does not induce a trade-off between accuracy and demand. We describe why this happens, and improve the task so that it embodies an accuracy-demand trade-off, providing evidence for theories of cost-based arbitration between cognitive strategies.

Fuel not fun: Reinterpreting attenuated brain responses to reward in obesity

There is a well-established literature linking obesity to altered dopamine signaling and brain response to food-related stimuli. Neuroimaging studies frequently report enhanced responses in dopaminergic regions during food anticipation and decreased responses during reward receipt. This has been interpreted as reflecting anticipatory "reward surfeit", and consummatory "reward deficiency". In particular, attenuated response in the dorsal striatum to primary food rewards is proposed to reflect anhedonia, which leads to overeating in an attempt to compensate for the reward deficit. In this paper, we propose an alternative view. We consider brain response to food-related stimuli in a reinforcement-learning framework, which can be employed to separate the contributions of reward sensitivity and reward-related learning that are typically entangled in the brain response to reward. Consequently, we posit that decreased striatal responses to milkshake receipt reflect reduced reward-related learning rather than reward deficiency or anhedonia because reduced reward sensitivity would translate uniformly into reduced anticipatory and consummatory responses to reward. By re-conceptualizing reward deficiency as a shift in learning about subjective value of rewards, we attempt to reconcile neuroimaging findings with the putative role of dopamine in effort, energy expenditure and exploration and suggest that attenuated brain responses to energy dense foods reflect the "fuel", not the fun entailed by the reward.

Risk Factors for Addiction and Their Association with Model-Based Behavioral Control

Addiction shows familial aggregation and previous endophenotype research suggests that healthy relatives of addicted individuals share altered behavioral and cognitive characteristics with individuals suffering from addiction. In this study we asked whether impairments in behavioral control proposed for addiction, namely a shift from goal-directed, model-based toward habitual, model-free control, extends toward an unaffected sample (n = 20) of adult children of alcohol-dependent fathers as compared to a sample without any personal or family history of alcohol addiction (n = 17). Using a sequential decision-making task designed to investigate model-free and model-based control combined with a computational modeling analysis, we did not find any evidence for altered behavioral control in individuals with a positive family history of alcohol addiction. Independent of family history of alcohol dependence, we however observed that the interaction of two different risk factors of addiction, namely impulsivity and cognitive capacities, predicts the balance of model-free and model-based behavioral control. Post-hoc tests showed a positive association of model-based behavior with cognitive capacity in the lower, but not in the higher impulsive group of the original sample. In an independent sample of particularly high- vs. low-impulsive individuals, we confirmed the interaction effect of cognitive capacities and high vs. low impulsivity on model-based control. In the confirmation sample, a positive association of omega with cognitive capacity was observed in highly impulsive individuals, but not in low impulsive individuals. Due to the moderate sample size of the study, further investigation of the association of risk factors for addiction with model-based behavior in larger sample sizes is warranted.

Don't Think, Just Feel the Music: Individuals with Strong Pavlovian-to-Instrumental Transfer Effects Rely Less on Model-based Reinforcement Learning

Behavioral choice can be characterized along two axes. One axis distinguishes reflexive, model-free systems that slowly accumulate values through experience and a model-based system that uses knowledge to reason prospectively. The second axis distinguishes Pavlovian valuation of stimuli from instrumental valuation of actions or stimulus-action pairs. This results in four values and many possible interactions between them, with important consequences for accounts of individual variation. We here explored whether individual variation along one axis was related to individual variation along the other. Specifically, we asked whether individuals' balance between model-based and model-free learning was related to their tendency to show Pavlovian interferences with instrumental decisions. In two independent samples with a total of 243 participants, Pavlovian-instrumental transfer effects were negatively correlated with the strength of model-based reasoning in a two-step task. This suggests a potential common underlying substrate predisposing individuals to both have strong Pavlovian interference and be less model-based and provides a framework within which to interpret the observation of both effects in addiction.