Not so smart? "Smart" drugs increase the level but decrease the quality of cognitive effort

The efficacy of pharmaceutical cognitive enhancers in everyday complex tasks remains to be established. Using the knapsack optimization problem as a stylized representation of difficulty in tasks encountered in daily life, we discover that methylphenidate, dextroamphetamine, and modafinil cause knapsack value attained in the task to diminish significantly compared to placebo, even if the chance of finding the optimal solution (~50%) is not reduced significantly. Effort (decision time and number of steps taken to find a solution) increases significantly, but productivity (quality of effort) decreases significantly. At the same time, productivity differences across participants decrease, even reverse, to the extent that above-average performers end up below average and vice versa. The latter can be attributed to increased randomness of solution strategies. Our findings suggest that "smart drugs" increase motivation, but a reduction in quality of effort, crucial to solve complex problems, annuls this effect.

Epilepsy and Ecstatic Experiences: The Role of the Insula

Ecstatic epilepsy is a rare form of focal epilepsy in which the aura (beginning of the seizures) consists of a blissful state of mental clarity/feeling of certainty. Such a state has also been described as a “religious” or mystical experience. While this form of epilepsy has long been recognized as a temporal lobe epilepsy, we have accumulated evidence converging toward the location of the symptomatogenic zone in the dorsal anterior insula during the 10 last years. The neurocognitive hypothesis for the genesis of a mental clarity is the suppression of the interoceptive prediction errors and of the unexpected surprise associated with any incoming internal or external signal, usually processed by the dorsal anterior insula. This mimics a perfect prediction of the world and induces a feeling of certainty. The ecstatic epilepsy is thus an amazing model for the role of anterior insula in uncertainty and surprise.

How Neurobiology Elucidates the Role of Emotions in Financial Decision-Making

Over the last 15 years, a revolution has been taking place in neuroscience, whereby models and methods of economics have led to deeper insights into the neurobiological foundations of human decision-making. These have revealed a number of widespread mis-conceptions, among others, about the role of emotions. Furthermore, the findings suggest that a purely behavior-based approach to studying decisions may miss crucial features of human choice long appreciated in biology, such as Pavlovian approach. The findings could help economists formalize elusive concepts such as intuition, as I show here for financial “trading intuition.”

Separating Probability and Reversal Learning in a Novel Probabilistic Reversal Learning Task for Mice

The exploration/exploitation tradeoff – pursuing a known reward vs. sampling from lesser known options in the hope of finding a better payoff – is a fundamental aspect of learning and decision making. In humans, this has been studied using multi-armed bandit tasks. The same processes have also been studied using simplified probabilistic reversal learning (PRL) tasks with binary choices. Our investigations suggest that protocols previously used to explore PRL in mice may prove beyond their cognitive capacities, with animals performing at a no-better-than-chance level. We sought a novel probabilistic learning task to improve behavioral responding in mice, whilst allowing the investigation of the exploration/exploitation tradeoff in decision making. To achieve this, we developed a two-lever operant chamber task with levers corresponding to different probabilities (high/low) of receiving a saccharin reward, reversing the reward contingencies associated with levers once animals reached a threshold of 80% responding at the high rewarding lever. We found that, unlike in existing PRL tasks, mice are able to learn and behave near optimally with 80% high/20% low reward probabilities. Altering the reward contingencies towards equality showed that some mice displayed preference for the high rewarding lever with probabilities as close as 60% high/40% low. Additionally, we show that animal choice behavior can be effectively modelled using reinforcement learning (RL) models incorporating learning rates for positive and negative prediction error, a perseveration parameter, and a noise parameter. This new decision task, coupled with RL analyses, advances access to investigate the neuroscience of the exploration/exploitation tradeoff in decision making.

Uncertainty and computational complexity

Modern theories of decision-making typically model uncertainty about decision options using the tools of probability theory. This is exemplified by the Savage framework, the most popular framework in decision-making research. There, decision-makers are assumed to choose from among available decision options as if they maximized subjective expected utility, which is given by the utilities of outcomes in different states weighted with subjective beliefs about the occurrence of those states. Beliefs are captured by probabilities and new information is incorporated using Bayes' Law. The primary concern of the Savage framework is to ensure that decision-makers' choices are rational. Here, we use concepts from computational complexity theory to expose two major weaknesses of the framework. Firstly, we argue that in most situations, subjective utility maximization is computationally intractable, which means that the Savage axioms are implausible. We discuss empirical evidence supporting this claim. Secondly, we argue that there exist many decision situations in which the nature of uncertainty is such that (random) sampling in combination with Bayes' Law is an ineffective strategy to reduce uncertainty. We discuss several implications of these weaknesses from both an empirical and a normative perspective. This article is part of the theme issue 'Risk taking and impulsive behaviour: fundamental discoveries, theoretical perspectives and clinical implications'.

Formalizing the Function of Anterior Insula in Rapid Adaptation

Anterior insula (aIns) is thought to play a crucial role in rapid adaptation in an ever-changing environment. Mathematically, it is known to track risk and surprise. Modern theories of learning, however, assign a dominant role to signed prediction errors (PEs), not to risk and surprise. Risk and surprise only enter to the extent that they modulate the learning rate, in an attempt to approximate Bayesian learning. Even without such modulation, adaptation is still possible, albeit slow. Here, I propose a new theory of learning, reference-model based learning (RMBL), where risk and surprise are central, and PEs play a secondary, though still crucial, role. The primary goal is to bring outcomes in line with expectations in the reference model (RM). Learning is modulated by how large the PEs are relative to model anticipation, i.e., to surprise as defined by the RM. In a target location prediction task where participants were continuously required to adapt, choices appeared to be closer with to RMBL predictions than to Bayesian learning. aIns reaction to surprise was more acute in the more difficult treatment, consistent with its hypothesized role in metacognition. I discuss links with related theories, such as Active Inference, Actor-Critic Models and Reference-Model Based Adaptive Control.

Neural computations underlying inverse reinforcement learning in the human brain

In inverse reinforcement learning an observer infers the reward distribution available for actions in the environment solely through observing the actions implemented by another agent. To address whether this computational process is implemented in the human brain, participants underwent fMRI while learning about slot machines yielding hidden preferred and non-preferred food outcomes with varying probabilities, through observing the repeated slot choices of agents with similar and dissimilar food preferences. Using formal model comparison, we found that participants implemented inverse RL as opposed to a simple imitation strategy, in which the actions of the other agent are copied instead of inferring the underlying reward structure of the decision problem. Our computational fMRI analysis revealed that anterior dorsomedial prefrontal cortex encoded inferences about action-values within the value space of the agent as opposed to that of the observer, demonstrating that inverse RL is an abstract cognitive process divorceable from the values and concerns of the observer him/herself.

How Humans Solve Complex Problems: The Case of the Knapsack Problem

Life presents us with problems of varying complexity. Yet, complexity is not accounted for in theories of human decision-making. Here we study instances of the knapsack problem, a discrete optimisation problem commonly encountered at all levels of cognition, from attention gating to intellectual discovery. Complexity of this problem is well understood from the perspective of a mechanical device like a computer. We show experimentally that human performance too decreased with complexity as defined in computer science. Defying traditional economic principles, participants spent effort way beyond the point where marginal gain was positive, and economic performance increased with instance difficulty. Human attempts at solving the instances exhibited commonalities with algorithms developed for computers, although biological resource constraints–limited working and episodic memories–had noticeable impact. Consistent with the very nature of the knapsack problem, only a minority of participants found the solution–often quickly–but the ones who did appeared not to realise. Substantial heterogeneity emerged, suggesting why prizes and patents, schemes that incentivise intellectual discovery but discourage information sharing, have been found to be less effective than mechanisms that reveal private information, such as markets.

Decision Neuroscience: Why We Become More Cautious with Age

A recent study suggests that risk-taking decreases with age and that this may be related to dopamine-modulated changes in Pavlovian approach behavior, and not a reduction in the subjective value of incremental rewards as traditional models from economics and psychology would have claimed.

Neural Mechanisms Behind Identification of Leptokurtic Noise and Adaptive Behavioral Response

Large-scale human interaction through, for example, financial markets causes ceaseless random changes in outcome variability, producing frequent and salient outliers that render the outcome distribution more peaked than the Gaussian distribution, and with longer tails. Here, we study how humans cope with this evolutionary novel leptokurtic noise, focusing on the neurobiological mechanisms that allow the brain, 1) to recognize the outliers as noise and 2) to regulate the control necessary for adaptive response. We used functional magnetic resonance imaging, while participants tracked a target whose movements were affected by leptokurtic noise. After initial overreaction and insufficient subsequent correction, participants improved performance significantly. Yet, persistently long reaction times pointed to continued need for vigilance and control. We ran a contrasting treatment where outliers reflected permanent moves of the target, as in traditional mean-shift paradigms. Importantly, outliers were equally frequent and salient. There, control was superior and reaction time was faster. We present a novel reinforcement learning model that fits observed choices better than the Bayes-optimal model. Only anterior insula discriminated between the 2 types of outliers. In both treatments, outliers initially activated an extensive bottom-up attention and belief network, followed by sustained engagement of the fronto-parietal control network.

Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism

For making decisions in everyday life we often have first to infer the set of environmental features that are relevant for the current task. Here we investigated the computational mechanisms underlying the evolution of beliefs about the relevance of environmental features in a dynamical and noisy environment. For this purpose we designed a probabilistic Wisconsin card sorting task (WCST) with belief solicitation, in which subjects were presented with stimuli composed of multiple visual features. At each moment in time a particular feature was relevant for obtaining reward, and participants had to infer which feature was relevant and report their beliefs accordingly. To test the hypothesis that attentional focus modulates the belief update process, we derived and fitted several probabilistic and non-probabilistic behavioral models, which either incorporate a dynamical model of attentional focus, in the form of a hierarchical winner-take-all neuronal network, or a diffusive model, without attention-like features. We used Bayesian model selection to identify the most likely generative model of subjects' behavior and found that attention-like features in the behavioral model are essential for explaining subjects' responses. Furthermore, we demonstrate a method for integrating both connectionist and Bayesian models of decision making within a single framework that allowed us to infer hidden belief processes of human subjects.

Neural mechanisms underlying human consensus decision-making

Consensus building in a group is a hallmark of animal societies, yet little is known about its underlying computational and neural mechanisms. Here, we applied a computational framework to behavioral and fMRI data from human participants performing a consensus decision-making task with up to five other participants. We found that participants reached consensus decisions through integrating their own preferences with information about the majority group members' prior choices, as well as inferences about how much each option was stuck to by the other people. These distinct decision variables were separately encoded in distinct brain areas-the ventromedial prefrontal cortex, posterior superior temporal sulcus/temporoparietal junction, and intraparietal sulcus-and were integrated in the dorsal anterior cingulate cortex. Our findings provide support for a theoretical account in which collective decisions are made through integrating multiple types of inference about oneself, others, and environments, processed in distinct brain modules.

Economic choices reveal probability distortion in macaque monkeys

Economic choices are largely determined by two principal elements, reward value (utility) and probability. Although nonlinear utility functions have been acknowledged for centuries, nonlinear probability weighting (probability distortion) was only recently recognized as a ubiquitous aspect of real-world choice behavior. Even when outcome probabilities are known and acknowledged, human decision makers often overweight low probability outcomes and underweight high probability outcomes. Whereas recent studies measured utility functions and their corresponding neural correlates in monkeys, it is not known whether monkeys distort probability in a manner similar to humans. Therefore, we investigated economic choices in macaque monkeys for evidence of probability distortion. We trained two monkeys to predict reward from probabilistic gambles with constant outcome values (0.5 ml or nothing). The probability of winning was conveyed using explicit visual cues (sector stimuli). Choices between the gambles revealed that the monkeys used the explicit probability information to make meaningful decisions. Using these cues, we measured probability distortion from choices between the gambles and safe rewards. Parametric modeling of the choices revealed classic probability weighting functions with inverted-S shape. Therefore, the animals overweighted low probability rewards and underweighted high probability rewards. Empirical investigation of the behavior verified that the choices were best explained by a combination of nonlinear value and nonlinear probability distortion. Together, these results suggest that probability distortion may reflect evolutionarily preserved neuronal processing.

Chimpanzee choice rates in competitive games match equilibrium game theory predictions

The capacity for strategic thinking about the payoff-relevant actions of conspecifics is not well understood across species. We use game theory to make predictions about choices and temporal dynamics in three abstract competitive situations with chimpanzee participants. Frequencies of chimpanzee choices are extremely close to equilibrium (accurate-guessing) predictions, and shift as payoffs change, just as equilibrium theory predicts. The chimpanzee choices are also closer to the equilibrium prediction, and more responsive to past history and payoff changes, than two samples of human choices from experiments in which humans were also initially uninformed about opponent payoffs and could not communicate verbally. The results are consistent with a tentative interpretation of game theory as explaining evolved behavior, with the additional hypothesis that chimpanzees may retain or practice a specialized capacity to adjust strategy choice during competition to perform at least as well as, or better than, humans have.

Using Neural Data to Test A Theory of Investor Behavior: An Application to Realization Utility

We use measures of neural activity provided by functional magnetic resonance imaging (fMRI) to test the "realization utility" theory of investor behavior, which posits that people derive utility directly from the act of realizing gains and losses. Subjects traded stocks in an experimental market while we measured their brain activity. We find that all subjects exhibit a strong disposition effect in their trading, even though it is suboptimal. Consistent with the realization utility explanation for this behavior, we find that activity in the ventromedial prefrontal cortex, an area known to encode the value of options during choices, correlates with the capital gains of potential trades; that the neural measures of realization utility correlate across subjects with their individual tendency to exhibit a disposition effect; and that activity in the ventral striatum, an area known to encode information about changes in the present value of experienced utility, exhibits a positive response when subjects realize capital gains. These results provide support for the realization utility model and, more generally, demonstrate how neural data can be helpful in testing models of investor behavior.

In the mind of the market: theory of mind biases value computation during financial bubbles

The ability to infer intentions of other agents, called theory of mind (ToM), confers strong advantages for individuals in social situations. Here, we show that ToM can also be maladaptive when people interact with complex modern institutions like financial markets. We tested participants who were investing in an experimental bubble market, a situation in which the price of an asset is much higher than its underlying fundamental value. We describe a mechanism by which social signals computed in the dorsomedial prefrontal cortex affect value computations in ventromedial prefrontal cortex, thereby increasing an individual's propensity to 'ride' financial bubbles and lose money. These regions compute a financial metric that signals variations in order flow intensity, prompting inference about other traders' intentions. Our results suggest that incorporating inferences about the intentions of others when making value judgments in a complex financial market could lead to the formation of market bubbles.

The chronometry of risk processing in the human cortex

The neuroscience of human decision-making has focused on localizing brain activity correlating with decision variables and choice, most commonly using functional MRI (fMRI). Poor temporal resolution means these studies are agnostic in relation to how decisions unfold in time. Consequently, here we address the temporal evolution of neural activity related to encoding of risk using magnetoencephalography (MEG), and show modulations of electromagnetic power in posterior parietal and dorsomedial prefrontal cortex (DMPFC) which scale with both variance and skewness in a lottery, detectable within 500 ms following stimulus presentation. Electromagnetic responses in somatosensory cortex following this risk encoding predict subsequent choices. Furthermore, within anterior insula we observed early and late effects of subject-specific risk preferences, suggestive of a role in both risk assessment and risk anticipation during choice. The observation that cortical activity tracks specific and independent components of risk from early time-points in a decision-making task supports the hypothesis that specialized brain circuitry underpins risk perception.

Evidence for model-based computations in the human amygdala during Pavlovian conditioning

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.

A hot topic in the neurobiology of learning is the idea that there may be two distinct mechanisms for learning in the brain: a model-based learning system in which predictions are made with respect to a rich internal model of the learning environment, versus a “model-free” mechanism in which trial-and-error learning occurs without any rich internal representation of the world. While the focus in the literature to date has been on the role of these mechanisms in instrumental conditioning, almost nothing is known about whether more fundamental kinds of learning such as Pavlovian conditioning also involve model-based processes. Furthermore, nothing is known about the extent to which the amygdala, which is known to be a core structure for Pavlovian learning, contains neural signals consistent with a model-based mechanism. To address this question, we used a novel Pavlovian conditioning task and scanned human volunteers with a special high-resolution fMRI sequence that enabled us to obtain signals within the amygdala with over four times the resolution of conventional imaging protocols. Using this approach in combination with sophisticated computational analyses, we find evidence to suggest that the human amygdala is involved in model-based computations during Pavlovian conditioning.

Activity in inferior parietal and medial prefrontal cortex signals the accumulation of evidence in a probability learning task

In an uncertain environment, probabilities are key to predicting future events and making adaptive choices. However, little is known about how humans learn such probabilities and where and how they are encoded in the brain, especially when they concern more than two outcomes. During functional magnetic resonance imaging (fMRI), young adults learned the probabilities of uncertain stimuli through repetitive sampling. Stimuli represented payoffs and participants had to predict their occurrence to maximize their earnings. Choices indicated loss and risk aversion but unbiased estimation of probabilities. BOLD response in medial prefrontal cortex and angular gyri increased linearly with the probability of the currently observed stimulus, untainted by its value. Connectivity analyses during rest and task revealed that these regions belonged to the default mode network. The activation of past outcomes in memory is evoked as a possible mechanism to explain the engagement of the default mode network in probability learning. A BOLD response relating to value was detected only at decision time, mainly in striatum. It is concluded that activity in inferior parietal and medial prefrontal cortex reflects the amount of evidence accumulated in favor of competing and uncertain outcomes.

Do not Bet on the Unknown Versus Try to Find Out More: Estimation Uncertainty and "Unexpected Uncertainty" Both Modulate Exploration

Little is known about how humans solve the exploitation/exploration trade-off. In particular, the evidence for uncertainty-driven exploration is mixed. The current study proposes a novel hypothesis of exploration that helps reconcile prior findings that may seem contradictory at first. According to this hypothesis, uncertainty-driven exploration involves a dilemma between two motives: (i) to speed up learning about the unknown, which may beget novel reward opportunities; (ii) to avoid the unknown because it is potentially dangerous. We provide evidence for our hypothesis using both behavioral and simulated data, and briefly point to recent evidence that the brain differentiates between these two motives.

Hedging your bets by learning reward correlations in the human brain

Human subjects are proficient at tracking the mean and variance of rewards and updating these via prediction errors. Here, we addressed whether humans can also learn about higher-order relationships between distinct environmental outcomes, a defining ecological feature of contexts where multiple sources of rewards are available. By manipulating the degree to which distinct outcomes are correlated, we show that subjects implemented an explicit model-based strategy to learn the associated outcome correlations and were adept in using that information to dynamically adjust their choices in a task that required a minimization of outcome variance. Importantly, the experimentally generated outcome correlations were explicitly represented neuronally in right midinsula with a learning prediction error signal expressed in rostral anterior cingulate cortex. Thus, our data show that the human brain represents higher-order correlation structures between rewards, a core adaptive ability whose immediate benefit is optimized sampling.

Separate encoding of model-based and model-free valuations in the human brain

Behavioral studies have long shown that humans solve problems in two ways, one intuitive and fast (System 1, model-free), and the other reflective and slow (System 2, model-based). The neurobiological basis of dual process problem solving remains unknown due to challenges of separating activation in concurrent systems. We present a novel neuroeconomic task that predicts distinct subjective valuation and updating signals corresponding to these two systems. We found two concurrent value signals in human prefrontal cortex: a System 1 model-free reinforcement signal and a System 2 model-based Bayesian signal. We also found a System 1 updating signal in striatal areas and a System 2 updating signal in lateral prefrontal cortex. Further, signals in prefrontal cortex preceded choices that are optimal according to either updating principle, while signals in anterior cingulate cortex and globus pallidus preceded deviations from optimal choice for reinforcement learning. These deviations tended to occur when uncertainty regarding optimal values was highest, suggesting that disagreement between dual systems is mediated by uncertainty rather than conflict, confirming recent theoretical proposals.

The human prefrontal cortex mediates integration of potential causes behind observed outcomes

Prefrontal cortex has long been implicated in tasks involving higher order inference in which decisions must be rendered, not only about which stimulus is currently rewarded, but also which stimulus dimensions are currently relevant. However, the precise computational mechanisms used to solve such tasks have remained unclear. We scanned human participants with functional MRI, while they performed a hierarchical intradimensional/extradimensional shift task to investigate what strategy subjects use while solving higher order decision problems. By using a computational model-based analysis, we found behavioral and neural evidence that humans solve such problems not by occasionally shifting focus from one to the other dimension, but by considering multiple explanations simultaneously. Activity in human prefrontal cortex was better accounted for by a model that integrates over all available evidences than by a model in which attention is selectively gated. Importantly, our model provides an explanation for how the brain determines integration weights, according to which it could distribute its attention. Our results demonstrate that, at the point of choice, the human brain and the prefrontal cortex in particular are capable of a weighted integration of information across multiple evidences.

Differentiable contributions of human amygdalar subregions in the computations underlying reward and avoidance learning

To understand how the human amygdala contributes to associative learning, it is necessary to differentiate the contributions of its subregions. However, major limitations in the techniques used for the acquisition and analysis of functional magnetic resonance imaging (fMRI) data have hitherto precluded segregation of function with the amygdala in humans. Here, we used high-resolution fMRI in combination with a region-of-interest-based normalization method to differentiate functionally the contributions of distinct subregions within the human amygdala during two different types of instrumental conditioning: reward and avoidance learning. Through the application of a computational-model-based analysis, we found evidence for a dissociation between the contributions of the basolateral and centromedial complexes in the representation of specific computational signals during learning, with the basolateral complex contributing more to reward learning, and the centromedial complex more to avoidance learning. These results provide unique insights into the computations being implemented within fine-grained amygdala circuits in the human brain.

The affective impact of financial skewness on neural activity and choice

Few finance theories consider the influence of “skewness” (or large and asymmetric but unlikely outcomes) on financial choice. We investigated the impact of skewed gambles on subjects' neural activity, self-reported affective responses, and subsequent preferences using functional magnetic resonance imaging (FMRI). Neurally, skewed gambles elicited more anterior insula activation than symmetric gambles equated for expected value and variance, and positively skewed gambles also specifically elicited more nucleus accumbens (NAcc) activation than negatively skewed gambles. Affectively, positively skewed gambles elicited more positive arousal and negatively skewed gambles elicited more negative arousal than symmetric gambles equated for expected value and variance. Subjects also preferred positively skewed gambles more, but negatively skewed gambles less than symmetric gambles of equal expected value. Individual differences in both NAcc activity and positive arousal predicted preferences for positively skewed gambles. These findings support an anticipatory affect account in which statistical properties of gambles—including skewness—can influence neural activity, affective responses, and ultimately, choice.

Positive temporal dependence of the biological clock implies hyperbolic discounting

Temporal preferences of animals and humans often exhibit inconsistencies, whereby an earlier, smaller reward may be preferred when it occurs immediately but not when it is delayed. Such choices reflect hyperbolic discounting of future rewards, rather than the exponential discounting required for temporal consistency. Simultaneously, however, evidence has emerged that suggests that animals and humans have an internal representation of time that often differs from the calendar time used in detection of temporal inconsistencies. Here, we prove that temporal inconsistencies emerge if fixed durations in calendar time are experienced as positively related (positive quadrant dependent). Hence, what are time-consistent choices within the time framework of the decision maker appear as time-inconsistent to an outsider who analyzes choices in calendar time. As the biological clock becomes more variable, the fit of the hyperbolic discounting model improves. A recent alternative explanation for temporal choice inconsistencies builds on persistent under-estimation of the length of distant time intervals. By increasing the expected speed of our stochastic biological clock for time farther into the future, we can emulate this explanation. Ours is therefore an encompassing theoretical framework that predicts context-dependent degrees of intertemporal choice inconsistencies, to the extent that context can generate changes in autocorrelation, variability, and expected speed of the biological clock. Our finding should lead to novel experiments that will clarify the role of time perception in impulsivity, with critical implications for, among others, our understanding of aging, drug abuse, and pathological gambling.

Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings

Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating.

The impact of disappointment in decision making: inter-individual differences and electrical neuroimaging

Disappointment, the emotion experienced when faced to reward prediction errors (RPEs), considerably impacts decision making (DM). Individuals tend to modify their behavior in an often unpredictable way just to avoid experiencing negative emotions. Despite its importance, disappointment remains much less studied than regret and its impact on upcoming decisions largely unexplored. Here, we adapted the Trust Game to effectively elicit, quantify, and isolate disappointment by relying on the formal definition provided by Bell's in economics. We evaluated the effects of experienced disappointment and elation on future cooperation and trust as well as the rationality and utility of the different behavioral and neural mechanisms used to cope with disappointment. All participants in our game trusted less and particularly expected less from unknown opponents as a result of disappointing outcomes in the previous trial but not necessarily after elation indicating that behavioral consequences of positive and negative RPEs are not the same. A large variance in the tolerance to disappointment was observed across subjects, with some participants needing only a small disappointment to impulsively bias their subsequent decisions. As revealed by high-density EEG recordings the most tolerant individuals - who thought twice before making a decision and earned more money - relied on different neural generators to contend with neutral and unexpected outcomes. This study thus provides some support to the idea that different neural systems underlie reflexive and reflective decisions within the same individuals as predicted by the dual-system theory of social judgment and DM.

MAOA-L carriers are better at making optimal financial decisions under risk

Genes can affect behaviour towards risks through at least two distinct neurocomputational mechanisms: they may affect the value assigned to different risky options, or they may affect the way in which the brain adjudicates between options based on their value. We combined methods from neuroeconomics and behavioural genetics to investigate the impact that the genes encoding for monoamine oxidase-A (MAOA), the serotonin transporter (5-HTT) and the dopamine D4 receptor (DRD4) have on these two computations. Consistent with previous literature, we found that carriers of the MAOA-L polymorphism were more likely to take financial risks. Our computational choice model, rooted in established decision theory, showed that MAOA-L carriers exhibited such behaviour because they are able to make better financial decisions under risk, and not because they are more impulsive. In contrast, we found no behavioural or computational differences among the 5-HTT and DRD4 polymorphisms.

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.

Toward a Mechanistic Understanding of Human Decision Making

This article considers the contribution of functional neuroimaging toward understanding the computational underpinnings of human decision making. We outline the main processes likely underlying the capacity to make simple choices and describe their associated neural substrates. Relevant processes include the ability to encode a representation of the expected value or utility associated with each option in a decision problem, to learn such expectations through experience, and to modify action selection in order to choose those actions leading to the greatest reward. We provide several examples of how functional neuroimaging data have helped to shape and inform theories of decision making over and above results available from traditional behavioral measures.

Investigating signal integration with canonical correlation analysis of fMRI brain activation data

How the brain integrates signals from specific areas has been a longstanding critical question for neurobiologists. Two recent observations suggest a new approach to fMRI data analysis of this question. First, in many instances, the brain analyzes inputs by decomposing the information along several salient dimensions. For example, earlier work demonstrated that the brain splits a monetary gamble in terms of expected reward (ER) and variance of the reward (risk) [Preuschoff, K., Bossaerts, P., Quartz, S., 2006. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381-390]. However, since ER and risk activate separate brain regions, the brain needs to integrate these activations to obtain an overall evaluation of the gamble. Second, recent evidence suggests that the correlation of the activity between neurons may serve a specific organizational purpose [Romo, R., Hernandez, A., Zainos, A., Salinas, E., 2003. Correlated neuronal discharges that increase coding efficiency during perceptual discrimination. Neuron 38, 649-657; Salinas, E., Sejnowski, T.J., 2001. Correlated neuronal activity and the flow of neural information. Nat. Rev. Neurosci. 2, 539]. Specifically, it is hypothesized that correlations allow brain regions to integrate several signals in a way that minimizes noise. Under this hypothesis, we show here that canonical correlation analysis of fMRI data identifies how the signals from several regions are combined. A general linear model then verifies whether the identified combination indeed activates a projection area in the brain. We illustrate the proposed procedure on data recorded while human subjects played a simple card game. We show that the brain adds the signals of ER and risk to form a measure that activates the medial prefrontal cortex, consistent with the role of this brain structure in the evaluation of monetary gambles.

Neural antecedents of financial decisions

To explain investing decisions, financial theorists invoke two opposing metrics: expected reward and risk. Recent advances in the spatial and temporal resolution of brain imaging techniques enable investigators to visualize changes in neural activation before financial decisions. Research using these methods indicates that although the ventral striatum plays a role in representation of expected reward, the insula may play a more prominent role in the representation of expected risk. Accumulating evidence also suggests that antecedent neural activation in these regions can be used to predict upcoming financial decisions. These findings have implications for predicting choices and for building a physiologically constrained theory of decision-making.

Adding Prediction Risk to the Theory of Reward Learning

This article analyzes the simple Rescorla-Wagner learning rule from the vantage point of least squares learning theory. In particular, it suggests how measures of risk, such as prediction risk, can be used to adjust the learning constant in reinforcement learning. It argues that prediction risk is most effectively incorporated by scaling the prediction errors. This way, the learning rate needs adjusting only when the covariance between optimal predictions and past (scaled) prediction errors changes. Evidence is discussed that suggests that the dopaminergic system in the (human and nonhuman) primate brain encodes prediction risk, and that prediction errors are indeed scaled with prediction risk (adaptive encoding).

The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans

Many real-life decision-making problems incorporate higher-order structure, involving interdependencies between different stimuli, actions, and subsequent rewards. It is not known whether brain regions implicated in decision making, such as the ventromedial prefrontal cortex (vmPFC), use a stored model of the task structure to guide choice (model-based decision making) or merely learn action or state values without assuming higher-order structure as in standard reinforcement learning. To discriminate between these possibilities, we scanned human subjects with functional magnetic resonance imaging while they performed a simple decision-making task with higher-order structure, probabilistic reversal learning. We found that neural activity in a key decision-making region, the vmPFC, was more consistent with a computational model that exploits higher-order structure than with simple reinforcement learning. These results suggest that brain regions, such as the vmPFC, use an abstract model of task structure to guide behavioral choice, computations that may underlie the human capacity for complex social interactions and abstract strategizing.

Neural Differentiation of Expected Reward and Risk in Human Subcortical Structures

In decision-making under uncertainty, economic studies emphasize the importance of risk in addition to expected reward. Studies in neuroscience focus on expected reward and learning rather than risk. We combined functional imaging with a simple gambling task to vary expected reward and risk simultaneously and in an uncorrelated manner. Drawing on financial decision theory, we modeled expected reward as mathematical expectation of reward, and risk as reward variance. Activations in dopaminoceptive structures correlated with both mathematical parameters. These activations differentiated spatially and temporally. Temporally, the activation related to expected reward was immediate, while the activation related to risk was delayed. Analyses confirmed that our paradigm minimized confounds from learning, motivation, and salience. These results suggest that the primary task of the dopaminergic system is to convey signals of upcoming stochastic rewards, such as expected reward and risk, beyond its role in learning, motivation, and salience.