Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making

Decision makers consider all options in choice triplets

Most contemporary decision-making research focuses on choices between only two alternative options, in spite of the fact that most real-world decisions involve more than two options. Beyond this practical point, multi-option decisions are also important from a theoretical perspective. Experimental and computational studies have demonstrated that the composition of a set of choice options has predictable effects on choice outcomes. Specifically, with more options available to choose from, responses are slower and more stochastic. This effect is amplified when the values of the options (including the worst option in the set) are more similar to each other. In this study, we provide further evidence of these known effects. We also provide evidence that metacognitive factors such as feelings of confidence in the response or mental effort exertion during deliberation show similar effects as the cognitive factors (consistency between choices and value estimates, response speed). Finally, we provide novel evidence that value estimates are refined during deliberation for all options in choice triplets, similar to what has previously been show for choice pairs.

Mutual inclusivity improves decision-making by smoothing out choice's competitive edge

Decisions form a central bottleneck to most tasks, one that people often experience as costly. Previous work proposes mitigating those costs by lowering one's threshold for deciding. Here we test an alternative solution, one that targets the basis of most choice costs: the idea that choosing one option sacrifices others (mutual exclusivity). Across 6 studies (N = 565), we test whether this tension can be relieved by framing choices as inclusive (allowing selection of more than 1 option, as in buffets). We find that inclusivity makes choices more efficient by selectively reducing competition between potential responses as participants accumulate information for each of their options. Inclusivity also made participants feel less conflicted, especially when they could not decide which good option to keep or which bad option to get rid of. These inclusivity benefits were also distinguishable from the effects of manipulating decision threshold (increased urgency), which improved choices but not experiences thereof.

Distinct neural computations scale the violation of expected reward and emotion in social transgressions

Traditional decision-making models conceptualize humans as adaptive learners utilizing the differences between expected and actual rewards (prediction errors, PEs) to maximize outcomes, but rarely consider the influence of violations of emotional expectations (emotional PEs) and how it differs from reward PEs. Here, we conducted a fMRI experiment (n = 43) using a modified Ultimatum Game to examine how reward and emotional PEs affect punishment decisions in terms of rejecting unfair offers. Our results revealed that reward relative to emotional PEs exerted a stronger prediction to punishment decisions. On the neural level, the left dorsomedial prefrontal cortex (dmPFC) was strongly activated during reward receipt whereas the emotions engaged the bilateral anterior insula. Reward and emotional PEs were also encoded differently in brain-wide multivariate patterns, with a more sensitive neural signature observed within fronto-insular circuits for reward PE. We further identified a fronto-insular network encompassing the left anterior cingulate cortex, bilateral insula, left dmPFC and inferior frontal gyrus that encoded punishment decisions. In addition, a stronger fronto-insular pattern expression under reward PE predicted more punishment decisions. These findings underscore that reward and emotional violations interact to shape decisions in complex social interactions, while the underlying neurofunctional PEs computations are distinguishable.

Context changes retrieval of prospective outcomes during decision deliberation

Foreseeing the future outcomes is the art of decision-making. Substantial evidence shows that, during choice deliberation, the brain can retrieve prospective decision outcomes. However, decisions are seldom made in a vacuum. Context carries information that can radically affect the outcomes of a choice. Nevertheless, most investigations of retrieval processes examined decisions in isolation, disregarding the context in which they occur. Here, we studied how context shapes prospective outcome retrieval during deliberation. We designed a decision-making task where participants were presented with object-context pairs and made decisions which led to a certain outcome. We show during deliberation, likely outcomes were retrieved in transient patterns of neural activity, as early as 3 s before participants decided. The strength of prospective outcome retrieval explains participants' behavioral efficiency, but only when context affects the decision outcome. Our results suggest context imparts strong constraints on retrieval processes and how neural representations are shaped during decision-making.

Representational spaces in orbitofrontal and ventromedial prefrontal cortex: task states, values, and beyond

The orbitofrontal cortex (OFC) and ventromedial-prefrontal cortex (vmPFC) play a key role in decision-making and encode task states in addition to expected value. We review evidence suggesting a connection between value and state representations and argue that OFC / vmPFC integrate stimulus, context, and outcome information. Comparable encoding principles emerge in late layers of deep reinforcement learning (RL) models, where single nodes exhibit similar forms of mixed-selectivity, which enables flexible readout of relevant variables by downstream neurons. Based on these lines of evidence, we suggest that outcome-maximization leads to complex representational spaces that are insufficiently characterized by linear value signals that have been the focus of most prior research on the topic. Major outstanding questions concern the role of OFC/ vmPFC in learning across tasks, in encoding of task-irrelevant aspects, and the role of hippocampus-PFC interactions.

Frontal midline theta power during the cue-target-interval reflects increased cognitive effort in rewarded task-switching

Cognitive performance largely depends on how much effort is invested during task-execution. This also means that we rarely perform as good as we could. Cognitive effort is adjusted to the expected outcome of performance, meaning that it is driven by motivation. The results from recent studies suggest that the expenditure of cognitive control is particularly prone to being affected by modulations of cognitive effort. Although recent EEG studies investigated the neural underpinnings of the interaction of effort and control, reports on how cognitive effort is reflected by oscillatory activity of the EEG are quite sparse. It is the goal of the present study to bridge this gap by performing an exploratory analysis of high-density EEG data from a switching-task using manipulations of monetary incentives. A beamformer approach is used to localize the sensor-level effects in source-space. The results indicate that the manipulation of cognitive effort was successful. The participants reported significantly higher motivation and cognitive effort in high versus low reward trials. Performance was also significantly increased. The analysis of the EEG data revealed that the increase of cognitive effort was reflected by an increased mid-frontal theta activity during the cue-target interval, suggesting an increased use of proactive control. This interpretation is supported by the result from a regression analysis performed on single-trial data, showing higher mid-frontal theta power prior to target-onset being associated with faster responses. Alpha-desynchronization throughout the trial was also more pronounced in high reward trials, signaling a bias of attention towards the processing of external stimuli. Source reconstruction suggests that these effects are located in areas related to cognitive control, and visual processing.

Common neural choice signals can emerge artefactually amid multiple distinct value signals

Previous work has identified characteristic neural signatures of value-based decision-making, including neural dynamics that closely resemble the ramping evidence accumulation process believed to underpin choice. Here we test whether these signatures of the choice process can be temporally dissociated from additional, choice-'independent' value signals. Indeed, EEG activity during value-based choice revealed distinct spatiotemporal clusters, with a stimulus-locked cluster reflecting affective reactions to choice sets and a response-locked cluster reflecting choice difficulty. Surprisingly, 'neither' of these clusters met the criteria for an evidence accumulation signal. Instead, we found that stimulus-locked activity can 'mimic' an evidence accumulation process when aligned to the response. Re-analysing four previous studies, including three perceptual decision-making studies, we show that response-locked signatures of evidence accumulation disappear when stimulus-locked and response-locked activity are modelled jointly. Collectively, our findings show that neural signatures of value can reflect choice-independent processes and look deceptively like evidence accumulation.

Motivational context determines the impact of aversive outcomes on mental effort allocation

It is well known that people will exert effort on a task if sufficiently motivated, but how they distribute these efforts across different strategies (e.g., efficiency vs. caution) remains uncertain. Past work has shown that people invest effort differently for potential positive outcomes (rewards) versus potential negative outcomes (penalties). However, this research failed to account for differences in the context in which negative outcomes motivate someone - either as punishment or reinforcement. It is therefore unclear whether effort profiles differ as a function of outcome valence, motivational context, or both. Using computational modeling and our novel Multi-Incentive Control Task, we show that the influence of aversive outcomes on one's effort profile is entirely determined by their motivational context. Participants (N:91) favored increased caution in response to larger penalties for incorrect responses, and favored increased efficiency in response to larger reinforcement for correct responses, whether positively or negatively incentivized. STATEMENT OF RELEVANCE: People have to constantly decide how to allocate their mental effort, and in doing so can be motivated by both the positive outcomes that effort accrues and the negative outcomes that effort avoids. For example, someone might persist on a project for work in the hopes of being promoted or to avoid being reprimanded or even fired. Understanding how people weigh these different types of incentives is critical for understanding variability in human achievement as well as sources of motivational impairments (e.g., in major depression). We show that people not only consider both potential positive and negative outcomes when allocating mental effort, but that the profile of effort they engage under negative incentives differs depending on whether that outcome is contingent on sustaining good performance (negative reinforcement) or avoiding bad performance (punishment). Clarifying the motivational factors that determine effort exertion is an important step for understanding motivational impairments in psychopathology.

Food Avoidance and Aversive Goal Value Computation in Anorexia Nervosa

Anorexia nervosa (AN) is associated with food restriction and significantly low body weight, but the neurobiology of food avoidance in AN is unknown. Animal research suggests that food avoidance can be triggered by conditioned fear that engages the anterior cingulate and nucleus accumbens. We hypothesized that the neural activation during food avoidance in AN could be modeled based on aversive goal value processing. Nineteen females with AN and thirty healthy controls matched for age underwent functional magnetic resonance brain imaging while conducting a food avoidance task. During active control free-bid and computer-generated forced-bid trials, participants bid money to avoid eating food items. Brain activation was parametrically modulated with the trial-by-trial placed bids. During free-bid trials, the AN group engaged the caudate nucleus, nucleus accumbens, ventral anterior cingulate, and inferior and medial orbitofrontal cortex more than the control group. High- versus low-bid trials in the AN group were associated with higher caudate nucleus response. Emotion dysregulation and intolerance of uncertainty scores were inversely associated with nucleus accumbens free-bid trial brain response in AN. This study supports the idea that food avoidance behavior in AN involves aversive goal value computation in the nucleus accumbens, caudate nucleus, anterior cingulate, and orbitofrontal cortex.

Goal commitment is supported by vmPFC through selective attention

When striking a balance between commitment to a goal and flexibility in the face of better options, people often demonstrate strong goal perseveration. Here, using functional MRI (n = 30) and lesion patient (n = 26) studies, we argue that the ventromedial prefrontal cortex (vmPFC) drives goal commitment linked to changes in goal-directed selective attention. Participants performed an incremental goal pursuit task involving sequential decisions between persisting with a goal versus abandoning progress for better alternative options. Individuals with stronger goal perseveration showed higher goal-directed attention in an interleaved attention task. Increasing goal-directed attention also affected abandonment decisions: while pursuing a goal, people lost their sensitivity to valuable alternative goals while remaining more sensitive to changes in the current goal. In a healthy population, individual differences in both commitment biases and goal-oriented attention were predicted by baseline goal-related activity in the vmPFC. Among lesion patients, vmPFC damage reduced goal commitment, leading to a performance benefit.

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

Dissociable Contributions of the Medial Parietal Cortex to Recognition Memory

Human neuroimaging studies of episodic memory retrieval routinely observe the engagement of specific cortical regions beyond the medial temporal lobe. Of these, medial parietal cortex (MPC) is of particular interest given its distinct functional characteristics during different retrieval tasks. Specifically, while recognition and autobiographical recall tasks are both used to probe episodic retrieval, these paradigms consistently drive distinct spatial patterns of response within MPC. However, other studies have emphasized alternate MPC functional dissociations in terms of brain network connectivity profiles or stimulus category selectivity. As the unique contributions of MPC to episodic memory remain unclear, adjudicating between these different accounts can provide better consensus regarding MPC function. Therefore, we used a precision-neuroimaging dataset (7T functional magnetic resonance imaging) to examine how MPC regions are differentially engaged during recognition memory and how these task-related dissociations may also reflect distinct connectivity and stimulus category functional profiles. We observed interleaved, though spatially distinct, subregions of MPC where responses were sensitive to either recognition decisions or the semantic representation of stimuli. In addition, this dissociation was further accentuated by functional subregions displaying distinct profiles of connectivity with the hippocampus during task and rest. Finally, we show that recent observations of dissociable person and place selectivity within the MPC reflect category-specific responses from within identified semantic regions that are sensitive to mnemonic demands. Together, by examining precision functional mapping within individuals, these data suggest that previously distinct observations of functional dissociation within MPC conform to a common principle of organization throughout hippocampal-neocortical memory systems.

Mutual inclusivity improves decision-making by smoothing out choice's competitive edge

Decisions form a central bottleneck to most tasks, one that people often experience as costly. Past work proposes mitigating those costs by lowering one's threshold for deciding. Here, we test an alternative solution, one that targets the basis for most choice costs: that choosing one option sacrifices others (mutual exclusivity). Across 5 studies (N = 462), we test whether this tension can be relieved by framing choices as inclusive (allowing selection of more than one option, as in buffets). We find that inclusivity makes choices more efficient, by selectively reducing competition between potential responses as participants accumulate information for each of their options. Inclusivity also made participants feel less conflicted, especially when they couldn't decide which good option to keep or which bad option to get rid of. These inclusivity benefits were also distinguishable from the effects of manipulating decision threshold (increased urgency), which improved choices but not experiences thereof.

A confidence framing effect: Flexible use of evidence in metacognitive monitoring

Human behavior is flexibly regulated by specific goals of cognitive tasks. One notable example is goal-directed modulation of metacognitive behavior, where logically equivalent decision-making problems can yield different patterns of introspective confidence depending on the frame in which they are presented. While this observation highlights the important heuristic nature of metacognitive monitoring, computational mechanisms underlying this phenomenon remain elusive. We confirmed the confidence framing effect in two-alternative dot-number discrimination and in previously published preference-choice data, demonstrating distinctive confidence patterns between "choose more" or "choose less" frames. Formal model comparisons revealed a simple confidence heuristic behind this phenomenon, which assigns greater weight to chosen than unchosen stimulus evidence. This computation appears to be based on internal evidence constituted under specific task demands rather than physical stimulus intensity itself, a view justified in terms of ecological rationality. These results shed light on the adaptive nature of human decision-making and metacognitive monitoring.

Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex

The signed value and unsigned salience of reward prediction errors (RPEs) are critical to understanding reinforcement learning (RL) and cognitive control. Dorsomedial prefrontal cortex (dMPFC) and insula (INS) are key regions for integrating reward and surprise information, but conflicting evidence for both signed and unsigned activity has led to multiple proposals for the nature of RPE representations in these brain areas. Recently developed RL models allow neurons to respond differently to positive and negative RPEs. Here, we use intracranially recorded high frequency activity (HFA) to test whether this flexible asymmetric coding strategy captures RPE coding diversity in human INS and dMPFC. At the region level, we found a bias towards positive RPEs in both areas which paralleled behavioral adaptation. At the local level, we found spatially interleaved neural populations responding to unsigned RPE salience and valence-specific positive and negative RPEs. Furthermore, directional connectivity estimates revealed a leading role of INS in communicating positive and unsigned RPEs to dMPFC. These findings support asymmetric coding across distinct but intermingled neural populations as a core principle of RPE processing and inform theories of the role of dMPFC and INS in RL and cognitive control.

Motivational context determines the impact of aversive outcomes on mental effort allocation

It is well known that people will exert effort on a task if sufficiently motivated, but how they distribute these efforts across different strategies (e.g., efficiency vs. caution) remains uncertain. Past work has shown that people invest effort differently for potential positive outcomes (rewards) versus potential negative outcomes (penalties). However, this research failed to account for differences in the context in which negative outcomes motivate someone - either as punishment or reinforcement. It is therefore unclear whether effort profiles differ as a function of outcome valence, motivational context, or both. Using computational modeling and our novel Multi-Incentive Control Task, we show that the influence of aversive outcomes on one's effort profile is entirely determined by their motivational context. Participants (N:91) favored increased caution in response to larger penalties for incorrect responses, and favored increased efficiency in response to larger reinforcement for correct responses, whether positively or negatively incentivized.

A goal-centric outlook on learning

Goals play a central role in human cognition. However, computational theories of learning and decision-making often take goals as given. Here, we review key empirical findings showing that goals shape the representations of inputs, responses, and outcomes, such that setting a goal crucially influences the central aspects of any learning process: states, actions, and rewards. We thus argue that studying goal selection is essential to advance our understanding of learning. By following existing literature in framing goal selection within a hierarchy of decision-making problems, we synthesize important findings on the principles underlying goal value attribution and exploration strategies. Ultimately, we propose that a goal-centric perspective will help develop more complete accounts of learning in both biological and artificial agents.

Dissociable contributions of the medial parietal cortex to recognition memory

Human neuroimaging studies of episodic memory retrieval routinely observe the engagement of specific cortical regions beyond the medial temporal lobe. Of these, medial parietal cortex (MPC) is of particular interest given its ubiquitous, and yet distinct, functional characteristics during different types of retrieval tasks. Specifically, while recognition memory and autobiographical recall tasks are both used to probe episodic retrieval, these paradigms consistently drive distinct patterns of response within MPC. This dissociation adds to growing evidence suggesting a common principle of functional organization across memory related brain structures, specifically regarding the control or content demands of memory-based decisions. To carefully examine this putative organization, we used a high-resolution fMRI dataset collected at ultra-high field (7T) while subjects performed thousands of recognition-memory trials to identify MPC regions responsive to recognition-decisions or semantic content of stimuli within and across individuals. We observed interleaving, though distinct, functional subregions of MPC where responses were sensitive to either recognition decisions or the semantic representation of stimuli, but rarely both. In addition, this functional dissociation within MPC was further accentuated by distinct profiles of connectivity bias with the hippocampus during task and rest. Finally, we show that recent observations of person and place selectivity within MPC reflect category specific responses from within identified semantic regions that are sensitive to mnemonic demands. Together, these data better account for how distinct patterns of MPC responses can occur as a result of task demands during episodic retrieval and may reflect a common principle of organization throughout hippocampal-neocortical memory systems.

Intrinsic rewards explain context-sensitive valuation in reinforcement learning

When observing the outcome of a choice, people are sensitive to the choice's context, such that the experienced value of an option depends on the alternatives: getting $1 when the possibilities were 0 or 1 feels much better than when the possibilities were 1 or 10. Context-sensitive valuation has been documented within reinforcement learning (RL) tasks, in which values are learned from experience through trial and error. Range adaptation, wherein options are rescaled according to the range of values yielded by available options, has been proposed to account for this phenomenon. However, we propose that other mechanisms-reflecting a different theoretical viewpoint-may also explain this phenomenon. Specifically, we theorize that internally defined goals play a crucial role in shaping the subjective value attributed to any given option. Motivated by this theory, we develop a new "intrinsically enhanced" RL model, which combines extrinsically provided rewards with internally generated signals of goal achievement as a teaching signal. Across 7 different studies (including previously published data sets as well as a novel, preregistered experiment with replication and control studies), we show that the intrinsically enhanced model can explain context-sensitive valuation as well as, or better than, range adaptation. Our findings indicate a more prominent role of intrinsic, goal-dependent rewards than previously recognized within formal models of human RL. By integrating internally generated signals of reward, standard RL theories should better account for human behavior, including context-sensitive valuation and beyond.

Task state representations in vmPFC mediate relevant and irrelevant value signals and their behavioral influence

The ventromedial prefrontal-cortex (vmPFC) is known to contain expected value signals that inform our choices. But expected values even for the same stimulus can differ by task. In this study, we asked how the brain flexibly switches between such value representations in a task-dependent manner. Thirty-five participants alternated between tasks in which either stimulus color or motion predicted rewards. We show that multivariate vmPFC signals contain a rich representation that includes the current task state or context (motion/color), the associated expected value, and crucially, the irrelevant value of the alternative context. We also find that irrelevant value representations in vmPFC compete with relevant value signals, interact with task-state representations and relate to behavioral signs of value competition. Our results shed light on vmPFC's role in decision making, bridging between its role in mapping observations onto the task states of a mental map, and computing expected values for multiple states.

Episodic Memory Retrieval Affects the Onset and Dynamics of Evidence Accumulation during Value-based Decisions

In neuroeconomics, there is much interest in understanding simple value-based choices where agents choose between visually presented goods, comparable to a shopping scenario in a supermarket. However, many everyday decisions are made in the physical absence of the considered goods, requiring agents to recall information about the goods from memory. Here, we asked whether and how this reliance on an internal memory representation affects the temporal dynamics of decision making on a cognitive and neural level. Participants performed a remember-and-decide task in which they made simple purchasing decisions between money offers and snack items while undergoing EEG. Snack identity was presented either visually (value trials) or had to be recalled from memory (memory trials). Behavioral data indicated comparable choice consistency across both trial types, but considerably longer RTs in memory trials. Drift-diffusion modeling suggested that this RT difference was because of longer nondecision time of decision processes as well as altered evidence accumulation dynamics (lower accumulation rate and higher decision threshold). The nondecision time effect was supported by a delayed onset of the lateralized readiness potential. These results show that both decision and nondecision processes are prolonged when participants need to resort to internal memory representations during value-based decisions.

A tripartite view of the posterior cingulate cortex

The posterior cingulate cortex (PCC) is one of the least understood regions of the cerebral cortex. By contrast, the anterior cingulate cortex has been the subject of intensive investigation in humans and model animal systems, leading to detailed behavioural and computational theoretical accounts of its function. The time is right for similar progress to be made in the PCC given its unique anatomical and physiological properties and demonstrably important contributions to higher cognitive functions and brain diseases. Here, we describe recent progress in understanding the PCC, with a focus on convergent findings across species and techniques that lay a foundation for establishing a formal theoretical account of its functions. Based on this converging evidence, we propose that the broader PCC region contains three major subregions - the dorsal PCC, ventral PCC and retrosplenial cortex - that respectively support the integration of executive, mnemonic and spatial processing systems. This tripartite subregional view reconciles inconsistencies in prior unitary theories of PCC function and offers promising new avenues for progress.

Self-judgment dissected: A computational modeling analysis of self-referential processing and its relationship to trait mindfulness facets and depression symptoms

Cognitive theories of depression, and mindfulness theories of well-being, converge on the notion that self-judgment plays a critical role in mental health. However, these theories have rarely been tested via tasks and computational modeling analyses that can disentangle the information processes operative in self-judgments. We applied a drift-diffusion computational model to the self-referential encoding task (SRET) collected before and after an 8-week mindfulness intervention (n = 96). A drift-rate regression parameter representing positive-relative to negative-self-referential judgment strength positively related to mindful awareness and inversely related to depression, both at baseline and over time; however, this parameter did not significantly relate to the interaction between mindful awareness and nonjudgmentalness. At the level of individual depression symptoms, at baseline, a spectrum of symptoms (inversely) correlated with the drift-rate regression parameter, suggesting that many distinct depression symptoms relate to valenced self-judgment between subjects. By contrast, over the intervention, changes in only a smaller subset of anhedonia-related depression symptoms showed substantial relationships with this parameter. Both behavioral and model-derived measures showed modest split-half and test-retest correlations. Results support cognitive theories that implicate self-judgment in depression and mindfulness theories, which imply that mindful awareness should lead to more positive self-views.

Phantom controllers: Misspecified models create the false appearance of adaptive control during value-based choice

Decision scientists have grown increasingly interested in how people adaptively control their decision making. Researchers have demonstrated that parameters governing the accumulation of evidence towards a choice, such as the decision threshold, are shaped by information available prior to or in parallel with one's evaluation of an option set (e.g., recent outcomes or choice conflict). A recent account has taken a bold leap forward in this approach, suggesting that adjustments in decision parameters can be motivated by the value of the options under consideration. This motivated control account predicts that when faced with difficult choices (similarly valued options) under time pressure, people will adaptively lower their decision threshold to ensure that they make a choice in time. This account was supported by drift diffusion modeling of a deadlined choice task, demonstrating that decision thresholds decrease for difficult relative to easy choices. Here, we reanalyze the data from this experiment, and show that evidence for this novel account does not hold up to further scrutiny. Using a more systematic and comprehensive modeling approach, we show that this previously observed threshold adjustment disappears (or even reverses) under a more complete model of the data. Importantly, we further show how this and other apparent evidence for motivated control arises as an artifact of model (mis)specification, where one model's putatively controlled decision process (e.g., value-driven threshold adjustments) can mimic another model's stimulus-driven decision processes (e.g., accumulator competition or collapsing bounds). Collectively, this work reveals crucial insights and constraints in the pursuit of understanding how control guides decision-making, and when it doesn't.

Goals, usefulness and abstraction in value-based choice

Colombian drug lord Pablo Escobar, while on the run, purportedly burned two million dollars in banknotes to keep his daughter warm. A stark reminder that, in life, circumstances and goals can quickly change, forcing us to reassess and modify our values on-the-fly. Studies in decision-making and neuroeconomics have often implicitly equated value to reward, emphasising the hedonic and automatic aspect of the value computation, while overlooking its functional (concept-like) nature. Here we outline the computational and biological principles that enable the brain to compute the usefulness of an option or action by creating abstractions that flexibly adapt to changing goals. We present different algorithmic architectures, comparing ideas from artificial intelligence (AI) and cognitive neuroscience with psychological theories and, when possible, drawing parallels.

Aversive anticipations modulate electrocortical correlates of decision-making and reward reversal learning, but not behavioral performance

Predicting the consequences of one's own decisions is crucial for organizing future behavior. However, when reward contingencies vary frequently, flexible adaptation of decisions is likely to depend on the situation. We examined the effects of an instructed threat context on choice behavior (i.e., reversal learning) and its electrocortical correlates. In a probabilistic decision-making task, 30 participants had to choose between two options that were either contingent on monetary gains or losses. Reward contingencies were reversed after reaching a probabilistic threshold. Decision-making and reversal learning were examined with two contextual background colors, which were instructed as signals for threat-of-shock or safety. Self-report data confirmed the threat context as more unpleasant, arousing, and threatening relative to safety condition. However, against our expectations, behavioral performance was comparable during the threat and safety conditions (i.e., errors-to-criterion, number of reversal, error rates, and choice times). Regarding electrocortical activity, feedback processing changed throughout the visual processing stream. The feedback-related negativity (FRN) reflected expectancy-driven processing (unexpected vs. congruent losses and gains), and the threat-selective P3 component revealed non-specific discrimination of gains vs. losses. Finally, the late positive potentials (LPP) showed strongly valence-specific processing (unexpected and congruent losses vs. gains). Thus, regardless of contextual threat, early and late cortical activity reflects an attentional shift from expectation- to outcome-based feedback processing. Findings are discussed in terms of reward, threat, and reversal-learning mechanisms with implications for emotion regulation and anxiety disorders.

Rat Anterior Cingulate Cortex Continuously Signals Decision Variables in a Patch Foraging Task

In patch foraging tasks, animals must decide whether to remain with a depleting resource or to leave it in search of a potentially better source of reward. In such tasks, animals consistently follow the general predictions of optimal foraging theory (the marginal value theorem; MVT): to leave a patch when the reward rate in the current patch depletes to the average reward rate across patches. Prior studies implicate an important role for the anterior cingulate cortex (ACC) in foraging decisions based on MVT: within single trials, ACC activity increases immediately preceding foraging decisions, and across trials, these dynamics are modulated as the value of staying in the patch depletes to the average reward rate. Here, we test whether these activity patterns reflect dynamic encoding of decision-variables and whether these signals are directly involved in decision-making. We developed a leaky accumulator model based on the MVT that generates estimates of decision variables within and across trials, and tested model predictions against ACC activity recorded from male rats performing a patch foraging task. Model predicted changes in MVT decision variables closely matched rat ACC activity. Next, we pharmacologically inactivated ACC in male rats to test the contribution of these signals to decision-making. ACC inactivation had a profound effect on rats' foraging decisions and response times (RTs) yet rats still followed the MVT decision rule. These findings indicate that the ACC encodes foraging-related variables for reasons unrelated to patch-leaving decisions.SIGNIFICANCE STATEMENT The ability to make adaptive patch-foraging decisions, to remain with a depleting resource or search for better alternatives, is critical to animal well-being. Previous studies have found that anterior cingulate cortex (ACC) activity is modulated at different points in the foraging decision process, raising questions about whether the ACC guides ongoing decisions or serves a more general purpose of regulating cognitive control. To investigate the function of the ACC in foraging, the present study developed a dynamic model of behavior and neural activity, and tested model predictions using recordings and inactivation of ACC. Findings revealed that ACC continuously signals decision variables but that these signals are more likely used to monitor and regulate ongoing processes than to guide foraging decisions.

Filling the gaps: Cognitive control as a critical lens for understanding mechanisms of value-based decision-making

While often seeming to investigate rather different problems, research into value-based decision making and cognitive control have historically offered parallel insights into how people select thoughts and actions. While the former studies how people weigh costs and benefits to make a decision, the latter studies how they adjust information processing to achieve their goals. Recent work has highlighted ways in which decision-making research can inform our understanding of cognitive control. Here, we provide the complementary perspective: how cognitive control research has informed understanding of decision-making. We highlight three particular areas of research where this critical interchange has occurred: (1) how different types of goals shape the evaluation of choice options, (2) how people use control to adjust the ways they make their decisions, and (3) how people monitor decisions to inform adjustments to control at multiple levels and timescales. We show how adopting this alternate viewpoint offers new insight into the determinants of both decisions and control; provides alternative interpretations for common neuroeconomic findings; and generates fruitful directions for future research.

High-value decisions are fast and accurate, inconsistent with diminishing value sensitivity

It is a widely held belief that people's choices are less sensitive to changes in value as value increases. For example, the subjective difference between $11 and $12 is believed to be smaller than between $1 and $2. This idea is consistent with applications of the Weber-Fechner Law and divisive normalization to value-based choice and with psychological interpretations of diminishing marginal utility. According to random utility theory in economics, smaller subjective differences predict less accurate choices. Meanwhile, in the context of sequential sampling models in psychology, smaller subjective differences also predict longer response times. Based on these models, we would predict decisions between high-value options to be slower and less accurate. In contrast, some have argued on normative grounds that choices between high-value options should be made with less caution, leading to faster and less accurate choices. Here, we model the dynamics of the choice process across three different choice domains, accounting for both discriminability and response caution. Contrary to predictions, we mostly observe faster and more accurate decisions (i.e., higher drift rates) between high-value options. We also observe that when participants are alerted about incoming high-value decisions, they exert more caution and not less. We rule out several explanations for these results, using tasks with both subjective and objective values. These results cast doubt on the notion that increasing value reduces discriminability.

Advances in modeling learning and decision-making in neuroscience

An organism's survival depends on its ability to learn about its environment and to make adaptive decisions in the service of achieving the best possible outcomes in that environment. To study the neural circuits that support these functions, researchers have increasingly relied on models that formalize the computations required to carry them out. Here, we review the recent history of computational modeling of learning and decision-making, and how these models have been used to advance understanding of prefrontal cortex function. We discuss how such models have advanced from their origins in basic algorithms of updating and action selection to increasingly account for complexities in the cognitive processes required for learning and decision-making, and the representations over which they operate. We further discuss how a deeper understanding of the real-world complexities in these computations has shed light on the fundamental constraints on optimal behavior, and on the complex interactions between corticostriatal pathways to determine such behavior. The continuing and rapid development of these models holds great promise for understanding the mechanisms by which animals adapt to their environments, and what leads to maladaptive forms of learning and decision-making within clinical populations.

Executive Function Assigns Value to Novel Goal-Congruent Outcomes

People often learn from the outcomes of their actions, even when these outcomes do not involve material rewards or punishments. How does our brain provide this flexibility? We combined behavior, computational modeling, and functional neuroimaging to probe whether learning from abstract novel outcomes harnesses the same circuitry that supports learning from familiar secondary reinforcers. Behavior and neuroimaging revealed that novel images can act as a substitute for rewards during instrumental learning, producing reliable reward-like signals in dopaminergic circuits. Moreover, we found evidence that prefrontal correlates of executive control may play a role in shaping flexible responses in reward circuits. These results suggest that learning from novel outcomes is supported by an interplay between high-level representations in prefrontal cortex and low-level responses in subcortical reward circuits. This interaction may allow for human reinforcement learning over arbitrarily abstract reward functions.

Magnitude-sensitivity: rethinking decision-making

Magnitude-sensitivity refers to the result that performance in decision-making, across domains and organisms, is affected by the total value of the possible alternatives. This simple result offers a window into fundamental issues in decision-making and has led to a reconsideration of ecological decision-making, prominent computational models of decision-making, and optimal decision-making. Moreover, magnitude-sensitivity has inspired the design of new robotic systems that exploit natural solutions and apply optimal decision-making policies. In this article, we review the key theoretical and empirical results about magnitude-sensitivity and highlight the importance that this phenomenon has for the understanding of decision-making. Furthermore, we discuss open questions and ideas for future research.

Evaluating the learning of stimulus-control associations through incidental memory of reinforcement events

Cognitive control describes the ability to use internal goals to strategically guide how we process and respond to our environment. Changes in the environment lead to adaptation in control strategies. This type of control learning can be observed in performance adjustments in response to varying proportions of easy to hard trials over blocks of trials on classic control tasks. Known as the list-wide proportion congruent (LWPC) effect, increased difficulty is met with enhanced attentional control. Recent research has shown that motivational manipulations may enhance the LWPC effect, but the underlying mechanisms are not yet understood. We manipulated Stroop proportion congruency over blocks of trials and, after each trial, provided participants with either performance-contingent feedback ("correct/incorrect") or noncontingent feedback ("response logged") above trial-unique, task-irrelevant images (reinforcement events). The LWPC task was followed by a surprise recognition memory task, which allowed us to test whether attention to feedback (incidental memory for the images) varies as a function of proportion congruency, time, performance contingency, and individual differences. We replicated a robust LWPC effect in a large sample (N = 402) but observed no differences in behavior between feedback groups. Importantly, the memory data revealed better encoding of feedback images from context-defining trials (e.g., congruent trials in a mostly congruent block), especially early in a new context and in congruent conditions. Individual differences in reward and punishment sensitivity were not strongly associated with control-learning effects. These results suggest that statistical learning of contextual demand may have a larger impact on control learning than individual differences in motivation. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience

Reinforcement learning (RL) is a concept that has been invaluable to fields including machine learning, neuroscience, and cognitive science. However, what RL entails differs between fields, leading to difficulties when interpreting and translating findings. After laying out these differences, this paper focuses on cognitive (neuro)science to discuss how we as a field might over-interpret RL modeling results. We too often assume-implicitly-that modeling results generalize between tasks, models, and participant populations, despite negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique (neuro)cognitive processes, a concept we call interpretability, when evidence suggests that they capture different functions across studies and tasks. We conclude that future computational research needs to pay increased attention to implicit assumptions when using RL models, and suggest that a more systematic understanding of contextual factors will help address issues and improve the ability of RL to explain brain and behavior.

Neural encoding of perceived patch value during competitive and hazardous virtual foraging

Natural observations suggest that in safe environments, organisms avoid competition to maximize gain, while in hazardous environments the most effective survival strategy is to congregate with competition to reduce the likelihood of predatory attack. We probed the extent to which survival decisions in humans follow these patterns, and examined the factors that determined individual-level decision-making. In a virtual foraging task containing changing levels of competition in safe and hazardous patches with virtual predators, we demonstrate that human participants inversely select competition avoidant and risk diluting strategies depending on perceived patch value (PPV), a computation dependent on reward, threat, and competition. We formulate a mathematically grounded quantification of PPV in social foraging environments and show using multivariate fMRI analyses that PPV is encoded by mid-cingulate cortex (MCC) and ventromedial prefrontal cortices (vMPFC), regions that integrate action and value signals. Together, these results suggest humans utilize and integrate multidimensional information to adaptively select patches highest in PPV, and that MCC and vMPFC play a role in adapting to both competitive and predatory threats in a virtual foraging setting.

Humans adapt decision strategies in response to environmental demands. Here the authors show that decisions in a virtual foraging task can be modelled based on perceived patch value, which includes reward, competition and threat, and is associated with activity in ventromedial prefrontal and medial cingulate cortices.

Mental representations distinguish value-based decisions from perceptual decisions

In our daily lives, we make a wide variety of decisions. One major distinction that has been made is between perceptual decisions and value-based (economic) decisions. We argue that this distinction is ill-defined, because these decisions vary on multiple dimensions. We present an alternative way to categorize decisions, based on two dimensions: subjective versus objective criteria, and evaluation of a stimulus versus a representation. We experimentally study the decision-making process (with eye-tracking) in each of the four resulting categories, using the same stimulus set of food images. Using a combination of individual-level and group-level modeling, we find surprisingly consistent patterns of behavior across the categories. However, we find stronger similarities between the subjective and objective categories, and stronger differences between the stimulus and representation categories.

Fixation patterns in simple choice reflect optimal information sampling

Simple choices (e.g., eating an apple vs. an orange) are made by integrating noisy evidence that is sampled over time and influenced by visual attention; as a result, fluctuations in visual attention can affect choices. But what determines what is fixated and when? To address this question, we model the decision process for simple choice as an information sampling problem, and approximate the optimal sampling policy. We find that it is optimal to sample from options whose value estimates are both high and uncertain. Furthermore, the optimal policy provides a reasonable account of fixations and choices in binary and trinary simple choice, as well as the differences between the two cases. Overall, the results show that the fixation process during simple choice is influenced dynamically by the value estimates computed during the decision process, in a manner consistent with optimal information sampling.

Does Attention Increase the Value of Choice Alternatives?

A growing recognition of the role of attention in decision-making has been driven by both the technology of eye tracking and the development of models that explicitly incorporate attention. One result of this convergence is the arresting claim that attention, by itself, can increase the perceived value of a decision alternative. In this review, we cover the origins of that claim, its empirical foundation, and the reasoning that supports it. The conclusion is that, to date, there is not sufficient evidence to support the claim. Alternative explanations for the extant evidentiary base are discussed, as is the balance between the bottom-up influence of empirical evidence and the top-down commitment to a conceptual framework.

Visual attention modulates the integration of goal-relevant evidence and not value

When choosing between options, such as food items presented in plain view, people tend to choose the option they spend longer looking at. The prevailing interpretation is that visual attention increases value. However, in previous studies, 'value' was coupled to a behavioural goal, since subjects had to choose the item they preferred. This makes it impossible to discern if visual attention has an effect on value, or, instead, if attention modulates the information most relevant for the goal of the decision-maker. Here, we present the results of two independent studies-a perceptual and a value-based task-that allow us to decouple value from goal-relevant information using specific task-framing. Combining psychophysics with computational modelling, we show that, contrary to the current interpretation, attention does not boost value, but instead it modulates goal-relevant information. This work provides a novel and more general mechanism by which attention interacts with choice.

Imaging the "social brain" in schizophrenia: A systematic review of neuroimaging studies of social reward and punishment

Decreased social functioning and high levels of loneliness and social isolation are common in schizophrenia spectrum disorders (SSD), contributing to reduced quality of life. One key contributor to social impairment is low social motivation, which may stem from aberrant neural processing of socially rewarding or punishing stimuli. To summarize research on the neurobiology of social motivation in SSD, we performed a systematic literature review of neuroimaging studies involving the presentation of social stimuli intended to elicit feelings of reward and/or punishment. Across 11 studies meeting criteria, people with SSD demonstrated weaker modulation of brain activity in regions within a proposed social interaction network, including prefrontal, cingulate, and striatal regions, as well as the amygdala and insula. Firm conclusions regarding neural differences in SSD in these regions, as well as connections within networks, are limited due to conceptual and methodological inconsistencies across the available studies. We conclude by making recommendations for the study of social reward and punishment processing in SSD in future research.

Formation of global self-beliefs in the human brain

Momentary feelings of confidence accompany many of our actions and decisions. In addition to such “local” feelings of confidence, we also construct “global” confidence estimates about our skills and abilities (global self-performance estimates or SPEs). Distorted SPEs may have a pervasive impact on motivation and self-evaluation, for instance affecting estimates of our competitiveness at work or in a sports team. Here, we found that components of a brain network previously implicated in the tracking of local confidence was additionally modulated by SPE level, whereas ventral striatum tracked SPEs irrespective of confidence. Our findings of a neurocognitive basis for global SPEs lay the groundwork for understanding how distorted SPEs arise in educational and clinical settings.

Humans create metacognitive beliefs about their performance across many levels of abstraction—from local confidence in individual decisions to global estimates of our skills and abilities. Despite a rich literature on the neural basis of local confidence judgements, how global self-performance estimates (SPEs) are constructed remains unknown. Using functional magnetic resonance imaging, we scanned human subjects while they performed several short blocks of tasks and reported on which task they think they performed best, providing a behavioral proxy for global SPEs. In a frontoparietal network sensitive to fluctuations in local confidence, we found that activity within ventromedial prefrontal cortex and precuneus was additionally modulated by global SPEs. In contrast, activity in ventral striatum was associated with subjects’ global SPEs irrespective of fluctuations in local confidence, and predicted the extent to which global SPEs tracked objective task difficulty across individuals. Our findings reveal neural representations of global SPEs that go beyond the tracking of local confidence, and lay the groundwork for understanding how a formation of global self-beliefs may go awry in conditions characterized by distorted self-evaluation.

Under construction: ventral and lateral frontal lobe contributions to value-based decision-making and learning

Even apparently simple choices, like selecting a dessert in a pastry shop, involve options characterized by multiple motivationally relevant attributes. Neuroeconomic research suggests that the human brain may track the subjective value of such options, allowing disparate reward-predictive information to be compared in a common currency. However, the brain mechanisms involved in identifying value-predictive features and combining these to assess the value of each decision option remain unclear. Here, we review recent evidence from studies of multi-attribute decision-making in people with focal frontal lobe damage and in healthy people undergoing functional magnetic resonance imaging. This work suggests that ventromedial and lateral prefrontal cortex and orbitofrontal cortex are important for forming value judgments under conditions of complexity. We discuss studies supporting the involvement of these regions in selecting among and evaluating option attributes during value judgment and decision-making and when learning from reward feedback. These findings are consistent with roles for these regions in guiding value construction. They argue for a more nuanced understanding of how ventral and lateral prefrontal cortex contribute to discovering and recognizing value, processes that are required under the complex conditions typical of many everyday decisions.