Flexible control as surrogate reward or dynamic reward maximization

The utility of a given experience, like interacting with a particular friend or tasting a particular food, fluctuates continually according to homeostatic and hedonic principles. Consequently, to maximize reward, an individual must be able to escape or attain outcomes as preferences change, by switching between actions. Recent work on human and artificial intelligence has defined such flexible instrumental control in information theoretic terms and postulated that it may serve as a reward surrogate. Another possibility, however, is that the adaptability afforded by flexible control is tacitly implemented by planning for dynamic changes in outcome values. In the current study, an expected utility model that computes decision values over a range of possible monetary gains and losses associated with sensory outcomes provided the best fit to behavioral choice data and performed best in terms of earned rewards. Moreover, consistent with previous work on perceived control and personality, individual differences in dimensional schizotypy were correlated with behavioral choice preferences in conditions with the greatest and lowest levels of flexible control. These results contribute to a growing literature on the role of instrumental control in goal-directed choice.

Analytic Causal Knowledge for Constructing Useable Empirical Causal Knowledge: Two Experiments on Pre-schoolers

The present paper examines a type of abstract domain-general knowledge required for the process of constructing useable domain-specific causal knowledge, the evident goal of causal learning. It tests the hypothesis that analytic knowledge of causal-invariance decomposition functions is essential for this process. Such knowledge specifies the decomposition of an observed outcome into contributions from constituent causes under the default assumption that the empirical knowledge acquired is invariant across contextual/background causes. The paper reports two psychological experiments (and replication studies) with pre-school-age children on generalization across contexts involving binary cause and effect variables. The critical role of causal invariance for constructing useable causal knowledge predicts that even young children should (tacitly) use the causal-invariance decomposition function for such variables rather than a non-causal-invariance decomposition function common in statistical practice in research involving binary outcomes. The findings support the rational shaping of empirical causal knowledge by the causal-invariance constraint, ruling out alternative explanations in terms of non-causal-invariance decomposition functions, heuristics, and biases. For the same causal structure involving candidate causes and outcomes that are binary variables with a "present" value and an "absent" value, the paper argues against the possibility of multiple rational characterizations of the "sameness of causal influence" that justifies generalization across contexts.

Stakes and expertise modulate conformity in economic choice

The influences of expertise and group size on an individual’s tendency to align with a majority opinion have been attributed to informational and normative conformity, respectively: Whereas the former refers to the treatment of others’ decisions as proxies for outcomes, the latter involves positive affect elicited by group membership. In this study, using a social gambling task, we pitted alignment with a high- vs. low-expertise majority against a hypothetical monetary reward, thus relating conformity to a broader literature on valuation and choice, and probed the countering influence of a high-expertise minority opinion. We found that the expertise of a countering minority group significantly modulated alignment with a low-expertise majority, but only if such alignment did not come at a cost. Conversely, participants’ knowledge of payoff probabilities predicted the degree of majority alignment only when a high-expertise majority endorsed a more costly option. Implications for the relative influences of expertise and stakes on conformity are discussed.

Neural Correlates of Causal Confounding

As scientists, we are keenly aware that if putative causes perfectly covary, the independent influence of neither can be discerned-a "no confounding" constraint on inference, fundamental to philosophical and statistical perspectives on causation. Intriguingly, a substantial behavioral literature suggests that naïve human reasoners, adults and children, are tacitly sensitive to causal confounding. Here, a combination of fMRI and computational cognitive modeling was used to investigate neural substrates mediating such sensitivity. While being scanned, participants observed and judged the influences of various putative causes with confounded or nonconfounded, deterministic or stochastic, influences. During judgments requiring generalization of causal knowledge from a feedback-based learning context to a transfer probe, activity in the dorsomedial pFC was better accounted for by a Bayesian causal model, sensitive to both confounding and stochasticity, than a purely error-driven algorithm, sensitive only to stochasticity. Implications for the detection and estimation of distinct forms of uncertainty, and for a neural mediation of domain-general constraints on causal induction, are discussed.

Mesolimbic dopamine projections mediate cue-motivated reward seeking but not reward retrieval in rats

Efficient foraging requires an ability to coordinate discrete reward-seeking and reward-retrieval behaviors. We used pathway-specific chemogenetic inhibition to investigate how rats’ mesolimbic and mesocortical dopamine circuits contribute to the expression and modulation of reward seeking and retrieval. Inhibiting ventral tegmental area dopamine neurons disrupted the tendency for reward-paired cues to motivate reward seeking, but spared their ability to increase attempts to retrieve reward. Similar effects were produced by inhibiting dopamine inputs to nucleus accumbens, but not medial prefrontal cortex. Inhibiting dopamine neurons spared the suppressive effect of reward devaluation on reward seeking, an assay of goal-directed behavior. Attempts to retrieve reward persisted after devaluation, indicating they were habitually performed as part of a fixed action sequence. Our findings show that complete bouts of reward seeking and retrieval are behaviorally and neurally dissociable from bouts of reward seeking without retrieval. This dichotomy may prove useful for uncovering mechanisms of maladaptive behavior.

The Expression and Transfer of Valence Associated with Social Conformity

Consensus seeking – abandoning one’s own judgment to align with a group majority – is a fundamental feature of human social interaction. Notably, such striving for majority affiliation often occurs in the absence of any apparent economic or social gain, suggesting that achieving consensus might have intrinsic value. Here, using a simple gambling task, in which the decisions of ostensible previous gamblers were indicated below available options on each trial, we examined the affective properties of agreeing with a group majority by assessing the trade-off between social and non-social currencies, and the transfer of social valence to concomitant stimuli. In spite of demonstrating near-perfect knowledge of objective reward probabilities, participant’s choices and evaluative judgments reflected a reliable preference for conformity, consistent with the hypothesized value of social alignment.

Selective impairment of goal-directed decision-making following lesions to the human ventromedial prefrontal cortex

See Manohar and Akam (doi10.1093/brain/awx119) for a scientific commentary on this article.

Instrumental Divergence and the Value of Control

A critical aspect of flexible choice is that alternative actions yield distinct consequences: Only when available action alternatives produce distinct outcome states does discrimination and selection between actions allow an agent to flexibly obtain the currently most desired outcome. Here, we use instrumental divergence - the degree to which alternative actions differ with respect to their outcome probability distributions - as an index of flexible instrumental control, and assess the influence of this novel decision variable on choice preference. In Experiment 1, when other decision variables, such as expected value and outcome entropy, were held constant, we found a significant preference for high instrumental divergence. In Experiment 2, we used an "auto- vs. self-play" manipulation to eliminate outcome diversity as a source of behavioral preferences, and to contrast flexible instrumental control with the complete absence of voluntary choice. Our results suggest that flexible instrumental control over decision outcomes may have intrinsic value.

When Is a Cause the “Same”?

Two competing psychological approaches to causal learning make different predictions regarding what aspect of perceived causality is generalized across contexts. Two experiments tested these predictions. In one experiment, the task required a judgment regarding the existence of a simple causal relation; in the other, the task required a judgment regarding the existence of an interaction between a candidate cause and unobserved background causes. The task materials did not mention assessments of causal strength. Results indicate that causal power (Cartwright, 1989; Cheng, 1997) is the mental construct that people carry from one context to another.

How multiple causes combine: independence constraints on causal inference

According to the causal power view, two core constraints-that causes occur independently (i.e., no confounding) and influence their effects independently-serve as boundary conditions for causal induction. This study investigated how violations of these constraints modulate uncertainty about the existence and strength of a causal relationship. Participants were presented with pairs of candidate causes that were either confounded or not, and that either interacted or exerted their influences independently. Consistent with the causal power view, uncertainty about the existence and strength of causal relationships was greater when causes were confounded or interacted than when unconfounded and acting independently. An elemental Bayesian causal model captured differences in uncertainty due to confounding but not those due to an interaction. Implications of distinct sources of uncertainty for the selection of contingency information and causal generalization are discussed.

Differentiating neural systems mediating the acquisition vs. expression of goal-directed and habitual behavioral control

Considerable behavioral data indicate that operant actions can become habitual, as demonstrated by insensitivity to changes in the action-outcome contingency and in subjective outcome values. Notably, although several studies have investigated the neural substrates of habits, none has clearly differentiated the areas of the human brain that support habit formation from those that implement habitual control. We scanned participants with functional magnetic resonance imaging as they learned and performed an operant task in which the conditional structure of the environment encouraged either goal-directed encoding of the consequences of actions, or a habit-like mapping of actions to antecedent cues. Participants were also scanned during a subsequent assessment of insensitivity to outcome devaluation. We identified dissociable roles of the cerebellum and ventral striatum, across learning and test performance, in behavioral insensitivity to outcome devaluation. We also showed that the inferior parietal lobule (an area previously implicated in several aspects of goal-directed action selection, including the attribution of intent and awareness of agency) predicted sensitivity to outcome devaluation. Finally, we revealed a potential functional homology between the human subgenual cortex and rodent infralimbic cortex in the implementation of habitual control. In summary, our findings suggested a broad systems division, at the cortical and subcortical levels, between brain areas mediating the encoding and expression of action-outcome and stimulus-response associations.

Contributions of the striatum to learning, motivation, and performance: an associative account

It has long been recognized that the striatum is composed of distinct functional sub-units that are part of multiple cortico-striatal-thalamic circuits. Contemporary research has focused on the contribution of striatal sub-regions to three main phenomena: learning of associations between stimuli, actions and rewards; selection between competing response alternatives; and motivational modulation of motor behavior. Recent proposals have argued for a functional division of the striatum along these lines, attributing, for example, learning to one region and performance to another. Here, we consider empirical data from human and animal studies, as well as theoretical notions from both the psychological and computational literatures, and conclude that striatal sub-regions instead differ most clearly in terms of the associations being encoded in each region.

The outcome of allo-HSCT for 92 patients with myelofibrosis in the Nordic countries

Between 1982 and 2009 a total of 92 patients with myelofibrosis (MF) in chronic phase underwent allo-SCT in nine Nordic transplant centers. Myeloablative conditioning (MAC) was given to 40 patients, and reduced intensity conditioning (RIC) was used in 52 patients. The mean age in the two groups at transplantation was 46±12 and 55±8 years, respectively (P<0.001). When adjustment for age differences was made, the survival of the patients treated with RIC was significantly better (P=0.003). Among the RIC patients, the survival was significantly (P=0.003) better for the patients with age <60 years (a 10-year survival close to 80%) than for the older patients. The type of stem cell donor did not significantly affect the survival. No significant difference was found in TRM at 100 days between the MAC- and the RIC-treated patients. The probability of survival at 5 years was 49% for the MAC-treated patients and 59% in the RIC group (P=0.125). Patients treated with RIC experienced significantly less aGVHD compared with patients treated with MAC (P<0.001). The OS at 5 years was 70, 59 and 41% for patients with Lille score 0, 1 and 2, respectively (P=0.038, when age adjustment was made). Twenty-one percent of the patients in the RIC group were given donor lymphocyte infusion because of incomplete donor chimerism, compared with none of the MAC-treated patients (P<0.002). Nine percent of the patients needed a second transplant because of graft failure, progressive disease or transformation to AML, with no significant difference between the groups. Our conclusions are (1) allo-SCT performed with RIC gives a better survival compared with MAC. (2) age over 60 years is strongly related to a worse outcome and (3) patients with higher Lille score had a shorter survival.

Extracting functional equivalence from reversing contingencies

In 2 experiments assessing acquired equivalence, human participants were initially presented with 4 cues, 2 of which were paired with 1 outcome and 2 of which were paired with a 2nd outcome. These contingencies were then reversed across several training blocks such that, although each cue was paired equally often with each of the two outcomes across blocks, cues A and B always signaled the same outcome within blocks (as did cues C and D). In both experiments, performance on a subsequent transfer discrimination was enhanced when participants were required to generalize between stimuli that had been paired with the same outcome within each block of training. Additional tests did not yield evidence of a bias toward a specific set of cue-outcome contingencies in either experiment. Moreover, interviews conducted at the end of Experiment 2 revealed that performance on the transfer discrimination was enhanced only in participants who discovered the equivalence relationships during initial training. The results challenge simple associative, and attentional, accounts of acquired equivalence and favor the view that this effect is mediated by comparisons of the similarity of adjacent cue-outcome structures.

The influence of virtual sample size on confidence and causal-strength judgments

The authors investigated whether confidence in causal judgments varies with virtual sample size--the frequency of cases in which the outcome is (a) absent before the introduction of a generative cause or (b) present before the introduction of a preventive cause. Participants were asked to evaluate the influence of various candidate causes on an outcome as well as to rate their confidence in those judgments. They were presented with information on the relative frequencies of the outcome given the presence and absence of various candidate causes. These relative frequencies, sample size, and the direction of the causal influence (generative vs. preventive) were manipulated. It was found that both virtual and actual sample size affected confidence. Further, confidence affected estimates of strength, but confidence and strength are dissociable. The results enable a consistent explanation of the puzzling previous finding that observed causal-strength ratings often deviated from the predictions of both of the 2 dominant models of causal strength.

The integrative function of the basal ganglia in instrumental conditioning

Recent research in instrumental conditioning has focused on the striatum, particularly the role of the dorsal striatum in the learning processes that contribute to instrumental performance in rats. This research has found evidence of what appear to be parallel, functionally and anatomically distinct circuits involving dorsomedial striatum (DMS) and dorsolateral striatum (DLS) that contribute to two independent instrumental learning processes. Evidence suggests that the formation of the critical action-outcome associations mediating goal-directed action are localized to the dorsomedial striatum, whereas the sensorimotor connections that control the performance of habitual actions are localized to the dorsolateral striatum. In addition to the dorsal striatum, these learning processes appear to engage distinct cortico-striatal networks and to be embedded in a complex of converging and partially segregated loops that constitute the cortico-striatal thalamo-cortical feedback circuit. As the entry point for the basal ganglia, cortical circuits involving the dorsal striatum are clearly in a position to control a variety of motor functions but, as recent studies of various neurodegenerative disorders have made clear, they are also involved in a number of cognitive and executive functions including action selection, planning, and decision-making.

Mediated conditioning versus retrospective revaluation in humans: the influence of physical and functional similarity of cues

Two experiments assessed whether similarity between the two elements of a compound would influence the degree of mediated extinction versus recovery from overshadowing in human causal judgements. In both Experiments 1 and 2, we assessed the influence of extinguishing one element of a two-element compound on judgements about the other element. In Experiment 1 we manipulated the physical similarity of the two elements of the compound; in Experiment 2, we used equivalence and distinctiveness pretraining in order to vary their functional similarity. We found that these procedures influenced mediated extinction and recovery from overshadowing as a function of both physical and acquired similarity and distinctiveness, respectively. The implications of these results for previously reported differences between humans and nonprimate animals are discussed.

Stimulus salience and retrospective revaluation

The effect of conditioning or extinguishing the more salient element of a previously reinforced compound on responding to the less salient element of that compound was assessed in rats. Experiment 1 established that the 2 elements making up an audiovisual compound differed significantly in salience. In Experiment 2A, compound conditioning was followed by either reinforcement or extinction of either the less or more salient element. On test, evidence of retrospective revaluation of the less salient element was found but not of the more salient element. In Experiment 2B, extinction of the more salient element was found to be more effective than its reinforcement in producing retrospective revaluation of the less salient element. The implications of these results are discussed.