Fast rule switching and slow rule updating in a perceptual categorization task

To adapt to a changing world, we must be able to switch between rules already learned and, at other times, learn rules anew. Often we must do both at the same time, switching between known rules while also constantly re-estimating them. Here, we show these two processes, rule switching and rule learning, rely on distinct but intertwined computations, namely fast inference and slower incremental learning. To this end, we studied how monkeys switched between three rules. Each rule was compositional, requiring the animal to discriminate one of two features of a stimulus and then respond with an associated eye movement along one of two different response axes. By modeling behavior, we found the animals learned the axis of response using fast inference (rule switching) while continuously re-estimating the stimulus-response associations within an axis (rule learning). Our results shed light on the computational interactions between rule switching and rule learning, and make testable neural predictions for these interactions.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

An Adaptive Motivation Approach to Understanding the 'How' and 'Why' of Wellbeing

A new model provides insight into the 'how' and 'why' of wellbeing to better understand the 'what'. Informed by evolutionary psychology and neuroscience, it proposes that systems for adaptive motivation underpin experiential and reflective wellbeing. The model proposes that the brain learns to predict situations, and errors arise between the predictions and experience. These prediction errors drive emotional experience, learning, motivation, decision-making, and the formation of wellbeing-relevant memories. The model differentiates four layers of wellbeing: objective, experiential, reflective, and narrative, which relate to the model in different ways. Constituents of wellbeing, human motives, and specific emotions integrate into the model. A simple computational implementation of the model reproduced several established wellbeing phenomena, including: the greater frequency of pleasant to unpleasant emotions, the stronger emotional salience of unpleasant emotions, hedonic adaptation to changes in circumstances, heritable influences on wellbeing, and affective forecasting errors. It highlights the importance of individual differences, and implies that high wellbeing will correlate with the experience of infrequent, routine, and predictable avoidance cues and frequent, varied, and novel approach cues. The model suggests that wellbeing arises directly from a system for adaptive motivation. This system functions like a mental dashboard that calls attention to situational changes and motivates the kinds of behaviours that gave humans a relative advantage in their ancestral environment. The model offers a set of fundamental principles and processes that may underlie diverse conceptualisations of wellbeing.

CASCADE your teaching session!

Medical education has seen a shift toward interactive teaching in small groups that actively involves students in learning. However, didactic teaching, despite drawbacks such as student isolation and low stimulation of critical thinking, is still a very commonly used teaching method. For didactic teaching to be effective, teachers must possess strategies and skills that enable them to teach effectively in large groups and increase students' knowledge retention. This can be achieved if class sessions are planned in advance with the aim of making them interactive and engaging with all students. In the following article, I write about my experiences using the CASCADE method, an acronym of the words "create," "assess," "sequence," "charisma," "activate," "discussion," and "e-device," which creates interactive didactic sessions.

DNA methylation changes following narrative exposure therapy in a randomized controlled trial with female former child soldiers

The aftermath of traumatization lives on in the neural and epigenetic traces creating a momentum of affliction in the psychological and social realm. Can psychotherapy reorganise these memories through changes in DNA methylation signatures? Using a randomised controlled parallel group design, we examined methylome-wide changes in saliva samples of 84 female former child soldiers from Eastern DR Congo before and six months after Narrative Exposure Therapy. Treatment predicted differentially methylated positions (DMPs) related to ALCAM, RIPOR2, AFAP1 and MOCOS. In addition, treatment associations overlapped at gene level with baseline clinical and social outcomes. Treatment related DMPs are involved in memory formation-the key agent in trauma focused treatments-and enriched for molecular pathways commonly affected by trauma related disorders. Results were partially replicated in an independent sample of 53 female former child soldiers from Northern Uganda. Our results suggest a molecular impact of psychological treatment in women with war-related childhood trauma.Trial registration: Addressing Heightened Levels of Aggression in Traumatized Offenders With Psychotherapeutic Means (ClinicalTrials.gov Identifier: NCT02992561, 14/12/2016).

Longitudinal network re-organization across learning and development

While it is well understood that the brain experiences changes across short-term experience/learning and long-term development, it is unclear how these two mechanisms interact to produce developmental outcomes. Here we test an interactive model of learning and development where certain learning-related changes are constrained by developmental changes in the brain against an alternative development-as-practice model where outcomes are determined primarily by the accumulation of experience regardless of age. Participants (8-29 years) participated in a three-wave, accelerated longitudinal study during which they completed a feedback learning task during an fMRI scan. Adopting a novel longitudinal modeling approach, we probed the unique and moderated effects of learning, experience, and development simultaneously on behavioral performance and network modularity during the task. We found nonlinear patterns of development for both behavior and brain, and that greater experience supported increased learning and network modularity relative to naïve subjects. We also found changing brain-behavior relationships across adolescent development, where heightened network modularity predicted improved learning, but only following the transition from adolescence to young adulthood. These results present compelling support for an interactive view of experience and development, where changes in the brain impact behavior in context-specific fashion based on developmental goals.

Individual differences in experienced and observational decision-making illuminate interactions between reinforcement learning and declarative memory

Decision making can be shaped both by trial-and-error experiences and by memory of unique contextual information. Moreover, these types of information can be acquired either by means of active experience or by observing others behave in similar situations. The interactions between reinforcement learning parameters that inform decision updating and memory formation of declarative information in experienced and observational learning settings are, however, unknown. In the current study, participants took part in a probabilistic decision-making task involving situations that either yielded similar outcomes to those of an observed player or opposed them. By fitting alternative reinforcement learning models to each subject, we discerned participants who learned similarly from experience and observation from those who assigned different weights to learning signals from these two sources. Participants who assigned different weights to their own experience versus those of others displayed enhanced memory performance as well as subjective memory strength for episodes involving significant reward prospects. Conversely, memory performance of participants who did not prioritize their own experience over others did not seem to be influenced by reinforcement learning parameters. These findings demonstrate that interactions between implicit and explicit learning systems depend on the means by which individuals weigh relevant information conveyed via experience and observation.

Associations Between Delay Discounting and Connectivity of the Valuation-control System in Healthy Young Adults

The process of valuation assists in determining if an object or course of action is rewarding. Delay discounting is the observed decay of a rewards' subjective value over time. Encoding the subjective value of rewards across a spectrum has been attributed to brain regions belonging to the valuation and executive control systems. The valuation system (VS) encodes reward value over short and long delays, influencing reinforcement learning and reward representation. The executive control system (ECS) becomes more active as choice difficulty increases, integrating contextual and mnemonic information with salience signals in the modulation of decision-making. Here, we aimed to identify resting-state functional connectivity-based patterns of the VS and ECS correlated with value-setting and delay discounting (outside-scanner paradigm) in a large (n = 992) cohort of healthy young adults from the Human Connectome Project (HCP). Results suggest the VS may be involved in value-setting of small, immediate rewards while the ECS may be involved in value-setting and delay discounting for large and small rewards over a range of delays. We observed magnitude sensitive connections involving the posterior cingulate cortex, time-sensitive connections with the ventromedial and lateral prefrontal cortex while connections involving the posterior parietal cortex appeared both magnitude- and time-sensitive. The ventromedial prefrontal cortex and posterior parietal cortex could act as "comparator" regions, weighing the value of small rewards against large rewards across various delay duration to aid in decision-making.

The Looking Glass for Intelligence Quotient Tests: The Interplay of Motivation, Cognitive Functioning, and Affect

The Intelligence Quotient (IQ) tests and the corresponding psychometric explanations dominate both the scientific and popular views about human intelligence. Though the IQ tests have been in currency for long, there exists a gap in what they are believed to measure and what they do. While the IQ tests index the quality of cognitive functioning in selected domains of mental repertoire, the applied settings often inflate their predictive value leading to an interpretive gap. The present article contends that studying the influence of motivational and affective processes on cognitive functioning would help to evolve a more psychologically comprehensive account of the IQ tests and bridge the interpretive gap. To conclude, the article suggests possible future research directions that could strengthen the predictive value of the IQ tests.

Towards the “Homo Deus” excellence perspective?

Purpose

The evolution of management underlines the importance of the human, systemic, technological and contingency element and their interaction along with an amplified awareness of organizations for achieving excellence. This paper aims to discuss whether the fusion of digital, biological and physical world leads to a new excellence perspective and to investigate the potential value of informative neuroscientific findings for setting the foundations for smart services.

Design/methodology/approach

This study is based on a literature review regarding the advances of neurosciences and its implications for business. Their usefulness and potential contribution for the provision of smart services are investigated.

Findings

The fusion of technological evolution and biological sciences may potentially give birth to a new excellence conceptualization complemented by genetic data whose consequences are hard to predict. Neurosciences offer insights for various human behavior areas that can be used by business practitioners, to refine their thinking and management style and build brain-friendly organizational contexts. The combination of using neuroscientific evidence and technology in service systems sets the foundations for an “intelligent” provision of services in a quality context.

Originality/value

The paper investigates the conceptual development of excellence within the receding context of the “smart era” and the potential contribution of neurosciences for the provision of smart services with reference to quality pioneers’ theories and ideas.

Toward Computational Motivation for Multi-Agent Systems and Swarms

Motivation is a crucial part of animal and human mental development, fostering competence, autonomy, and open-ended development. Motivational constructs have proved to be an integral part of explaining human and animal behavior. Computer scientists have proposed various computational models of motivation for artificial agents, with the aim of building artificial agents capable of autonomous goal generation. Multi-agent systems and swarm intelligence are natural extensions to the individual agent setting. However, there are only a few works that focus on motivation theories in multi-agent or swarm settings. In this study, we review current computational models of motivation settings, mechanisms, functions and evaluation methods and discuss how we can produce systems with new kinds of functions not possible using individual agents. We describe in detail this open area of research and the major research challenges it holds.

The algorithmic architecture of exploration in the human brain

Balancing exploration and exploitation is one of the central problems in reinforcement learning. We review recent studies that have identified multiple algorithmic strategies underlying exploration. In particular, humans use a combination of random and uncertainty-directed exploration strategies, which rely on different brain systems, have different developmental trajectories, and are sensitive to different task manipulations. Humans are also able to exploit sophisticated structural knowledge to aid their exploration, such as information about correlations between options. New computational models, drawing inspiration from machine learning, have begun to formalize these ideas and offer new ways to understand the neural basis of reinforcement learning.

Viewing Adaptive Social Choice Through the Lens of Associative Learning

Because humans live in a dynamic and evolving social world, modeling the factors that guide social behavior has remained a challenge for psychology. In contrast, much progress has been made on understanding some of the more basic elements of human behavior, such as associative learning and memory, which has been successfully modeled in other species. Here we argue that applying an associative learning approach to social behavior can offer valuable insights into the human moral experience. We propose that the basic principles of associative learning-conserved across a range of species-can, in many situations, help to explain seemingly complex human behaviors, including altruistic, cooperative, and selfish acts. We describe examples from the social decision-making literature using Pavlovian learning phenomena (e.g., extinction, cue competition, stimulus generalization) to detail how a history of positive or negative social outcomes influences cognitive and affective mechanisms that shape moral choice. Examining how we might understand social behaviors and their likely reliance on domain-general mechanisms can help to generate testable hypotheses to further understand how social value is learned, represented, and expressed behaviorally.

Interactions of Motivation and Cognitive Control

There is general agreement that both motivation and cognitive control play critical roles in shaping goal-directed behavior, but only recently has scientific interest focused around the question of motivation-control interactions. Here we briefly survey this literature, organizing contemporary findings around three issues: 1) whether motivation preferentially impacts cognitive control processes, 2) the neural mechanisms that underlie motivation-cognition interactions, and 3) why motivation might be relevant for overcoming the costs of control. Dopamine (DA) is discussed as a key neuromodulator in these motivation-cognition interactions. We conclude by highlighting open issues, specifically Pavlovian versus instrumental control distinctions and effects of motivational valence and conflict, which could benefit from future research attention.

Interactions between metabolic, reward and cognitive processes in appetite control: Implications for novel weight management therapies

Traditional models of appetite control have emphasised the role of parallel homeostatic and hedonic systems, but more recently the distinction between independent homeostatic and hedonic systems has been abandoned in favour of a framework that emphasises the cross talk between the neurochemical substrates of the two systems. In addition, evidence has emerged more recently, that higher level cognitive functions such as learning, memory and attention play an important role in everyday appetite control and that homeostatic signals also play a role in cognition. Here, we review this evidence and present a comprehensive model of the control of appetite that integrates cognitive, homeostatic and reward mechanisms. We discuss the implications of this model for understanding the factors that may contribute to disordered patterns of eating and suggest opportunities for developing more effective treatment approaches for eating disorders and weight management.

Young adults self-derive and retain new factual knowledge through memory integration

The present research investigated the retention of new factual knowledge derived through integration of information acquired across temporally distributed learning episodes. Young adults were exposed to novel facts as they read long lists of seemingly unrelated information, one sentence at a time. They then were presented open-ended questions, the answers to which could be self-derived through integration of pairs of facts from the list. Experiment 1 was the first test of self-derivation of new factual knowledge through integration in adults using open-ended testing (as opposed to forced-choice testing). Participants successfully self-derived integrated knowledge under these more challenging conditions. Experiment 2 was a test for long-term retention of newly self-derived information. Newly derived knowledge remained accessible after a 1-week delay. Striking individual differences were also observed, which were related to whether individuals spontaneously identified the relational structure of the learning task. Insight into the relation between explicit task knowledge and strategic processing was also revealed through examination of response speed at the time of test. Specifically, knowledge of the task structure was associated with response latencies on unsuccessful (but not successful) trials, such that participants who were aware of the opportunity to integrate spent longer when they were subsequently unsuccessful, presumably reflecting directed search strategies and heightened perseverance when those processes failed. Together, the present findings provide direct evidence for the role of memory integration in the long-term accumulation of a semantic knowledge base and have theoretical implications for our understanding of this fundamental form of learning.

Reward Anticipation Dynamics during Cognitive Control and Episodic Encoding: Implications for Dopamine

Dopamine (DA) modulatory activity critically supports motivated behavior. This modulation operates at multiple timescales, but the functional roles of these distinct dynamics on cognition are still being characterized. Reward processing has been robustly linked to DA activity; thus, examining behavioral effects of reward anticipation at different timing intervals, corresponding to different putative dopaminergic dynamics, may help in characterizing the functional role of these dynamics. Towards this end, we present two research studies investigating reward motivation effects on cognitive control and episodic memory, converging in their manipulation of rapid vs. multi-second reward anticipation (consistent with timing profiles of phasic vs. ramping DA, respectively) on performance. Under prolonged reward anticipation, both control and memory performances were enhanced, specifically when combined with other experimental factors: task-informative cues (control task) and reward uncertainty (memory task). Given observations of ramping DA under uncertainty (Fiorillo et al., 2003) and arguments that uncertainty may act as a control signal increasing environmental monitoring (Mushtaq et al., 2011), we suggest that task information and reward uncertainty can both serve as “need for control” signals that facilitate learning via enhanced monitoring, and that this activity may be supported by a ramping profile of dopaminergic activity. Observations of rapid (i.e., phasic) reward on control and memory performance can be interpreted in line with prior evidence, but review indicates that contributions of different dopaminergic timescales in these processes are not well-understood. Future experimental work to clarify these dynamics and characterize a cross-domain role for reward motivation and DA in goal-directed behavior is suggested.

Motivational Context Modulates Prediction Error Response in Schizophrenia

BACKGROUND

Recent findings demonstrate that patients with schizophrenia are worse at learning to predict rewards than losses, suggesting that motivational context modulates learning in this disease. However, these findings derive from studies in patients treated with antipsychotic medications, D2 receptor antagonists that may interfere with the neural systems that underlie motivation and learning. Thus, it remains unknown how motivational context affects learning in schizophrenia, separate from the effects of medication.

METHODS

To examine the impact of motivational context on learning in schizophrenia, we tested 16 unmedicated patients with schizophrenia and 23 matched controls on a probabilistic learning task while they underwent functional magnetic resonance imaging (fMRI) under 2 conditions: one in which they pursued rewards, and one in which they avoided losses. Computational models were used to derive trial-by-trial prediction error responses to feedback.

RESULTS

Patients performed worse than controls on the learning task overall, but there were no behavioral effects of condition. FMRI revealed an attenuated prediction error response in patients in the medial prefrontal cortex, striatum, and medial temporal lobe when learning to predict rewards, but not when learning to avoid losses.

CONCLUSIONS

Patients with schizophrenia showed differences in learning-related brain activity when learning to predict rewards, but not when learning to avoid losses. Together with prior work, these results suggest that motivational deficits related to learning in schizophrenia are characteristic of the disease and not solely a result of antipsychotic treatment.

When Does Model-Based Control Pay Off?

Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

When you make a choice about what groceries to get for dinner, you can rely on two different strategies. You can make your choice by relying on habit, simply buying the items you need to make a meal that is second nature to you. However, you can also plan your actions in a more deliberative way, realizing that the friend who will join you is a vegetarian, and therefore you should not make the burgers that have become a staple in your cooking. These two strategies differ in how computationally demanding and accurate they are. While the habitual strategy is less computationally demanding (costs less effort and time), the deliberative strategy is more accurate. Scientists have been able to study the distinction between these strategies using a task that allows them to measure how much people rely on habit and planning strategies. Interestingly, we have discovered that in this task, the deliberative strategy does not increase performance accuracy, and hence does not induce a trade-off between accuracy and demand. We describe why this happens, and improve the task so that it embodies an accuracy-demand trade-off, providing evidence for theories of cost-based arbitration between cognitive strategies.

Striatal opioid receptor availability is related to acute and chronic pain perception in arthritis: does opioid adaptation increase resilience to chronic pain?

The experience of pain in humans is modulated by endogenous opioids, but it is largely unknown how the opioid system adapts to chronic pain states. Animal models of chronic pain point to upregulation of opioid receptors (OpR) in the brain, with unknown functional significance. We sought evidence for a similar relationship between chronic pain and OpR availability in humans. Using positron emission tomography and the radiotracer (11)C-diprenorphine, patients with arthritis pain (n = 17) and healthy controls (n = 9) underwent whole-brain positron emission tomography scanning to calculate parametric maps of OpR availability. Consistent with the upregulation hypothesis, within the arthritis group, greater OpR availability was found in the striatum (including the caudate) of patients reporting higher levels of recent chronic pain, as well as regions of interest in the descending opioidergic pathway including the anterior cingulate cortex, thalamus, and periaqueductal gray. The functional significance of striatal changes were clarified with respect to acute pain thresholds: data across patients and controls revealed that striatal OpR availability was related to reduced pain perception. These findings are consistent with the view that chronic pain may upregulate OpR availability to dampen pain. Finally, patients with arthritis pain, compared with healthy controls, had overall less OpR availability within the striatum specifically, consistent with the greater endogenous opioid binding that would be expected in chronic pain states. Our observational evidence points to the need for further studies to establish the causal relationship between chronic pain states and OpR adaptation.

Implicit learning on a probabilistic classification task in adults and adolescents with Bulimia Nervosa

BACKGROUND

Dysfunction in frontostriatal circuits likely contributes to impaired regulatory control in Bulimia Nervosa (BN), resulting in binge-eating and purging behaviors that resemble maladaptive habits. Less is known about the implicit learning processes of these circuits, which may contribute to habit formation.

METHODS

We compared 52 adolescent and adult females with BN to 55 healthy matched-controls during performance of a probabilistic classification learning task, one form of implicit learning. Groups were compared in accuracy and response times, using mixed-models with block, age, and diagnosis as predictors, corrected for multiple comparisons with confounds covaried.

RESULTS

BN participants showed differences in performance on a probabilistic classification learning task that varied by age. Adolescents with BN initially performed as accurately as healthy adolescents, but showed poorer perseverance over time. Adults with BN initially performed less accurately than healthy adults, but improved to perform equivalently. Symptom severity was associated with poorer accuracy in both adults and adolescents with BN.

CONCLUSIONS

Frontostriatal dysfunction may underlie abnormalities in regulatory control and probabilistic classification learning in BN, likely contributing to the dysregulation of implicitly learned, maladaptive binge-eating and purging behaviors. Such dysfunction in BN may progress with increasing age, first manifesting in poor regulatory control over behaviors and then expanding to implicit learning processes that may underlie habitual behaviors.

Neurobiological model of the persistence of anorexia nervosa

Anorexia Nervosa (AN) is characterized by the maintenance of an undernourished, or starved, state. Persistent restrictive eating, or the recurrent intake of a diet that is inadequate to sustain a healthy weight, is the central behavior maintaining AN. To understand this disturbance, we need to understand the neural mechanisms that allow or promote the persistent choice of inadequate caloric intake. While a range of neural disturbances have been reported in AN, abnormalities in systems relevant to reward processing and the development of habit systems have been consistently described in both structural and functional neuroimaging studies. Most recently, brain and behavior have been directly examined by investigating the neural underpinnings of restrictive food choice. These recent data suggest that, among individuals with AN, dorsal frontostriatal circuits play a greater role in guiding decisions regarding what to eat than among healthy individuals. This line of research attempts to leverage advances in the field of cognitive neuroscience to further our understanding of persistent maladaptive choices of individuals with AN, in the hope that such advances will help in the development of novel treatments for this potentially fatal disorder.

Cognitive processing of food rewards

Cues associated with tasty foods, such as their smell or taste, are strong motivators of eating, but the power of food cues on behaviour varies from moment to moment and from person to person. Variation in the rewarding value of a food with metabolic state explains why food cues are more attractive when hungry. However, cognitive processes are also important determinants of our responses to food cues. An urge to consume a tempting food may be resisted if, for example, a person has a longer term goal of weight loss. There is also evidence that responses to food cues can be facilitated or inhibited by memory processes. The aim of this review is to add to the literature on cognitive control of eating by reviewing recent evidence on the influence of working memory and episodic memory processes on responses to food cues. It is argued that processing of food information in working memory affects how much attention is paid to food cues in the environment and promotes the motivation to seek out food in the absence of direct contact with food cues. It is further argued that memories of specific recent eating episodes play an important role in directing food choices and influencing when and how much we eat. However, these memory processes are prone to disruption. When this happens, eating behaviour may become more cue-driven and less flexible. In the modern food environment, disruption of cognitive processing of food reward cues may lead to overconsumption and obesity.

Habitual control of goal selection in humans

Humans choose actions based on both habit and planning. Habitual control is computationally frugal but adapts slowly to novel circumstances, whereas planning is computationally expensive but can adapt swiftly. Current research emphasizes the competition between habits and plans for behavioral control, yet many complex tasks instead favor their integration. We consider a hierarchical architecture that exploits the computational efficiency of habitual control to select goals while preserving the flexibility of planning to achieve those goals. We formalize this mechanism in a reinforcement learning setting, illustrate its costs and benefits, and experimentally demonstrate its spontaneous application in a sequential decision-making task.

Training to Optimize Learning after Traumatic Brain Injury

One of the major foci of rehabilitation after traumatic brain injury is the design and implementation of interventions to train individuals to learn new knowledge and skills or new ways to access and execute previously acquired knowledge and skills. To optimize these interventions, rehabilitation professionals require a clear understanding of how traumatic brain injury impacts learning, and how specific approaches may enhance learning after traumatic brain injury. This brief conceptual review provides an overview of learning, the impact of traumatic brain injury on explicit and implicit learning, and the current state of the science examining selected training approaches designed to advance learning after traumatic brain injury. Potential directions for future scientific inquiry are discussed throughout the review.

Mechanisms of motivation-cognition interaction: challenges and opportunities

Recent years have seen a rejuvenation of interest in studies of motivation-cognition interactions arising from many different areas of psychology and neuroscience. The present issue of Cognitive, Affective, & Behavioral Neuroscience provides a sampling of some of the latest research from a number of these different areas. In this introductory article, we provide an overview of the current state of the field, in terms of key research developments and candidate neural mechanisms receiving focused investigation as potential sources of motivation-cognition interaction. However, our primary goal is conceptual: to highlight the distinct perspectives taken by different research areas, in terms of how motivation is defined, the relevant dimensions and dissociations that are emphasized, and the theoretical questions being targeted. Together, these distinctions present both challenges and opportunities for efforts aiming toward a more unified and cross-disciplinary approach. We identify a set of pressing research questions calling for this sort of cross-disciplinary approach, with the explicit goal of encouraging integrative and collaborative investigations directed toward them.

Multiple memory systems as substrates for multiple decision systems

It has recently become widely appreciated that value-based decision making is supported by multiple computational strategies. In particular, animal and human behavior in learning tasks appears to include habitual responses described by prominent model-free reinforcement learning (RL) theories, but also more deliberative or goal-directed actions that can be characterized by a different class of theories, model-based RL. The latter theories evaluate actions by using a representation of the contingencies of the task (as with a learned map of a spatial maze), called an "internal model." Given the evidence of behavioral and neural dissociations between these approaches, they are often characterized as dissociable learning systems, though they likely interact and share common mechanisms. In many respects, this division parallels a longstanding dissociation in cognitive neuroscience between multiple memory systems, describing, at the broadest level, separate systems for declarative and procedural learning. Procedural learning has notable parallels with model-free RL: both involve learning of habits and both are known to depend on parts of the striatum. Declarative memory, by contrast, supports memory for single events or episodes and depends on the hippocampus. The hippocampus is thought to support declarative memory by encoding temporal and spatial relations among stimuli and thus is often referred to as a relational memory system. Such relational encoding is likely to play an important role in learning an internal model, the representation that is central to model-based RL. Thus, insofar as the memory systems represent more general-purpose cognitive mechanisms that might subserve performance on many sorts of tasks including decision making, these parallels raise the question whether the multiple decision systems are served by multiple memory systems, such that one dissociation is grounded in the other. Here we investigated the relationship between model-based RL and relational memory by comparing individual differences across behavioral tasks designed to measure either capacity. Human subjects performed two tasks, a learning and generalization task (acquired equivalence) which involves relational encoding and depends on the hippocampus; and a sequential RL task that could be solved by either a model-based or model-free strategy. We assessed the correlation between subjects' use of flexible, relational memory, as measured by generalization in the acquired equivalence task, and their differential reliance on either RL strategy in the decision task. We observed a significant positive relationship between generalization and model-based, but not model-free, choice strategies. These results are consistent with the hypothesis that model-based RL, like acquired equivalence, relies on a more general-purpose relational memory system.

VBA: a probabilistic treatment of nonlinear models for neurobiological and behavioural data

This work is in line with an on-going effort tending toward a computational (quantitative and refutable) understanding of human neuro-cognitive processes. Many sophisticated models for behavioural and neurobiological data have flourished during the past decade. Most of these models are partly unspecified (i.e. they have unknown parameters) and nonlinear. This makes them difficult to peer with a formal statistical data analysis framework. In turn, this compromises the reproducibility of model-based empirical studies. This work exposes a software toolbox that provides generic, efficient and robust probabilistic solutions to the three problems of model-based analysis of empirical data: (i) data simulation, (ii) parameter estimation/model selection, and (iii) experimental design optimization.

Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans

How do we use our memories of the past to guide decisions we've never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward — such as when planning routes using a cognitive map or chess moves using predicted countermoves — and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation.

We are always learning regularities in the world around us: where things are, and in what order we might find them. Our knowledge of these contingencies can be relied upon if we later want to use them to make decisions. However, there is little agreement about the neurobiological mechanism by which learned contingencies are deployed for decision making. These are different kinds of decisions than simple habits, in which we take actions that have in the past given us reward. Neural mechanisms of habitual decisions are well-described by computational reinforcement learning approaches, but have not often been applied to ‘model-based’ decisions that depend on learned contingencies. In this article, we apply reinforcement learning to investigate model-based decisions. We tested participants on a serial reaction time task with changing sequential contingencies, and choice probes that depend on these contingencies. Fitting computational models to reaction times, we show that two sets of predictions drive simple response behavior, only one of which is used to make choices. Using fMRI, we observed learning and decision-related activity in hippocampal and ventral cortical areas that is computationally linked to the learned contingencies used to make choices. These results suggest a critical role for a hippocampal-cortical network in model-based decisions for reward.

Abnormal subcortical components of the corticostriatal system in young adults with DLI: a combined structural MRI and DTI study

Developmental Language Impairment (DLI) is a neurodevelopmental disorder affecting 12% to 14% of the school-age children in the United States. While substantial studies have shown a wide range of linguistic and non-linguistic difficulty in individuals with DLI, very little is known about the neuroanatomical mechanisms underlying this disorder. In the current study, we examined the subcortical components of the corticostriatal system in young adults with DLI, including the caudate nucleus, the putamen, the nucleus accumbens, the globus pallidus, and the thalamus. Additionally, the four cerebral lobes and the hippocampus were also comprised for an exploratory analysis. We used conventional magnetic resonance imaging (MRI) to measure regional brain volumes, as well as diffusion tensor imaging (DTI) to assess water diffusion anisotropy as quantified by fractional anisotropy (FA). Two groups of participants, one with DLI (n=12) and the other without (n=12), were recruited from a prior behavioral study, and all were matched on age, gender, and handedness. Volumetric analyses revealed region-specific abnormalities in individuals with DLI, showing pathological enlargement bilaterally in the putamen and the nucleus accumbens, and unilaterally in the right globus pallidus after the intracranial volumes were controlled. Regarding the DTI findings, the DLI group showed decreased FA values in the globus pallidus and the thalamus but these significant differences disappeared after controlling for the whole-brain FA value, indicating that microstructural abnormality is diffuse and affects other regions of the brain. Taken together, these results suggest region-specific corticostriatal abnormalities in DLI at the macrostructural level, but corticostriatal abnormalities at the microstructural level may be a part of a diffuse pattern of brain development. Future work is suggested to investigate the relationship between corticostriatal connectivity and individual differences in language development.

Action, Outcome, and Value

Dual-system approaches to psychology explain the fundamental properties of human judgment, decision making, and behavior across diverse domains. Yet, the appropriate characterization of each system is a source of debate. For instance, a large body of research on moral psychology makes use of the contrast between “emotional” and “rational/cognitive” processes, yet even the chief proponents of this division recognize its shortcomings. Largely independently, research in the computational neurosciences has identified a broad division between two algorithms for learning and choice derived from formal models of reinforcement learning. One assigns value to actions intrinsically based on past experience, while another derives representations of value from an internally represented causal model of the world. This division between action- and outcome-based value representation provides an ideal framework for a dual-system theory in the moral domain.

Generalization of value in reinforcement learning by humans

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

CID: a valid incentive delay paradigm for children

Despite several modifications and the wide use of the monetary incentive delay paradigm (MID; Knutson et al. in J Neurosci 21(16):RC159, 2001a) for assessing reward processing, evidence concerning its application in children is scarce. A first child-friendly MID modification has been introduced by Gotlib et al. (Arch Gen Psychiatry 67(4): 380-387, 2010); however, comparability in the results of different tasks and validity across different age groups remains unclear. We investigated the validity of a newly modified MID task for children (CID) using functional magnetic resonance imaging. The CID comprises the integration of a more age appropriate feedback phase. We focused on reward anticipation and their neural correlates. Twenty healthy young adults completed the MID and the CID. Additionally, 10 healthy children completed the CID. As expected, both paradigms elicited significant ventral and dorsal striatal activity in young adults during reward anticipation. No differential effects of the tasks on reaction times, accuracy rates or on the total amount of gain were observed. Furthermore, the CID elicited significant ventral striatal activity in healthy children. In conclusion, these findings demonstrate evidence for the validity of the CID paradigm. The CID can be recommended for the application in future studies on reward processing in children, adolescents, and in adults.

Translating upwards: linking the neural and social sciences via neuroeconomics

The social and neural sciences share a common interest in understanding the mechanisms that underlie human behaviour. However, interactions between neuroscience and social science disciplines remain strikingly narrow and tenuous. We illustrate the scope and challenges for such interactions using the paradigmatic example of neuroeconomics. Using quantitative analyses of both its scientific literature and the social networks in its intellectual community, we show that neuroeconomics now reflects a true disciplinary integration, such that research topics and scientific communities with interdisciplinary span exert greater influence on the field. However, our analyses also reveal key structural and intellectual challenges in balancing the goals of neuroscience with those of the social sciences. To address these challenges, we offer a set of prescriptive recommendations for directing future research in neuroeconomics.

Granularity and the acquisition of grammatical gender: how order-of-acquisition affects what gets learned

Why do adult language learners typically fail to acquire second languages with native proficiency? Does prior linguistic experience influence the size of the "units" adults attend to in learning, and if so, how does this influence what gets learned? Here, we examine these questions in relation to grammatical gender, which adult learners almost invariably struggle to master. We present a model of learning that predicts that exposure to smaller units (such as nouns) before exposure to larger linguistic units (such as sentences) can critically impair learning about predictive relations between units: such as that between a noun and its article. This prediction is then confirmed by a study of adult participants learning grammatical gender in an artificial language. Adults learned both nouns and their articles better when they were first heard nouns used in context with their articles prior to hearing the nouns individually, compared with learners who first heard the nouns in isolation, prior to hearing them used in context. In the light of these results, we discuss the role gender appears to play in language, the importance of meaning in artificial grammar learning, and the implications of this work for the structure of L2-training.

Striatal activations signal prediction errors on confidence in the absence of external feedback

Research on the neural bases of learning has mainly focused on reinforcement learning where the central role of the dopaminergic system is well established. However, in everyday life many decisions are not followed by feedback, in which case humans have been shown to code the most probable outcome into memory. We used functional magnetic resonance imaging (fMRI) to examine the neural basis of internally generated signals on correctness and decision confidence in the complete absence of feedback in a categorization task. During test trials after observational training activation in dopaminergic target regions was modulated by the correctness of the answer similarly as during feedback-based training. Moreover, activation in the nucleus accumbens and putamen was correlated with the prediction error on confidence as estimated by a reinforcement learning model. In this model subjective confidence ratings acquired after each trial served as outcome measure. Activation in the striatum therefore follows a similar pattern in response to prediction errors on confidence as it does during reinforcement learning in response to reward prediction errors, but with respect to internally generated signals based on knowledge of the structure of the environment. Furthermore, ventral striatal activation decreased with stimulus novelty, which might support the allocation of attention to unfamiliar stimuli. These results provide a parsimonious account for the neural bases of learning, indicating overlapping neural substrates of reinforcement learning and learning when outcome information has to be internally constructed.

Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events

Decisions are often guided by generalizing from past experiences. Fundamental questions remain regarding the cognitive and neural mechanisms by which generalization takes place. Prior data suggest that generalization may stem from inference-based processes at the time of generalization. By contrast, generalization may emerge from mnemonic processes occurring while premise events are encoded. Here, participants engaged in a two-phase learning and generalization task, wherein they learned a series of overlapping associations and subsequently generalized what they learned to novel stimulus combinations. Functional MRI revealed that successful generalization was associated with coupled changes in learning-phase activity in the hippocampus and midbrain (ventral tegmental area/substantia nigra). These findings provide evidence for generalization based on integrative encoding, whereby overlapping past events are integrated into a linked mnemonic representation. Hippocampal-midbrain interactions support the dynamic integration of experiences, providing a powerful mechanism for building a rich associative history that extends beyond individual events.