Changes in corticostriatal connectivity during reinforcement learning in humans

Belief inference for hierarchical hidden states in spatial navigation

Uncertainty abounds in the real world, and in environments with multiple layers of unobservable hidden states, decision-making requires resolving uncertainties based on mutual inference. Focusing on a spatial navigation problem, we develop a Tiger maze task that involved simultaneously inferring the local hidden state and the global hidden state from probabilistically uncertain observation. We adopt a Bayesian computational approach by proposing a hierarchical inference model. Applying this to human task behaviour, alongside functional magnetic resonance brain imaging, allows us to separate the neural correlates associated with reinforcement and reassessment of belief in hidden states. The imaging results also suggest that different layers of uncertainty differentially involve the basal ganglia and dorsomedial prefrontal cortex, and that the regions responsible are organised along the rostral axis of these areas according to the type of inference and the level of abstraction of the hidden state, i.e. higher-order state inference involves more anterior parts.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Leveraging Factors of Self-Efficacy and Motivation to Optimize Stroke Recovery

The International Classification of Functioning, Disability and Health framework recognizes that an individual's functioning post-stroke reflects an interaction between their health condition and contextual factors encompassing personal and environmental factors. Personal factors significantly impact rehabilitation outcomes as they determine how an individual evaluates their situation and copes with their condition in daily life. A key personal factor is self-efficacy-an individual's belief in their capacity to achieve certain outcomes. Self-efficacy influences an individual's motivational state to execute behaviors necessary for achieving desired rehabilitation outcomes. Stroke rehabilitation practice and research now acknowledge self-efficacy and motivation as critical elements in post-stroke recovery, and increasing evidence highlights their contributions to motor (re)learning. Given the informative value of neuroimaging-based biomarkers in stroke, elucidating the neurological underpinnings of self-efficacy and motivation may optimize post-stroke recovery. In this review, we examine the role of self-efficacy and motivation in stroke rehabilitation and recovery, identify potential neural substrates underlying these factors from current neuroimaging literature, and discuss how leveraging these factors and their associated neural substrates has the potential to advance the field of stroke rehabilitation.

Escitalopram modulates learning content-specific neuroplasticity of functional brain networks

Learning-induced neuroplastic changes, further modulated by content and setting, are mirrored in brain functional connectivity (FC). In animal models, selective serotonin reuptake inhibitors (SSRIs) have been shown to facilitate neuroplasticity. This is especially prominent during emotional relearning, such as fear extinction, which may translate to clinical improvements in patients. To investigate a comparable modulation of neuroplasticity in humans, 99 healthy subjects underwent three weeks of emotional (matching faces) or non-emotional learning (matching Chinese characters to unrelated German nouns). Shuffled pairings of the original content were subsequently relearned for the same time. During relearning, subjects received either a daily dose of the SSRI escitalopram or placebo. Resting-state functional magnetic resonance imaging was performed before and after the (re-)learning phases. FC changes in a network comprising Broca's area, the medial prefrontal cortex, the right inferior temporal and left lingual gyrus were modulated by escitalopram intake. More specifically, it increased the bidirectional connectivity between medial prefrontal cortex and lingual gyrus for non-emotional and the connectivity from medial prefrontal cortex to Broca's area for emotional relearning. The context dependence of these effects together with behavioral correlations supports the assumption that SSRIs in clinical practice improve neuroplasticity rather than psychiatric symptoms per se. Beyond expanding the complexities of learning, these findings emphasize the influence of external factors on human neuroplasticity.

Lower-level associations in Gilles de la Tourette syndrome: Convergence between hyperbinding of stimulus and response features and procedural hyperfunctioning theories

Gilles de la Tourette syndrome (GTS) can be characterized by enhanced cognitive functions related to creating, modifying and maintaining connections between stimuli and responses (S-R links). Specifically, two areas, procedural sequence learning and, as a novel finding, also event file binding, show converging evidence of hyperfunctioning in GTS. In this review, we describe how these two enhanced functions can be considered as cognitive mechanisms behind habitual behaviour, such as tics in GTS. Moreover, the presence of both procedural sequence learning and event file binding hyperfunctioning in the same disorder can be treated as evidence for their functional connections, even beyond GTS. Importantly though, we argue that hyperfunctioning of event file binding and procedural learning are not interchangeable: they have different time scales, different sensitivities to potential impairment in action sequencing and distinguishable contributions to the cognitive profile of GTS. An integrated theoretical account of hyperbinding and hyperlearning in GTS allows to formulate predictions for the emergence, activation and long-term persistence of tics in GTS.

Place vs. Response Learning: History, Controversy, and Neurobiology

The present article provides a historical review of the place and response learning plus-maze tasks with a focus on the behavioral and neurobiological findings. The article begins by reviewing the conflict between Edward C. Tolman's cognitive view and Clark L. Hull's stimulus-response (S-R) view of learning and how the place and response learning plus-maze tasks were designed to resolve this debate. Cognitive learning theorists predicted that place learning would be acquired faster than response learning, indicating the dominance of cognitive learning, whereas S-R learning theorists predicted that response learning would be acquired faster, indicating the dominance of S-R learning. Here, the evidence is reviewed demonstrating that either place or response learning may be dominant in a given learning situation and that the relative dominance of place and response learning depends on various parametric factors (i.e., amount of training, visual aspects of the learning environment, emotional arousal, et cetera). Next, the neurobiology underlying place and response learning is reviewed, providing strong evidence for the existence of multiple memory systems in the mammalian brain. Research has indicated that place learning is principally mediated by the hippocampus, whereas response learning is mediated by the dorsolateral striatum. Other brain regions implicated in place and response learning are also discussed in this section, including the dorsomedial striatum, amygdala, and medial prefrontal cortex. An exhaustive review of the neurotransmitter systems underlying place and response learning is subsequently provided, indicating important roles for glutamate, dopamine, acetylcholine, cannabinoids, and estrogen. Closing remarks are made emphasizing the historical importance of the place and response learning tasks in resolving problems in learning theory, as well as for examining the behavioral and neurobiological mechanisms of multiple memory systems. How the place and response learning tasks may be employed in the future for examining extinction, neural circuits of memory, and human psychopathology is also briefly considered.

Why and how the brain weights contributions from a mixture of experts

It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a "Mixture of Experts" in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavior at any one moment in time by keeping track of the reliability of the predictions within each system, and by allocating control over behavior in a manner that depends on the relative reliabilities across experts. fMRI and neurostimulation studies suggest a specific contribution of the anterior prefrontal cortex in this process. Further, such a mechanism also takes into consideration the complexity of the expert, favoring simpler over more cognitively complex experts. Results from the study of different expert systems in both experiential and social learning domains hint at the possibility that this reliability-based control mechanism is domain general, exerting control over many different expert systems simultaneously in order to produce sophisticated behavior.

Neural substrate of a cognitive intervention program using Go game: a positron emission tomography study

BACKGROUND

Influence of cognitive intervention programs on brain activity has not been enough explored.

AIMS

The aims of the present study were to clarify changes in brain activity from a cognitive intervention program utilizing the board game "Go" and to examine the relationship between brain activity and the acquisition of Go skills.

METHODS

Eighteen community-dwelling older adults were randomly assigned either to an intervention group (IG), in which members attended 12 Go lessons either in groups or individually using tablet computers, or a control group (CG), in which members attended health education lectures unrelated to Go. Fluorine-18 fluorodeoxyglucose (¹⁸F-FDG) positron emission tomography (PET), cognitive assessments, and Go tests were performed before and after the intervention.

RESULTS

The results showed different patterns of regional FDG uptake in both groups: regional cerebral glucose metabolism was significantly increased in the left middle temporal gyrus (MTG) and bilateral putamen (p < 0.01; cluster level) in the IG, and in the left superior frontal gyrus in the CG, (p < 0.01; cluster level). Furthermore, Go test scores were significantly improved in the IG (p < 0.05), and a significant association was observed between changes in Go test scores and glucose metabolism in the left MTG (p < 0.05).

CONCLUSIONS AND DISCUSSION

This study indicates that a cognitive intervention program using Go may enhance brain activity. Further studies with larger populations and longer observation periods are needed to clarify the neural mechanisms underlying our Go intervention program.

Brain networks underlying vulnerability and resilience to drug addiction

Resting-state functional connectivity provides novel insight into variations in neural networks associated with addiction to stimulant drugs in individuals with and without a family history of addiction, and both with and without personal drug use. An increased risk for addiction, either because of drug use or genetic/psychosocial vulnerability, is associated with hypoconnectivity in frontostriatal networks, which may weaken goal-directed decision-making. Resilience against addiction development, by contrast, is characterized by hyperconnectivity in two corticostriatal pathways, possibly reflecting compensatory responses in networks associated with regulatory control over habitual behaviors. It is thus conceivable that defying the risk of developing stimulant drug addiction requires increased efforts to control behavior—a hypothesis that may open up new pathways for therapeutic and preventative strategies.

Regular drug use can lead to addiction, but not everyone who takes drugs makes this transition. How exactly drugs of abuse interact with individual vulnerability is not fully understood, nor is it clear how individuals defy the risks associated with drugs or addiction vulnerability. We used resting-state functional MRI (fMRI) in 162 participants to characterize risk- and resilience-related changes in corticostriatal functional circuits in individuals exposed to stimulant drugs both with and without clinically diagnosed drug addiction, siblings of addicted individuals, and control volunteers. The likelihood of developing addiction, whether due to familial vulnerability or drug use, was associated with significant hypoconnectivity in orbitofrontal and ventromedial prefrontal cortical-striatal circuits—pathways critically implicated in goal-directed decision-making. By contrast, resilience against a diagnosis of substance use disorder was associated with hyperconnectivity in two networks involving 1) the lateral prefrontal cortex and medial caudate nucleus and 2) the supplementary motor area, superior medial frontal cortex, and putamen—brain circuits respectively implicated in top-down inhibitory control and the regulation of habits. These findings point toward a predisposing vulnerability in the causation of addiction, related to impaired goal-directed actions, as well as countervailing resilience systems implicated in behavioral regulation, and may inform novel strategies for therapeutic and preventative interventions.

Threat-induced modulation of hippocampal and striatal memory systems during navigation of a virtual environment

The brain is composed of multiple memory systems that mediate distinct types of navigation. The hippocampus is important for encoding and retrieving allocentric spatial cognitive maps, while the dorsal striatum mediates procedural memories based on stimulus-response (S-R) associations. These memory systems are differentially affected by emotional arousal. In particular, rodent studies show that stress typically impairs hippocampal spatial memory while it spares or sometimes enhances striatal S-R memory. The influence of emotional arousal on these separate navigational memory systems has received less attention in human subjects. We investigated the effect of dynamic changes in anticipatory anxiety on hippocampal spatial and dorsal striatal S-R memory systems while participants attempted to solve a virtual eight-arm radial maze. In Experiment 1, participants completed a hippocampus-dependent spatial version of the eight-arm radial maze that required allocentric spatial memory to successfully navigate the environment. In Experiment 2, participants completed a dorsal striatal S-R version of the maze where no allocentric spatial cues were present, requiring the use of S-R navigation. Anticipatory anxiety was modulated via threat of receiving an unpleasant electrical shock to the wrist during memory retrieval. Results showed that threat of shock was associated with more errors and increased use of non-spatial navigational strategies in the hippocampal spatial task, but did not influence memory performance in the striatal S-R task. Findings indicate a dissociation regarding the influence of anticipatory anxiety on memory systems that has implications for understanding how fear and anxiety contribute to memory-related symptoms in human psychopathologies.

Dopaminergic medication reduces striatal sensitivity to negative outcomes in Parkinson's disease

Reduced levels of dopamine in Parkinson's disease contribute to changes in learning, resulting from the loss of midbrain neurons that transmit a dopaminergic teaching signal to the striatum. Dopamine medication used by patients with Parkinson's disease has previously been linked to behavioural changes during learning as well as to adjustments in value-based decision-making after learning. To date, however, little is known about the specific relationship between dopaminergic medication-driven differences during learning and subsequent changes in approach/avoidance tendencies in individual patients. Twenty-four Parkinson's disease patients ON and OFF dopaminergic medication and 24 healthy controls subjects underwent functional MRI while performing a probabilistic reinforcement learning experiment. During learning, dopaminergic medication reduced an overemphasis on negative outcomes. Medication reduced negative (but not positive) outcome learning rates, while concurrent striatal blood oxygen level-dependent responses showed reduced prediction error sensitivity. Medication-induced shifts in negative learning rates were predictive of changes in approach/avoidance choice patterns after learning, and these changes were accompanied by systematic striatal blood oxygen level-dependent response alterations. These findings elucidate the role of dopamine-driven learning differences in Parkinson's disease, and show how these changes during learning impact subsequent value-based decision-making.

Synchronicity: The Role of Midbrain Dopamine in Whole-Brain Coordination

Midbrain dopamine seems to play an outsized role in motivated behavior and learning. Widely associated with mediating reward-related behavior, decision making, and learning, dopamine continues to generate controversies in the field. While many studies and theories focus on what dopamine cells encode, the question of how the midbrain derives the information it encodes is poorly understood and comparatively less addressed. Recent anatomical studies suggest greater diversity and complexity of afferent inputs than previously appreciated, requiring rethinking of prior models. Here, we elaborate a hypothesis that construes midbrain dopamine as implementing a Bayesian selector in which individual dopamine cells sample afferent activity across distributed brain substrates, comprising evidence to be evaluated on the extent to which stimuli in the on-going sensorimotor stream organizes distributed, parallel processing, reflecting implicit value. To effectively generate a temporally resolved phasic signal, a population of dopamine cells must exhibit synchronous activity. We argue that synchronous activity across a population of dopamine cells signals consensus across distributed afferent substrates, invigorating responding to recognized opportunities and facilitating further learning. In framing our hypothesis, we shift from the question of how value is computed to the broader question of how the brain achieves coordination across distributed, parallel processing. We posit the midbrain is part of an “axis of agency” in which the prefrontal cortex (PFC), basal ganglia (BGS), and midbrain form an axis mediating control, coordination, and consensus, respectively.

Evolutionary diversity as a catalyst for biological discovery

The tremendous diversity of animal behaviors has inspired generations of scientists from an array of biological disciplines. To complement investigations of ecological and evolutionary factors contributing to behavioral evolution, modern sequencing, gene editing, computational and neuroscience tools now provide a means to discover the proximate mechanisms upon which natural selection acts to generate behavioral diversity. Social behaviors are motivated behaviors that can differ tremendously between closely related species, suggesting phylogenetic plasticity in their underlying biological mechanisms. In addition, convergent evolution has repeatedly given rise to similar forms of social behavior and mating systems in distantly related species. Social behavioral divergence and convergence provides an entry point for understanding the neurogenetic mechanisms contributing to behavioral diversity. We argue that the greatest strides in discovering mechanisms contributing to social behavioral diversity will be achieved through integration of interdisciplinary comparative approaches with modern tools in diverse species systems. We review recent advances and future potential for discovering mechanisms underlying social behavioral variation; highlighting patterns of social behavioral evolution, oxytocin and vasopressin neuropeptide systems, genetic/transcriptional "toolkits," modern experimental tools, and alternative species systems, with particular emphasis on Microtine rodents and Lake Malawi cichlid fishes.

Decreased subregional specificity of the putamen in Parkinson's Disease revealed by dynamic connectivity-derived parcellation

Parkinson's Disease (PD) is associated with decreased ability to perform habitual tasks, relying instead on goal-directed behaviour subserved by different cortical/subcortical circuits, including parts of the putamen. We explored the functional subunits in the putamen in PD using novel dynamic connectivity features derived from resting state fMRI recorded from thirty PD subjects and twenty-eight age-matched healthy controls (HC). Dynamic functional segmentation of the putamina was obtained by determining the correlation between each voxel in each putamen along a moving window and applying a joint temporal clustering algorithm to establish cluster membership of each voxel at each window. Contiguous voxels that had consistent cluster membership across all windows were then considered to be part of a homogeneous functional subunit. As PD subjects robustly had two homogenous clusters in the putamina, we also segmented the putamina in HC into two dynamic clusters for a fair comparison. We then estimated the dynamic connectivity using sliding windowed correlation between the mean signal from the identified homogenous subunits and 56 other predefined cortical and subcortical ROIs. Specifically, the mean dynamic connectivity strength and connectivity deviation were then compared to evaluate subregional differences. HC subjects had significant differences in mean dynamic connectivity and connectivity deviation between the two putaminal subunits. The posterior subunit connected strongly to sensorimotor areas, the cerebellum, as well as the middle frontal gyrus. The anterior subunit had strong mean dynamic connectivity to the nucleus accumbens, hippocampus, amygdala, caudate and cingulate. In contrast, PD subjects had fewer differences in mean dynamic connectivity between subunits, indicating a degradation of subregional specificity. Overall UPDRS III and MoCA scores could be predicted using mean dynamic connectivity strength and connectivity deviation. Side of onset of the disease was also jointly related with functional connectivity features. Our results suggest a robust loss of specificity of mean dynamic connectivity and connectivity deviation in putaminal subunits in PD that is sensitive to disease severity. In addition, altered mean dynamic connectivity and connectivity deviation features in PD suggest that looking at connectivity dynamics offers an additional dimension for assessment of neurodegenerative disorders.

Interactions between frontal and posterior oscillatory dynamics support adjustment of stimulus processing during reinforcement learning

Reinforcement learning (RL) in humans is subserved by a network of striatal and frontal brain areas. The electrophysiological signatures of feedback evaluation are increasingly well understood, but how those signatures relate to the use of feedback to guide subsequent behavioral adjustment remains unclear. One mechanism for post-feedback behavioral optimization is the modulation of sensory processing. We used source-reconstructed MEG to test whether feedback affects the interactions between sources of oscillatory activity in the learning network and task-relevant stimulus-processing areas. Participants performed a probabilistic RL task in which they learned associations between colored faces and response buttons using trial-and-error feedback. Delta-band (2-4 Hz) and theta-band (4-8 Hz) power in multiple frontal regions were sensitive to feedback valence. Low and high beta-band power (12-20 and 20-30 Hz) in occipital, parietal, and temporal regions differentiated between color and face information. Consistent with our hypothesis, single-trial power-power correlations between frontal and posterior-sensory areas were modulated by the interaction between feedback valence and the relevant stimulus characteristic (color versus identity). These results suggest that long-range oscillatory coupling supports post-feedback updating of stimulus processing.

Solving the Credit Assignment Problem With the Prefrontal Cortex

In naturalistic multi-cue and multi-step learning tasks, where outcomes of behavior are delayed in time, discovering which choices are responsible for rewards can present a challenge, known as the credit assignment problem. In this review, I summarize recent work that highlighted a critical role for the prefrontal cortex (PFC) in assigning credit where it is due in tasks where only a few of the multitude of cues or choices are relevant to the final outcome of behavior. Collectively, these investigations have provided compelling support for specialized roles of the orbitofrontal (OFC), anterior cingulate (ACC), and dorsolateral prefrontal (dlPFC) cortices in contingent learning. However, recent work has similarly revealed shared contributions and emphasized rich and heterogeneous response properties of neurons in these brain regions. Such functional overlap is not surprising given the complexity of reciprocal projections spanning the PFC. In the concluding section, I overview the evidence suggesting that the OFC, ACC and dlPFC communicate extensively, sharing the information about presented options, executed decisions and received rewards, which enables them to assign credit for outcomes to choices on which they are contingent. This account suggests that lesion or inactivation/inhibition experiments targeting a localized PFC subregion will be insufficient to gain a fine-grained understanding of credit assignment during learning and instead poses refined questions for future research, shifting the focus from focal manipulations to experimental techniques targeting cortico-cortical projections.

Putamen functional connectivity during inhibitory control in smokers and non-smokers

The putamen has been shown to play a key role in inhibitory control and addiction, and consists of distinct subregions associated with distinct functions. The anterior putamen is thought to be specialized in goal-directed control or response-monitoring in connection with frontal regions, whereas the posterior part is specialized in habitual or automatic responding in connection with sensorimotor regions. The present study is the first to delineate functional networks of the anterior and posterior putamen in a Go-NoGo response inhibition task, and to examine differences between smokers (n = 25) and non-smokers (n = 23) within these networks. Functional connectivity analyses were conducted on fMRI data from a Go-NoGo study, using the generalized form of psychophysiological interaction with anterior and posterior putamen seed regions. In the context of inhibition, the anterior putamen exhibited connectivity with the anterior cingulate cortex (ACC) and precuneus (p_FWE  < .05), which was in line with previous literature. Conversely, the posterior putamen showed connectivity with regions implicated in sensorimotor processing. When we compared smokers to non-smokers, we did not observe the expected weaker connectivity between the anterior putamen and ACC during inhibition in smokers. Instead, our study revealed stronger inhibition-related connectivity between the anterior putamen and right insula in smokers. This finding highlights the involvement of putamen - insula interactions in addiction and impulse control.

Premonitory urges and tics in Tourette syndrome: computational mechanisms and neural correlates

Tourette syndrome is characterized by open motor behaviors - tics - but another crucial aspect of the disorder is the presence of premonitory urges: uncomfortable sensations that typically precede tics and are temporarily alleviated by tics. We review the evidence implicating the somatosensory cortices and the insula in premonitory urges and the motor cortico-basal ganglia-thalamo-cortical loop in tics. We consider how these regions interact during tic execution, suggesting that the insula plays an important role as a nexus linking the sensory and emotional character of premonitory urges with their translation into tics. We also consider how these regions interact during tic learning, integrating the neural evidence with a computational perspective on how premonitory-urge alleviation reinforces tics.

The Roles of Phasic and Tonic Dopamine in Tic Learning and Expression

Tourette syndrome (TS) prominently involves dopaminergic disturbances, but the precise nature of those disturbances has remained elusive. A substantial body of empirical work and recent computational models have characterized the specific roles of phasic and tonic dopamine (DA) in action learning and selection, respectively. Using insights from this work and models, we suggest that TS involves increases in both phasic and tonic DA, which produce increased propensities for tic learning and expression, respectively. We review the evidence from reinforcement-learning and habit-learning studies in TS, which supports the idea that TS involves increased phasic DA responses; we also review the evidence that tics engage the habit-learning circuitry. On the basis of these findings, we suggest that tics are exaggerated, maladaptive, and persistent motor habits reinforced by aberrant, increased phasic DA responses. Increased tonic DA amplifies the tendency to execute learned tics and also provides a fertile ground of motor hyperactivity for tic learning. We review evidence suggesting that antipsychotics may counter both the increased propensity for tic expression, by increasing excitability in the indirect pathway, and the increased propensity for tic learning, by shifting plasticity in the indirect pathway toward long-term potentiation (and possibly also through more complex mechanisms). Finally, we review evidence suggesting that low doses of DA agonists that effectively treat TS decrease both phasic and tonic DA, thereby also reducing the propensity for both tic learning and tic expression, respectively.

Model-Based Control in Dimensional Psychiatry

We use parallel interacting goal-directed and habitual strategies to make our daily decisions. The arbitration between these strategies is relevant to inflexible repetitive behaviors in psychiatric disorders. Goal-directed control, also known as model-based control, is based on an affective outcome relying on a learned internal model to prospectively make decisions. In contrast, habit control, also known as model-free control, is based on an integration of previous reinforced learning autonomous of the current outcome value and is implicit and more efficient but at the cost of greater inflexibility. The concept of model-based control can be further extended into pavlovian processes. Here we describe and compare tasks that tap into these constructs and emphasize the clinical relevance and translation of these tasks in psychiatric disorders. Together, these findings highlight a role for model-based control as a transdiagnostic impairment underlying compulsive behaviors and representing a promising therapeutic target.

Learning about Expectation Violation from Prediction Error Paradigms - A Meta-Analysis on Brain Processes Following a Prediction Error

Modifying patients' expectations by exposing them to expectation violation situations (thus maximizing the difference between the expected and the actual situational outcome) is proposed to be a crucial mechanism for therapeutic success for a variety of different mental disorders. However, clinical observations suggest that patients often maintain their expectations regardless of experiences contradicting their expectations. It remains unclear which information processing mechanisms lead to modification or persistence of patients' expectations. Insight in the processing could be provided by Neuroimaging studies investigating prediction error (PE, i.e., neuronal reactions to non-expected stimuli). Two methods are often used to investigate the PE: (1) paradigms, in which participants passively observe PEs ("passive" paradigms) and (2) paradigms, which encourage a behavioral adaptation following a PE ("active" paradigms). These paradigms are similar to the methods used to induce expectation violations in clinical settings: (1) the confrontation with an expectation violation situation and (2) an enhanced confrontation in which the patient actively challenges his expectation. We used this similarity to gain insight in the different neuronal processing of the two PE paradigms. We performed a meta-analysis contrasting neuronal activity of PE paradigms encouraging a behavioral adaptation following a PE and paradigms enforcing passiveness following a PE. We found more neuronal activity in the striatum, the insula and the fusiform gyrus in studies encouraging behavioral adaptation following a PE. Due to the involvement of reward assessment and avoidance learning associated with the striatum and the insula we propose that the deliberate execution of action alternatives following a PE is associated with the integration of new information into previously existing expectations, therefore leading to an expectation change. While further research is needed to directly assess expectations of participants, this study provides new insights into the information processing mechanisms following an expectation violation.

Lenticular nucleus correlates of general self-efficacy in young adults

General self-efficacy (GSE) is an important factor in education, social participation, and medical treatment. However, the only study that has investigated the direct association between GSE and a neural correlate did not identify specific brain regions, rather only assessed brain structures, and included older adult subjects. GSE is related to motivation, physical activity, learning, the willingness to initiate behaviour and expend effort, and adjustment. Thus, it was hypothesized in the present study that the neural correlates of GSE might be related to changes in the basal ganglia, which is a region related to the abovementioned self-efficacy factors. This study aimed to identify the brain structures associated with GSE in healthy young adults (n = 1204, 691 males and 513 females, age 20.7 ± 1.8 years) using regional grey matter density and volume (rGMD and rGMV), fractional anisotropy (FA) and mean diffusivity (MD) analyses of magnetic resonance imaging (MRI) data. The findings showed that scores on the GSE Scale (GSES) were associated with a lower MD value in regions from the right putamen to the globus pallidum; however, there were no significant association between GSES scores and regional brain structures using the other analyses (rGMD, rGMV, and FA). Thus, the present findings indicated that the lenticular nucleus is a neural correlate of GSE.

Learning, Reward, and Decision Making

In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model-based and model-free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.

Auditory Hallucinations and the Brain's Resting-State Networks: Findings and Methodological Observations

In recent years, there has been increasing interest in the potential for alterations to the brain's resting-state networks (RSNs) to explain various kinds of psychopathology. RSNs provide an intriguing new explanatory framework for hallucinations, which can occur in different modalities and population groups, but which remain poorly understood. This collaboration from the International Consortium on Hallucination Research (ICHR) reports on the evidence linking resting-state alterations to auditory hallucinations (AH) and provides a critical appraisal of the methodological approaches used in this area. In the report, we describe findings from resting connectivity fMRI in AH (in schizophrenia and nonclinical individuals) and compare them with findings from neurophysiological research, structural MRI, and research on visual hallucinations (VH). In AH, various studies show resting connectivity differences in left-hemisphere auditory and language regions, as well as atypical interaction of the default mode network and RSNs linked to cognitive control and salience. As the latter are also evident in studies of VH, this points to a domain-general mechanism for hallucinations alongside modality-specific changes to RSNs in different sensory regions. However, we also observed high methodological heterogeneity in the current literature, affecting the ability to make clear comparisons between studies. To address this, we provide some methodological recommendations and options for future research on the resting state and hallucinations.

Computational psychiatry as a bridge from neuroscience to clinical applications

Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ, the brain, and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data driven and theory driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

The influence of cannabinoids on learning and memory processes of the dorsal striatum

Extensive evidence indicates that the mammalian endocannabinoid system plays an integral role in learning and memory. Our understanding of how cannabinoids influence memory comes predominantly from studies examining cognitive and emotional memory systems mediated by the hippocampus and amygdala, respectively. However, recent evidence suggests that cannabinoids also affect habit or stimulus-response (S-R) memory mediated by the dorsal striatum. Studies implementing a variety of maze tasks in rats indicate that systemic or intra-dorsolateral striatum infusions of cannabinoid receptor agonists or antagonists impair habit memory. In mice, cannabinoid 1 (CB1) receptor knockdown can enhance or impair habit formation, whereas Δ(9)THC tolerance enhances habit formation. Studies in human cannabis users also suggest an enhancement of S-R/habit memory. A tentative conclusion based on the available data is that acute disruption of the endocannabinoid system with either agonists or antagonists impairs, whereas chronic cannabinoid exposure enhances, dorsal striatum-dependent S-R/habit memory. CB1 receptors are required for multiple forms of striatal synaptic plasticity implicated in memory, including short-term and long-term depression. Interactions with the hippocampus-dependent memory system may also have a role in some of the observed effects of cannabinoids on habit memory. The impairing effect often observed with acute cannabinoid administration argues for cannabinoid-based treatments for human psychopathologies associated with a dysfunctional habit memory system (e.g. post-traumatic stress disorder and drug addiction/relapse). In addition, the enhancing effect of repeated cannabinoid exposure on habit memory suggests a novel neurobehavioral mechanism for marijuana addiction involving the dorsal striatum-dependent memory system.