Stimulus-dependent adjustment of reward prediction error in the midbrain

Neural correlates of prediction error in patients with schizophrenia: evidence from an fMRI meta-analysis

Abnormal processes of learning from prediction errors, i.e. the discrepancies between expectations and outcomes, are thought to underlie motivational impairments in schizophrenia. Although dopaminergic abnormalities in the mesocorticolimbic reward circuit have been found in patients with schizophrenia, the pathway through which prediction error signals are processed in schizophrenia has yet to be elucidated. To determine the neural correlates of prediction error processing in schizophrenia, we conducted a meta-analysis of whole-brain neuroimaging studies that investigated prediction error signal processing in schizophrenia patients and healthy controls. A total of 14 studies (324 schizophrenia patients and 348 healthy controls) using the reinforcement learning paradigm were included. Our meta-analysis showed that, relative to healthy controls, schizophrenia patients showed increased activity in the precentral gyrus and middle frontal gyrus and reduced activity in the mesolimbic circuit, including the striatum, thalamus, amygdala, hippocampus, anterior cingulate cortex, insula, superior temporal gyrus, and cerebellum, when processing prediction errors. We also found hyperactivity in frontal areas and hypoactivity in mesolimbic areas when encoding prediction error signals in schizophrenia patients, potentially indicating abnormal dopamine signaling of reward prediction error and suggesting failure to represent the value of alternative responses during prediction error learning and decision making.

Learning under social versus nonsocial uncertainty: A meta-analytic approach

Much of the uncertainty that clouds our understanding of the world springs from the covert values and intentions held by other people. Thus, it is plausible that specialized mechanisms that compute learning signals under uncertainty of exclusively social origin operate in the brain. To test this hypothesis, we scoured academic databases for neuroimaging studies involving learning under uncertainty, and performed a meta‐analysis of brain activation maps that compared learning in the face of social versus nonsocial uncertainty. Although most of the brain activations associated with learning error signals were shared between social and nonsocial conditions, we found some evidence for functional segregation of error signals of exclusively social origin during learning in limited regions of ventrolateral prefrontal cortex and insula. This suggests that most behavioral adaptations to navigate social environments are reused from frontal and subcortical areas processing generic value representation and learning, but that a specialized circuitry might have evolved in prefrontal regions to deal with social context representation and strategic action.

An integrated cluster-wise significance measure for fMRI analysis

Cluster-wise inference is widely used in fMRI analysis. The cluster-level statistic is often obtained by counting the number of intra-cluster voxels which surpass a voxel-level statistical significance threshold. This measure can be sub-optimal regarding the power and false-positive error rate because the suprathreshold voxel count neglects the voxel-wise significance levels and ignores the dependence between voxels. This article aims to provide a new Integrated Cluster-wise significance Measure (ICM) for cluster-level significance determination in cluster-wise fMRI analysis by integrating cluster extent, voxel-level significance (e.g., p values), and activation dependence between within-cluster voxels. We develop a computationally efficient strategy for ICM based on probabilistic approximation theories. Consequently, the computational load for ICM-based cluster-wise inference (e.g., permutation tests) is affordable. We validate the proposed method via extensive simulations and then apply it to two fMRI data sets. The results demonstrate that ICM can improve the power with well-controlled family-wise error (FWE).

Basal ganglia lateralization in different types of reward

Reward processing is a fundamental human activity. The basal ganglia are recognized for their role in reward processes; however, specific roles of the different nuclei (e.g., nucleus accumbens, caudate, putamen and globus pallidus) remain unclear. Using quantitative meta-analyses we assessed whole-brain and basal ganglia specific contributions to money, erotic, and food reward processing. We analyzed data from 190 fMRI studies which reported stereotaxic coordinates of whole-brain, within-group results from healthy adult participants. Results showed concordance in overlapping and distinct cortical and sub-cortical brain regions as a function of reward type. Common to all reward types was concordance in basal ganglia nuclei, with distinct differences in hemispheric dominance and spatial extent in response to the different reward types. Food reward processing favored the right hemisphere; erotic rewards favored the right lateral globus pallidus and left caudate body. Money rewards engaged the basal ganglia bilaterally including its most anterior part, nucleus accumbens. We conclude by proposing a model of common reward processing in the basal ganglia and separate models for money, erotic, and food rewards.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

The Neural Correlates of Cued Reward Omission

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.

The impact of elevated body mass on brain responses during appetitive prediction error in postpartum women

Repeated exposure to highly palatable foods and elevated weight promote: 1) insensitivity to punishment in striatal regions and, 2) increased willingness to work for food. We hypothesized that BMI would be positively associated with negative prediction error BOLD response in the occipital cortex. Additionally, we postulated that food reinforcement value would be negatively associated with negative prediction error BOLD response in the orbital frontal cortex and amygdala. Postpartum women (n = 47; BMI = 25.5 ± 5.1) were 'trained' to associate specific cues paired to either a highly palatable milkshake or a sub-palatable milkshake. We then violated these cue-taste pairings in 40% of the trials by showing a palatable cue followed by the sub-palatable taste (negative prediction error). Contrary to our hypotheses, during negative prediction error (mismatched cue-taste) versus matched palatable cue-taste, women showed increased BOLD response in the central operculum (pFWE = 0.002; k = 1680; MNI: -57, -7,14) and postcentral gyrus (pFWE = 0.006, k = 1219; MNI: 62, -8,18). When comparing the matched sub-palatable cue-taste to the negative prediction error trials, BOLD response increased in the postcentral gyrus (r = -0.60, pFWE = 0.008), putamen (r = -0.55, pFWE = 0.02), and insula (r = -0.50, pFWE = 0.01). Similarly, viewing the palatable cue vs sub-palatable cue was related to BOLD response in the putamen (pFWE = 0.025, k = 53; MNI: -20, 6, -8) and the insula (pFWE = 0.04, k = 19, MNI:38, -12, -6). Neither BMI at 6-month postpartum nor food reinforcement value was related to BOLD response. The insula and putamen appear to encode for visual food cue processing, and the gustatory and somatosensory cortices appear to encode negative prediction errors. Differential response in the somatosensory cortex to the matched cue-taste pairs to negative prediction error may indicate that a palatable cue may dull aversive qualities in the stimulus.

Neural Mechanisms of Reward Prediction Error in Autism Spectrum Disorder

Few studies have explored neural mechanisms of reward learning in ASD despite evidence of behavioral impairments of predictive abilities in ASD. To investigate the neural correlates of reward prediction errors in ASD, 16 adults with ASD and 14 typically developing controls performed a prediction error task during fMRI scanning. Results revealed greater activation in the ASD group in the left paracingulate gyrus during signed prediction errors and the left insula and right frontal pole during thresholded unsigned prediction errors. Findings support atypical neural processing of reward prediction errors in ASD in frontostriatal regions critical for prediction coding and reward learning. Results provide a neural basis for impairments in reward learning that may contribute to traits common in ASD (e.g., intolerance of unpredictability).

Altered neural responsivity to food cues in relation to food preferences, but not appetite-related hormone concentrations after RYGB-surgery

BACKGROUND

After Roux-en-Y gastric bypass (RYGB) surgery, patients report a shift in food preferences away from high-energy foods.

OBJECTIVE

We aimed to elucidate the potential mechanisms underlying this shift in food preferences by assessing changes in neural responses to food pictures and odors before and after RYGB. Additionally, we investigated whether altered neural responsivity was associated with changes in plasma endocannabinoid and ghrelin concentrations.

DESIGN

19 RYGB patients (4 men; age 41 ± 10 years; BMI 41 ± 1 kg/m² before; BMI 36 ± 1 kg/m² after) participated in this study. Before and two months after RYGB surgery, they rated their food preferences using the Macronutrient and Taste Preference Ranking Task and BOLD fMRI responses towards pictures and odors of high-, and low-energy foods and non-food items were measured. Blood samples were taken to determine plasma endocannabinoid and ghrelin concentrations pre- and post-surgery.

RESULTS

Patients demonstrated a shift in food preferences away from high-fat/sweet and towards low-energy/savory food products, which correlated with decreased superior parietal lobule responsivity to high-energy food odor and a reduced difference in precuneus responsivity to high-energy versus low-energy food pictures. In the anteroventral prefrontal cortex (superior frontal gyrus) the difference in deactivation towards high-energy versus non-food odors reduced. The precuneus was less deactivated in response to all cues. Plasma concentrations of anandamide were higher after surgery, while plasma concentrations of other endocannabinoids and ghrelin did not change. Alterations in appetite-related hormone concentrations did not correlate with changes in neural responsivity.

CONCLUSIONS

RYGB leads to changed responsivity of the frontoparietal control network that orchestrates top-down control to high-energy food compared to low-energy food and non-food cues, rather than in reward related brain regions, in a satiated state. Together with correlations with the shift in food preference from high- to low-energy foods this indicates a possible role in new food preference formation.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

Learning about Expectation Violation from Prediction Error Paradigms - A Meta-Analysis on Brain Processes Following a Prediction Error

Modifying patients' expectations by exposing them to expectation violation situations (thus maximizing the difference between the expected and the actual situational outcome) is proposed to be a crucial mechanism for therapeutic success for a variety of different mental disorders. However, clinical observations suggest that patients often maintain their expectations regardless of experiences contradicting their expectations. It remains unclear which information processing mechanisms lead to modification or persistence of patients' expectations. Insight in the processing could be provided by Neuroimaging studies investigating prediction error (PE, i.e., neuronal reactions to non-expected stimuli). Two methods are often used to investigate the PE: (1) paradigms, in which participants passively observe PEs ("passive" paradigms) and (2) paradigms, which encourage a behavioral adaptation following a PE ("active" paradigms). These paradigms are similar to the methods used to induce expectation violations in clinical settings: (1) the confrontation with an expectation violation situation and (2) an enhanced confrontation in which the patient actively challenges his expectation. We used this similarity to gain insight in the different neuronal processing of the two PE paradigms. We performed a meta-analysis contrasting neuronal activity of PE paradigms encouraging a behavioral adaptation following a PE and paradigms enforcing passiveness following a PE. We found more neuronal activity in the striatum, the insula and the fusiform gyrus in studies encouraging behavioral adaptation following a PE. Due to the involvement of reward assessment and avoidance learning associated with the striatum and the insula we propose that the deliberate execution of action alternatives following a PE is associated with the integration of new information into previously existing expectations, therefore leading to an expectation change. While further research is needed to directly assess expectations of participants, this study provides new insights into the information processing mechanisms following an expectation violation.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.