The Role of Basal Ganglia Reinforcement Learning in Lexical Ambiguity Resolution

One Size Does Not Fit All: Idiographic Computational Models Reveal Individual Differences in Learning and Meta-Learning Strategies

Complex skill learning depends on the joint contribution of multiple interacting systems: working memory (WM), declarative long-term memory (LTM) and reinforcement learning (RL). The present study aims to understand individual differences in the relative contributions of these systems during learning. We built four idiographic, ACT-R models of performance on the stimulus-response learning, Reinforcement Learning Working Memory task. The task consisted of short 3-image, and long 6-image, feedback-based learning blocks. A no-feedback test phase was administered after learning, with an interfering task inserted between learning and test. Our four models included two single-mechanism RL and LTM models, and two integrated RL-LTM models: (a) RL-based meta-learning, which selects RL or LTM to learn based on recent success, and (b) a parameterized RL-LTM selection model at fixed proportions independent of learning success. Each model was the best fit for some proportion of our learners (LTM: 68.7%, RL: 4.8%, Meta-RL: 13.25%, bias-RL:13.25% of participants), suggesting fundamental differences in the way individuals deploy basic learning mechanisms, even for a simple stimulus-response task. Finally, long-term declarative memory seems to be the preferred learning strategy for this task regardless of block length (3- vs 6-image blocks), as determined by the large number of subjects whose learning characteristics were best captured by the LTM only model, and a preference for LTM over RL in both of our integrated-models, owing to the strength of our idiographic approach.

Individual Differences in Reward-Based Learning Predict Fluid Reasoning Abilities

The ability to reason and problem-solve in novel situations, as measured by the Raven's Advanced Progressive Matrices (RAPM), is highly predictive of both cognitive task performance and real-world outcomes. Here we provide evidence that RAPM performance depends on the ability to reallocate attention in response to self-generated feedback about progress. We propose that such an ability is underpinned by the basal ganglia nuclei, which are critically tied to both reward processing and cognitive control. This hypothesis was implemented in a neurocomputational model of the RAPM task, which was used to derive novel predictions at the behavioral and neural levels. These predictions were then verified in one neuroimaging and two behavioral experiments. Furthermore, an effective connectivity analysis of the neuroimaging data confirmed a role for the basal ganglia in modulating attention. Taken together, these results suggest that individual differences in a neural circuit related to reward processing underpin human fluid reasoning abilities.

The impact of social anxiety on feedback-based go and nogo learning

The term “Pavlovian” bias describes the phenomenon that learning to execute a response to obtain a reward or to inhibit a response to avoid punishment is much easier than learning the reverse. The present study investigated the interplay between this learning bias and individual levels of social anxiety. Since avoidance behavior is a hallmark feature of social anxiety and high levels of social anxiety have been associated with better learning from negative feedback, it is conceivable that the Pavlovian bias is altered in individuals with high social anxiety, with a strong tendency to avoid negative feedback, especially (but not only) in a nogo context. In addition, learning may be modulated by the individual propensity to learn from positive or negative feedback, which can be assessed as a trait-like feature. A sample of 84 healthy university students completed an orthogonalized go/nogo task that decoupled action type (go/nogo) and outcome valence (win/avoid) and a probabilistic selection task based upon which the individual propensity to learn from positive and negative feedback was determined. Self-reported social anxiety and learning propensity were used as predictors in linear mixed-effect model analysis of performance accuracy in the go/nogo task. Results revealed that high socially anxious subjects with a propensity to learn better from negative feedback showed particularly pronounced learning for nogo to avoid while lacking significant learning for nogo to win as well as go to avoid. This result pattern suggests that high levels of social anxiety in concert with negative learning propensity hamper the overcoming of Pavlovian bias in a win context while facilitating response inhibition in an avoidance context. The present data confirm the robust Pavlovian bias in feedback-based learning and add to a growing body of evidence for modulation of feedback learning by individual factors, such as personality traits. Specifically, results show that social anxiety is associated with altered Pavlovian bias, and might suggest that this effect could be driven by altered basal ganglia function primarily affecting the nogo pathway.