Modeling Avoidance in Mood and Anxiety Disorders Using Reinforcement Learning

Pupil dilation reflects effortful action invigoration in overcoming aversive Pavlovian biases

"Pavlovian" or "motivational" biases describe the phenomenon that the valence of prospective outcomes modulates action invigoration: Reward prospect invigorates action, whereas punishment prospect suppresses it. The adaptive role of these biases in decision-making is still unclear. One idea is that they constitute a fast-and-frugal decision strategy in situations characterized by high arousal, e.g., in presence of a predator, which demand a quick response. In this pre-registered study (N = 35), we tested whether such a situation-induced via subliminally presented angry versus neutral faces-leads to increased reliance on Pavlovian biases. We measured trial-by-trial arousal by tracking pupil diameter while participants performed an orthogonalized Motivational Go/NoGo Task. Pavlovian biases were present in responses, reaction times, and even gaze, with lower gaze dispersion under aversive cues reflecting "freezing of gaze." The subliminally presented faces did not affect responses, reaction times, or pupil diameter, suggesting that the arousal manipulation was ineffective. However, pupil dilations reflected facets of bias suppression, specifically the physical (but not cognitive) effort needed to overcome aversive inhibition: Particularly strong and sustained dilations occurred when participants managed to perform Go responses to aversive cues. Conversely, no such dilations occurred when they managed to inhibit responses to Win cues. These results suggest that pupil diameter does not reflect response conflict per se nor the inhibition of prepotent responses, but specifically effortful action invigoration as needed to overcome aversive inhibition. We discuss our results in the context of the "value of work" theory of striatal dopamine.

An Intensive time series investigation of the relationships across eating disorder-specific fear responses and behavior urges in partially remitted anorexia nervosa

Anorexia nervosa (AN) is a serious and persistent psychiatric illness. Many individuals with AN cycle between stages of remission (i.e., relapse), with research documenting that cognitive remission generally lags behind nutritional/weight restoration. Yet, little is known about which mechanisms promote movement from partial remission in AN (defined as nutritional, but not cognitive, recovery) to full remission. Fear-based processes, including avoidance and approach behaviors, likely contribute to the persistence of cognitive-behavioral AN symptoms after nutritional restoration. The current study used intensive longitudinal data to characterize these processes during partial remission (N = 41 participants with partially remitted AN; 4306 total observations). We aimed to a) characterize frequency of fear-based processes in real-time, b) investigate associations across fear-based processes and behavioral urges, and c) test if real-time associations among symptoms differed across commonly feared stimuli (e.g., food, social situations). On average, participants endorsed moderate fear and avoidance, with weight-gain fears rated higher than other feared stimuli. Momentary fear, avoidance, approach, and distress were all positively associated with AN behavior urges at one time-point and prospectively. Central symptoms and symptom connections differed across models with different feared stimuli. These findings provide empirical support for the theorized fear-avoidance-urge cycle in AN, which may contribute to the persistence of eating pathology during partial remission. Fear approach may be associated with temporary increases in urges, which should be considered during treatment. Future research should explore these associations in large, heterogeneous samples, and test the effectiveness of exposure-based interventions during partial remission.

Adolescents flexibly adapt action selection based on controllability inferences

From early in life, we encounter both controllable environments, in which our actions can causally influence the reward outcomes we experience, and uncontrollable environments, in which they cannot. Environmental controllability is theoretically proposed to organize our behavior. In controllable contexts, we can learn to proactively select instrumental actions that bring about desired outcomes. In uncontrollable environments, Pavlovian learning enables hard-wired, reflexive reactions to anticipated, motivationally salient events, providing "default" behavioral responses. Previous studies characterizing the balance between Pavlovian and instrumental learning systems across development have yielded divergent findings, with some studies observing heightened expression of Pavlovian learning during adolescence and others observing a reduced influence of Pavlovian learning during this developmental stage. In this study, we aimed to investigate whether a theoretical model of controllability-dependent arbitration between learning systems might explain these seemingly divergent findings in the developmental literature, with the specific hypothesis that adolescents' action selection might be particularly sensitive to environmental controllability. To test this hypothesis, 90 participants, aged 8-27, performed a probabilistic-learning task that enables estimation of Pavlovian influence on instrumental learning, across both controllable and uncontrollable conditions. We fit participants' data with a reinforcement-learning model in which controllability inferences adaptively modulate the dominance of Pavlovian versus instrumental control. Relative to children and adults, adolescents exhibited greater flexibility in calibrating the expression of Pavlovian bias to the degree of environmental controllability. These findings suggest that sensitivity to environmental reward statistics that organize motivated behavior may be heightened during adolescence.

Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases

Actions are biased by the outcomes they can produce: Humans are more likely to show action under reward prospect, but hold back under punishment prospect. Such motivational biases derive not only from biased response selection, but also from biased learning: humans tend to attribute rewards to their own actions, but are reluctant to attribute punishments to having held back. The neural origin of these biases is unclear. Specifically, it remains open whether motivational biases arise primarily from the architecture of subcortical regions or also reflect cortical influences, the latter being typically associated with increased behavioral flexibility and control beyond stereotyped behaviors. Simultaneous EEG-fMRI allowed us to track which regions encoded biased prediction errors in which order. Biased prediction errors occurred in cortical regions (dorsal anterior and posterior cingulate cortices) before subcortical regions (striatum). These results highlight that biased learning is not a mere feature of the basal ganglia, but arises through prefrontal cortical contributions, revealing motivational biases to be a potentially flexible, sophisticated mechanism.

Reinforcement learning and working memory in mood disorders: A computational analysis in a developmental transdiagnostic sample

BACKGROUND

Mood disorders commonly onset during adolescence and young adulthood and are conceptually and empirically related to reinforcement learning abnormalities. However, the nature of abnormalities associated with acute symptom severity versus lifetime diagnosis remains unclear, and prior research has often failed to disentangle working memory from reward processes.

METHODS

The present sample (N = 220) included adolescents and young adults with a lifetime history of unipolar disorders (n = 127), bipolar disorders (n = 28), or no history of psychopathology (n = 62), and varying severity of mood symptoms. Analyses fitted a reinforcement learning and working memory model to an instrumental learning task that varied working memory load, and tested associations between model parameters and diagnoses or current symptoms.

RESULTS

Current severity of manic or anhedonic symptoms negatively correlated with task performance. Participants reporting higher severity of current anhedonia, or with lifetime unipolar or bipolar disorders, showed lower reward learning rates. Participants reporting higher severity of current manic symptoms showed faster working memory decay and reduced use of working memory.

LIMITATIONS

Computational parameters should be interpreted in the task environment (a deterministic reward learning paradigm), and developmental population. Future work should test replication in other paradigms and populations.

CONCLUSIONS

Results indicate abnormalities in reinforcement learning processes that either scale with current symptom severity, or correspond with lifetime mood diagnoses. Findings may have implications for understanding reward processing anomalies related to state-like (current symptom) or trait-like (lifetime diagnosis) aspects of mood disorders.

Functional Magnetic Resonance Imaging Signatures of Pavlovian and Instrumental Valuation Systems during a Modified Orthogonalized Go/No-go Task

Motivational (i.e., Pavlovian) values interfere with instrumental responding and can lead to suboptimal decision-making. In humans, task-based neuroimaging studies have only recently started illuminating the functional neuroanatomy of Pavlovian biasing of instrumental control. To provide a mechanistic understanding of the neural dynamics underlying the Pavlovian and instrumental valuation systems, analysis of neuroimaging data has been informed by computational modeling of conditioned behavior. Nonetheless, because of collinearities in Pavlovian and instrumental predictions, previous research failed to tease out hemodynamic activity that is parametrically and dynamically modulated by coexistent Pavlovian and instrumental value expectations. Moreover, neural correlates of Pavlovian to instrumental transfer effects have so far only been identified in extinction (i.e., in the absence of learning). In this study, we devised a modified version of the orthogonalized go/no-go paradigm, which introduced Pavlovian-only catch trials to better disambiguate trial-by-trial Pavlovian and instrumental predictions in both sexes. We found that hemodynamic activity in the ventromedial pFC covaried uniquely with the model-derived Pavlovian value expectations. Notably, modulation of neural activity encoding for instrumental predictions in the supplementary motor cortex was linked to successful action selection in conflict conditions. Furthermore, hemodynamic activity in regions pertaining to the limbic system and medial pFC was correlated with synergistic Pavlovian and instrumental predictions and improved conditioned behavior during congruent trials. Altogether, our results provide new insights into the functional neuroanatomy of decision-making and corroborate the validity of our variant of the orthogonalized go/no-go task as a behavioral assay of the Pavlovian and instrumental valuation systems.

Approach-avoidance reinforcement learning as a translational and computational model of anxiety-related avoidance

Although avoidance is a prevalent feature of anxiety-related psychopathology, differences in the measurement of avoidance between humans and non-human animals hinder our progress in its theoretical understanding and treatment. To address this, we developed a novel translational measure of anxiety-related avoidance in the form of an approach-avoidance reinforcement learning task, by adapting a paradigm from the non-human animal literature to study the same cognitive processes in human participants. We used computational modelling to probe the putative cognitive mechanisms underlying approach-avoidance behaviour in this task and investigated how they relate to subjective task-induced anxiety. In a large online study (n = 372), participants who experienced greater task-induced anxiety avoided choices associated with punishment, even when this resulted in lower overall reward. Computational modelling revealed that this effect was explained by greater individual sensitivities to punishment relative to rewards. We replicated these findings in an independent sample (n = 627) and we also found fair-to-excellent reliability of measures of task performance in a sub-sample retested 1 week later (n = 57). Our findings demonstrate the potential of approach-avoidance reinforcement learning tasks as translational and computational models of anxiety-related avoidance. Future studies should assess the predictive validity of this approach in clinical samples and experimental manipulations of anxiety.

Brain mediators of biased social learning of self-perception in social anxiety disorder

Social anxiety disorder (SAD) is characterized by an excessive fear of social evaluation and a persistently negative view of the self. Here we test the hypothesis that negative biases in brain responses and in social learning of self-related information contribute to the negative self-image and low self-esteem characteristic of SAD. Adult participants diagnosed with social anxiety (N = 21) and matched controls (N = 23) rated their performance and received social feedback following a stressful public speaking task. We investigated how positive versus negative social feedback altered self-evaluation and state self-esteem and used functional Magnetic Resonance Imaging (fMRI) to characterize brain responses to positive versus negative feedback. Compared to controls, participants with SAD updated their self-evaluation and state self-esteem significantly more based on negative compared to positive social feedback. Responses in the frontoparietal network correlated with and mirrored these behavioral effects, with greater responses to positive than negative feedback in non-anxious controls but not in participants with SAD. Responses to social feedback in the anterior insula and other areas mediated the effects of negative versus positive feedback on changes in self-evaluation. In non-anxious participants, frontoparietal brain areas may contribute to a positive social learning bias. In SAD, frontoparietal areas are less recruited overall and less attuned to positive feedback, possibly reflecting differences in attention allocation and cognitive regulation. More negatively biased brain responses and social learning could contribute to maintaining a negative self-image in SAD and other internalizing disorders, thereby offering important new targets for interventions.

Interindividual differences in Pavlovian influence on learning are consistent

Pavlovian influences impair instrumental learning. It is easier to learn to approach reward-predictive signals and avoid punishment-predictive cues than their contrary. Whether the interindividual variability in this Pavlovian influence is consistent across time has been examined by a number of recent studies and met with mixed results. Here we introduce an open-source, web-based instance of a well-established Go-NoGo paradigm for measuring Pavlovian influence. We closely replicated the previous laboratory-based results. Moreover, the interindividual differences in Pavlovian influence were consistent across a two-week time window at the level of (i) raw measures of learning (i.e. performance accuracy), (ii) linear, descriptive estimates of Pavlovian bias (test-retest reliability: 0.40), and (iii) parameters obtained from reinforcement learning model fitting and model selection (test-retest reliability: 0.25). Nonetheless, the correlations reported here are still lower than the standards (i.e. 0.7) employed in psychometrics and self-reported measures. Our results provide support for trusting Pavlovian bias as a relatively stable individual characteristic and for using its measure in the computational understanding of human mental health.

Reliability of gamified reinforcement learning in densely sampled longitudinal assessments

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Gender Impacts the Relationship between Mood Disorder Symptoms and Effortful Avoidance Performance

We must often decide how much effort to exert or withhold to avoid undesirable outcomes or obtain rewards. In depression and anxiety, levels of avoidance can be excessive and reward-seeking may be reduced. Yet outstanding questions remain about the links between motivated action/inhibition and anxiety and depression levels, and whether they differ between men and women. Here, we examined the relationship between anxiety and depression scores, and performance on effortful active and inhibitory avoidance (Study 1) and reward seeking (Study 2) in humans. Undergraduates and paid online workers ([Formula: see text] = 545, [Formula: see text] = 310; [Formula: see text] = 368, [Formula: see text] = 450, [Formula: see text] = 22.58, [Formula: see text] = 17-62) were assessed on the Beck Depression Inventory II (BDI) and the Beck Anxiety Inventory (BAI) and performed an instructed online avoidance or reward-seeking task. Participants had to make multiple presses on active trials and withhold presses on inhibitory trials to avoid an unpleasant sound (Study 1) or obtain points toward a monetary reward (Study 2). Overall, men deployed more effort than women in both avoidance and reward-seeking, and anxiety scores were negatively associated with active reward-seeking performance based on sensitivity scores. Gender interacted with anxiety scores and inhibitory avoidance performance, such that women with higher anxiety showed worse avoidance performance. Our results illuminate effects of gender in the relationship between anxiety and depression levels and the motivation to actively and effortfully respond to obtain positive and avoid negative outcomes.

Individual differences in naturalistic learning link negative emotionality to the development of anxiety

Organisms learn from prediction errors (PEs) to predict the future. Laboratory studies using small financial outcomes find that humans use PEs to update expectations and link individual differences in PE-based learning to internalizing disorders. Because of the low-stakes outcomes in most tasks, it is unclear whether PE learning emerges in naturalistic, high-stakes contexts and whether individual differences in PE learning predict psychopathology risk. Using experience sampling to assess 625 college students' expected exam grades, we found evidence of PE-based learning and a general tendency to discount negative PEs, an "optimism bias." However, individuals with elevated negative emotionality, a personality trait linked to the development of anxiety disorders, displayed a global pessimism and learning differences that impeded accurate expectations and predicted future anxiety symptoms. A sensitivity to PEs combined with an aversion to negative PEs may result in a pessimistic and inaccurate model of the world, leading to anxiety.

Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty

Anxiety has been related to decreased physical exploration, but past findings on the interaction between anxiety and exploration during decision making were inconclusive. Here we examined how latent factors of trait anxiety relate to different exploration strategies when facing volatility-induced uncertainty. Across two studies (total N = 985), we demonstrated that people used a hybrid of directed, random and undirected exploration strategies, which were respectively sensitive to relative uncertainty, total uncertainty and value difference. Trait somatic anxiety, that is, the propensity to experience physical symptoms of anxiety, was inversely correlated with directed exploration and undirected exploration, manifesting as a lesser likelihood for choosing the uncertain option and reducing choice stochasticity regardless of uncertainty. Somatic anxiety is also associated with underestimation of relative uncertainty. Together, these results reveal the selective role of trait somatic anxiety in modulating both uncertainty-driven and value-driven exploration strategies.

Computational perspectives on human fear and anxiety

Fear and anxiety are adaptive emotions that serve important defensive functions, yet in excess, they can be debilitating and lead to poor mental health. Computational modelling of behaviour provides a mechanistic framework for understanding the cognitive and neurobiological bases of fear and anxiety, and has seen increasing interest in the field. In this brief review, we discuss recent developments in the computational modelling of human fear and anxiety. Firstly, we describe various reinforcement learning strategies that humans employ when learning to predict or avoid threat, and how these relate to symptoms of fear and anxiety. Secondly, we discuss initial efforts to explore, through a computational lens, approach-avoidance conflict paradigms that are popular in animal research to measure fear- and anxiety-relevant behaviours. Finally, we discuss negative biases in decision-making in the face of uncertainty in anxiety.

Time is a great healer: Peak-end memory bias in anxiety - Induced by threat of shock

Recently, we demonstrated that the peak-end memory bias, which is well established in the context of pain, can also be observed in anxiety: Retrospective evaluations of a frightening experience are worse when peak anxiety is experienced at the end of an episode. Here, we set out to conceptually replicate and extend this finding with rigorous experimental control in a threat of shock paradigm. We induced two intensity levels of anxiety by presenting visual cues that indicated different strengths of electric stimuli. Each of the 59 participants went through one of two conditions that only differed in the order of moderate and high threat phases. As a manipulation check, orbicularis-EMG to auditory startle probes, electrodermal activity, and state anxiety confirmed the effects of the specific threat exposure. Critically, after some time had passed, participants for whom exposure had ended with high threat reported more anxiety for the entire episode than those for whom it ended with moderate threat. Moreover, they ranked their experience as more aversive when compared to other unpleasant everyday experiences. This study overcomes several previous limitations and speaks to the generalizability of the peak-end bias. Most notably, the findings bear implications for exposure therapy in clinical anxiety.

The neuroanatomy of social trust predicts depression vulnerability

Trust attitude is a social personality trait linked with the estimation of others’ trustworthiness. Trusting others, however, can have substantial negative effects on mental health, such as the development of depression. Despite significant progress in understanding the neurobiology of trust, whether the neuroanatomy of trust is linked with depression vulnerability remains unknown. To investigate a link between the neuroanatomy of trust and depression vulnerability, we assessed trust and depressive symptoms and employed neuroimaging to acquire brain structure data of healthy participants. A high depressive symptom score was used as an indicator of depression vulnerability. The neuroanatomical results observed with the healthy sample were validated in a sample of clinically diagnosed depressive patients. We found significantly higher depressive symptoms among low trusters than among high trusters. Neuroanatomically, low trusters and depressive patients showed similar volume reduction in brain regions implicated in social cognition, including the dorsolateral prefrontal cortex (DLPFC), dorsomedial PFC, posterior cingulate, precuneus, and angular gyrus. Furthermore, the reduced volume of the DLPFC and precuneus mediated the relationship between trust and depressive symptoms. These findings contribute to understanding social- and neural-markers of depression vulnerability and may inform the development of social interventions to prevent pathological depression.

Learning whom to cooperate with: neurocomputational mechanisms for choosing cooperative partners

Cooperation is fundamental for survival and a functioning society. With substantial individual variability in cooperativeness, we must learn whom to cooperate with, and often make these decisions on behalf of others. Understanding how people learn about the cooperativeness of others, and the neurocomputational mechanisms supporting this learning, is therefore essential. During functional magnetic resonance imaging scanning, participants completed a novel cooperation-partner-choice task where they learned to choose between cooperative and uncooperative partners through trial-and-error both for themselves and vicariously for another person. Interestingly, when choosing for themselves, participants made faster and more exploitative choices than when choosing for another person. Activity in the ventral striatum preferentially responded to prediction errors (PEs) during self-learning, whereas activity in the perigenual anterior cingulate cortex (ACC) signaled both personal and vicarious PEs. Multivariate pattern analyses showed distinct coding of personal and vicarious choice-making and outcome processing in the temporoparietal junction (TPJ), dorsal ACC, and striatum. Moreover, in right TPJ the activity pattern that differentiated self and other outcomes was associated with individual differences in exploitation tendency. We reveal neurocomputational mechanisms supporting cooperative learning and show that this learning is reflected in trial-by-trial univariate signals and multivariate patterns that can distinguish personal and vicarious choices.

Modeling Variation in Empathic Sensitivity Using Go/No-Go Social Reinforcement Learning

Recent advances in computational behavioral modeling can help rigorously quantify differences in how individuals learn behaviors that affect both themselves and others. But social learning remains understudied in the context of understanding individual variation in social phenomena like aggression, which is defined by persistent engagement in behaviors that harm others. We adapted a go/no-go reinforcement learning task across social and non-social contexts such that monetary gains and losses explicitly impacted the subject, a study partner, or no one. We then quantified participants' (n = 61) sensitivity to others' rewards, sensitivity to others' losses, and the Pavlovian influence of expected outcomes on approach and avoidance behavior. Results showed that subjects learned in response to punishments and rewards that affected their partner in a way that was computationally similar to how they learned for themselves, consistent with the possibility that social learning engages empathic processes. Further supporting this interpretation, an individualized model parameter that indexed sensitivity to others' punishments was inversely associated with trait antisociality. Modeled sensitivity to others' losses also mapped onto post-task motivation ratings, but was not associated with self-reported trait empathy. This work is the first to apply a social reinforcement learning task that spans affect and action requirement (go/no-go) to measure multiple facets of empathic sensitivity.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-022-00119-4.

Baseline Pro-Inflammatory Cytokine Levels Moderate Psychological Inflexibility in Behavioral Treatment for Chronic Pain

Background: The medical and scientific communities struggle to understand chronic pain and find effective treatments. Multimodal approaches are encouraging but show significant individual differences. Methods: Seventy-eight persons (56 women) with chronic pain received Acceptance and Commitment Therapy and provided blood samples before and after treatment. The participants completed surveys with the blood sampling. Blood plasma was analyzed for IL-6 and TNF-α levels with the Olink Inflammation Panel (Olink Bioscience Uppsala, Sweden). The treatment effects and moderating effects of low-grade inflammation on changes in outcomes were analyzed using linear mixed models. Results: Pain interference (p < 0.001) and psychological inflexibility (p < 0.001) improved significantly during treatment, but pain intensity did not (p = 0.078). Cytokine levels did not change over the course of the treatment (IL-6/TNF-α p = 0.086/0.672). Mean baseline levels of IL-6 and TNF-α moderated improvement in psychological inflexibility during the course of treatment (p = 0.044), but cytokine levels did not moderate changes in pain interference (p = 0.205) or pain intensity (p = 0.536). Conclusions: Higher baseline inflammation levels were related to less improvement in psychological inflexibility. Low-grade inflammation may be one factor underlying the variability in behavioral treatment in chronic pain.

Reinforcement Learning in Patients With Mood and Anxiety Disorders vs Control Individuals: A Systematic Review and Meta-analysis

IMPORTANCE

Computational psychiatry studies have investigated how reinforcement learning may be different in individuals with mood and anxiety disorders compared with control individuals, but results are inconsistent.

OBJECTIVE

To assess whether there are consistent differences in reinforcement-learning parameters between patients with depression or anxiety and control individuals.

DATA SOURCES

Web of Knowledge, PubMed, Embase, and Google Scholar searches were performed between November 15, 2019, and December 6, 2019, and repeated on December 3, 2020, and February 23, 2021, with keywords (reinforcement learning) AND (computational OR model) AND (depression OR anxiety OR mood).

STUDY SELECTION

Studies were included if they fit reinforcement-learning models to human choice data from a cognitive task with rewards or punishments, had a case-control design including participants with mood and/or anxiety disorders and healthy control individuals, and included sufficient information about all parameters in the models.

DATA EXTRACTION AND SYNTHESIS

Articles were assessed for inclusion according to MOOSE guidelines. Participant-level parameters were extracted from included articles, and a conventional meta-analysis was performed using a random-effects model. Subsequently, these parameters were used to simulate choice performance for each participant on benchmarking tasks in a simulation meta-analysis. Models were fitted, parameters were extracted using bayesian model averaging, and differences between patients and control individuals were examined. Overall effect sizes across analytic strategies were inspected.

MAIN OUTCOMES AND MEASURES

The primary outcomes were estimated reinforcement-learning parameters (learning rate, inverse temperature, reward learning rate, and punishment learning rate).

RESULTS

A total of 27 articles were included (3085 participants, 1242 of whom had depression and/or anxiety). In the conventional meta-analysis, patients showed lower inverse temperature than control individuals (standardized mean difference [SMD], -0.215; 95% CI, -0.354 to -0.077), although no parameters were common across all studies, limiting the ability to infer differences. In the simulation meta-analysis, patients showed greater punishment learning rates (SMD, 0.107; 95% CI, 0.107 to 0.108) and slightly lower reward learning rates (SMD, -0.021; 95% CI, -0.022 to -0.020) relative to control individuals. The simulation meta-analysis showed no meaningful difference in inverse temperature between patients and control individuals (SMD, 0.003; 95% CI, 0.002 to 0.004).

CONCLUSIONS AND RELEVANCE

The simulation meta-analytic approach introduced in this article for inferring meta-group differences from heterogeneous computational psychiatry studies indicated elevated punishment learning rates in patients compared with control individuals. This difference may promote and uphold negative affective bias symptoms and hence constitute a potential mechanistic treatment target for mood and anxiety disorders.

Heightened false alarms of conditioned threat predict longitudinal increases in GAD and SAD symptoms over the first year of college

Lab-based fear-conditioning studies have repeatedly implicated exaggerated threat reactivity to benign (unreinforced) stimuli as concurrent markers of clinical anxiety, but little work has examined the strength of false alarms as a longitudinal predictor of anxiety problems. As such, we tested whether heightened false alarms of conditioned threat assessed in participants' first semester of college predicted second-semester symptoms of generalized anxiety disorder (GAD) and social anxiety disorder (SAD) - two anxiety conditions that are common in college students, have been associated with excessive false alarms, and have yet to be assessed with longitudinal conditioning designs. Here, we focused on the predictive effects of behavioral threat responses (threat expectancy, subjective anxiety, avoidance) given their greater potential for translation to the clinic. Results implicate conditioning-related increases in anxiety to safe stimuli resembling the danger-cue as prospective predictors of GAD. In contrast, SAD was predicted by non-specific elevations in anxiety to a broad set of safe stimuli, as well as by increased threat expectancy toward cues least resembling the conditioned danger cue. These findings suggest that risk for GAD and SAD are captured by distinct, behavioral indicators of false-alarms that may be more feasibly collected in clinical settings compared to alternative experimental anxiety measures like psychophysiological responses.

A Computational Model of Hopelessness and Active-Escape Bias in Suicidality

Currently, psychiatric practice lacks reliable predictive tools and a sufficiently detailed mechanistic understanding of suicidal thoughts and behaviors (STB) to provide timely and personalized interventions. Developing computational models of STB that integrate across behavioral, cognitive and neural levels of analysis could help better understand STB vulnerabilities and guide personalized interventions. To that end, we present a computational model based on the active inference framework. With this model, we show that several STB risk markers - hopelessness, Pavlovian bias and active-escape bias - are interrelated via the drive to maximize one's model evidence. We propose four ways in which these effects can arise: (1) increased learning from aversive outcomes, (2) reduced belief decay in response to unexpected outcomes, (3) increased stress sensitivity and (4) reduced sense of stressor controllability. These proposals stem from considering the neurocircuits implicated in STB: how the locus coeruleus - norepinephrine (LC-NE) system together with the amygdala (Amy), the dorsal prefrontal cortex (dPFC) and the anterior cingulate cortex (ACC) mediate learning in response to acute stress and volatility as well as how the dorsal raphe nucleus - serotonin (DRN-5-HT) system together with the ventromedial prefrontal cortex (vmPFC) mediate stress reactivity based on perceived stressor controllability. We validate the model by simulating performance in an Avoid/Escape Go/No-Go task replicating recent behavioral findings. This serves as a proof of concept and provides a computational hypothesis space that can be tested empirically and be used to distinguish planful versus impulsive STB subtypes. We discuss the relevance of the proposed model for treatment response prediction, including pharmacotherapy and psychotherapy, as well as sex differences as it relates to stress reactivity and suicide risk.

The prefrontal cortex, pathological anxiety, and anxiety disorders

Anxiety is experienced in response to threats that are distal or uncertain, involving changes in one's subjective state, autonomic responses, and behavior. Defensive and physiologic responses to threats that involve the amygdala and brainstem are conserved across species. While anxiety responses typically serve an adaptive purpose, when excessive, unregulated, and generalized, they can become maladaptive, leading to distress and avoidance of potentially threatening situations. In primates, anxiety can be regulated by the prefrontal cortex (PFC), which has expanded in evolution. This prefrontal expansion is thought to underlie primates' increased capacity to engage high-level regulatory strategies aimed at coping with and modifying the experience of anxiety. The specialized primate lateral, medial, and orbital PFC sectors are connected with association and limbic cortices, the latter of which are connected with the amygdala and brainstem autonomic structures that underlie emotional and physiological arousal. PFC pathways that interface with distinct inhibitory systems within the cortex, the amygdala, or the thalamus can regulate responses by modulating neuronal output. Within the PFC, pathways connecting cortical regions are poised to reduce noise and enhance signals for cognitive operations that regulate anxiety processing and autonomic drive. Specialized PFC pathways to the inhibitory thalamic reticular nucleus suggest a mechanism to allow passage of relevant signals from thalamus to cortex, and in the amygdala to modulate the output to autonomic structures. Disruption of specific nodes within the PFC that interface with inhibitory systems can affect the negative bias, failure to regulate autonomic arousal, and avoidance that characterize anxiety disorders.

Clinical Efficacy and Psychological Mechanisms of an App-Based Digital Therapeutic for Generalized Anxiety Disorder: Randomized Controlled Trial

Background

Current treatments for generalized anxiety disorder (GAD) often yield suboptimal outcomes, partly because of insufficient targeting of underlying psychological mechanisms (eg, avoidance reinforcement learning). Mindfulness training (MT) has shown efficacy for anxiety; yet, widespread adoption has been limited, partly because of the difficulty in scaling in-person–based delivery. Digital therapeutics are emerging as potentially viable treatments; however, very few have been empirically validated.

Objective

The aim of this study is to test the efficacy and mechanism of an app-delivered MT that was designed to target a potential mechanism of anxiety (reinforcement learning), based on which previous studies have shown concern regarding feedback and the perpetuation of anxiety through negative reinforcement.

Methods

Individuals with GAD were recruited using social media advertisements and randomized during an in-person visit to receive treatment as usual (n=33) or treatment as usual+app−delivered MT (Unwinding Anxiety; n=32). The latter was composed of 30 modules to be completed over a 2-month period. Associated changes in outcomes were assessed using self-report questionnaires 1 and 2 months after treatment initiation.

Results

We randomized 65 participants in this study, and a modified intent-to-treat approach was used for analysis. The median number of modules completed by the MT group was 25.5 (IQR 17) out of 30; 46% (13/28) of the participants completed the program. In addition, the MT group demonstrated a significant reduction in anxiety (GAD-7) compared with the control group at 2 months (67% vs 14%; median change in GAD-7: –8.5 [IQR 6.5] vs –1.0 [IQR 5.0]; P<.001; 95% CI 6-10). Increases in mindfulness at 1 month (nonreactivity subscale) mediated decreases in worry at 2 months (Penn State Worry Questionnaire; P=.02) and decreases in worry at 1 month mediated reductions in anxiety at 2 months (P=.03).

Conclusions

To our knowledge, this is the first report on the efficacy and mechanism of an app-delivered MT for GAD. These findings demonstrate the clinical efficacy of MT as a digital therapeutic for individuals with anxiety (number needed to treat=1.6). These results also link recent advances in our mechanistic understanding of anxiety with treatment development, showing that app-delivered MT targets key reinforcement learning pathways, resulting in tangible, clinically meaningful reductions in worry and anxiety. Evidence-based, mechanistically targeted digital therapeutics have the potential to improve health at a population level at a low cost.

Trial Registration

ClinicalTrials.gov NCT03683472; https://clinicaltrials.gov/ct2/show/NCT03683472

Striatal BOLD and Midfrontal Theta Power Express Motivation for Action

Action selection is biased by the valence of anticipated outcomes. To assess mechanisms by which these motivational biases are expressed and controlled, we measured simultaneous EEG-fMRI during a motivational Go/NoGo learning task (N = 36), leveraging the temporal resolution of EEG and subcortical access of fMRI. VmPFC BOLD encoded cue valence, importantly predicting trial-by-trial valence-driven response speed differences and EEG theta power around cue onset. In contrast, striatal BOLD encoded selection of active Go responses and correlated with theta power around response time. Within trials, theta power ramped in the fashion of an evidence accumulation signal for the value of making a "Go" response, capturing the faster responding to reward cues. Our findings reveal a dual nature of midfrontal theta power, with early components reflecting the vmPFC contribution to motivational biases, and late components reflecting their striatal translation into behavior, in line with influential recent "value of work" theories of striatal processing.

Effectiveness of mental simulations on the early mobilization of patients after cesarean section: a randomized controlled trial

We aimed to investigate whether psychological intervention (single mental simulation) among women after cesarean surgery (CC) can affect their willingness to verticalize, actual verticalization, and the duration of the first mobilization. In this prospective randomised, controlled study, 150 women after CC were divided into 3 groups: experimental group with process-simulation with elements of relaxation, experimental group with outcome-simulation with elements of relaxation and control group with elements of relaxation only. After a 5-h stay in the post-operative room, women listened to a recording with a stimulation. Pain and anxiety of verticalization were measured before and after listening to the recording and after verticalization. Almost 12% more patients verticalized in the process-simulation group than in the control group. Percentages of mobilized patients were: 39.4% the process-simulation group; 32.8% in the outcome-simulation group; 27.7% controls (p = 0.073). Mobilization was 5 min longer in the process-simulation group then in control (p < 0.01). Anxiety after the simulation was a significant covariate of the willingness to verticalize, actual verticalization and time spent in mobilization. We conclude that a single mental simulation can effectively motivate patients for their first verticalization after CC. Perceived anxiety before verticalization may affect the effectiveness of interventions, so we recommend to check it at the postoperative care.

ClinicalTrials.gov Identifier: NCT04829266.

Acute stress blunts prediction error signals in the dorsal striatum during reinforcement learning

Acute stress is pervasive in everyday modern life and is thought to affect how people make choices and learn from them. Reinforcement learning, which implicates learning from the unexpected rewarding and punishing outcomes of our choices (i.e., prediction errors), is critical for adjusted behaviour and seems to be affected by acute stress. However, the neural mechanisms by which acute stress disrupts this type of learning are still poorly understood. Here, we investigate whether and how acute stress blunts neural signalling of prediction errors during reinforcement learning using model-based functional magnetic resonance imaging. Male participants completed a well-established reinforcement-learning task involving monetary gains and losses whilst under stress and control conditions. Acute stress impaired participants’ (n = 23) behavioural performance towards obtaining monetary gains (p < 0.001), but not towards avoiding losses (p = 0.57). Importantly, acute stress blunted signalling of prediction errors during gain and loss trials in the dorsal striatum (p = 0.040) — with subsidiary analyses suggesting that acute stress preferentially blunted signalling of positive prediction errors. Our results thus reveal a neurocomputational mechanism by which acute stress may impair reward learning.

An informatics approach to examine decision-making impairments in the daily life of individuals with depression

Mental health informatics studies methods that collect, model, and interpret a wide variety of data to generate useful information with theoretical or clinical relevance to improve mental health and mental health care. This article presents a mental health informatics approach that is based on the decision-making theory of depression, whereby daily life data from a natural sequential decision-making task are collected and modeled using a reinforcement learning method. The model parameters are then estimated to uncover specific aspects of decision-making impairment in individuals with depression. Empirical results from a pilot study conducted to examine decision-making impairments in the daily lives of university students with depression are presented to illustrate this approach. Future research can apply and expand on this approach to investigate a variety of daily life situations and psychiatric conditions and to facilitate new informatics applications. Using this approach in mental health research may generate useful information with both theoretical and clinical relevance and high ecological validity.

Psychiatric symptoms influence reward-seeking and loss-avoidance decision-making through common and distinct computational processes

AIM

Psychiatric symptoms are often accompanied by impairments in decision-making to attain rewards and avoid losses. However, due to the complex nature of mental disorders (e.g., high comorbidity), symptoms that are specifically associated with deficits in decision-making remain unidentified. Furthermore, the influence of psychiatric symptoms on computations underpinning reward-seeking and loss-avoidance decision-making remains elusive. Here, we aim to address these issues by leveraging a large-scale online experiment and computational modeling.

METHODS

In the online experiment, we recruited 1900 non-diagnostic participants from the general population. They performed either a reward-seeking or loss-avoidance decision-making task, and subsequently completed questionnaires about psychiatric symptoms.

RESULTS

We found that one trans-diagnostic dimension of psychiatric symptoms related to compulsive behavior and intrusive thought (CIT) was negatively correlated with overall decision-making performance in both the reward-seeking and loss-avoidance tasks. A deeper analysis further revealed that, in both tasks, the CIT psychiatric dimension was associated with lower preference for the options that recently led to better outcomes (i.e. reward or no-loss). On the other hand, in the reward-seeking task only, the CIT dimension was associated with lower preference for recently unchosen options.

CONCLUSION

These findings suggest that psychiatric symptoms influence the two types of decision-making, reward-seeking and loss-avoidance, through both common and distinct computational processes.

How Representative are Neuroimaging Samples? Large-Scale Evidence for Trait Anxiety Differences Between fMRI and Behaviour-Only Research Participants

Over the past three decades, functional magnetic resonance imaging (fMRI) has become crucial to study how cognitive processes are implemented in the human brain. However, the question of whether participants recruited into fMRI studies differ from participants recruited into other study contexts has received little to no attention. This is particularly pertinent when effects fail to generalize across study contexts: for example, a behavioural effect discovered in a non-imaging context not replicating in a neuroimaging environment. Here, we tested the hypothesis, motivated by preliminary findings (N = 272), that fMRI participants differ from behaviour-only participants on one fundamental individual difference variable: trait anxiety. Analysing trait anxiety scores and possible confounding variables from healthy volunteers across multiple institutions (N = 3317), we found robust support for lower trait anxiety in fMRI study participants, consistent with a sampling or self-selection bias. The bias was larger in studies that relied on phone screening (compared with full in-person psychiatric screening), recruited at least partly from convenience samples (compared with community samples), and in pharmacology studies. Our findings highlight the need for surveying trait anxiety at recruitment and for appropriate screening procedures or sampling strategies to mitigate this bias.

Computational approaches and machine learning for individual-level treatment predictions

RATIONALE

The impact of neuroscience-based approaches for psychiatry on pragmatic clinical decision-making has been limited. Although neuroscience has provided insights into basic mechanisms of neural function, these insights have not improved the ability to generate better assessments, prognoses, diagnoses, or treatment of psychiatric conditions.

OBJECTIVES

To integrate the emerging findings in machine learning and computational psychiatry to address the question: what measures that are not derived from the patient's self-assessment or the assessment by a trained professional can be used to make more precise predictions about the individual's current state, the individual's future disease trajectory, or the probability to respond to a particular intervention?

RESULTS

Currently, the ability to use individual differences to predict differential outcomes is very modest possibly related to the fact that the effect sizes of interventions are small. There is emerging evidence of genetic and neuroimaging-based heterogeneity of psychiatric disorders, which contributes to imprecise predictions. Although the use of machine learning tools to generate clinically actionable predictions is still in its infancy, these approaches may identify subgroups enabling more precise predictions. In addition, computational psychiatry might provide explanatory disease models based on faulty updating of internal values or beliefs.

CONCLUSIONS

There is a need for larger studies, clinical trials using machine learning, or computational psychiatry model parameters predictions as actionable outcomes, comparing alternative explanatory computational models, and using translational approaches that apply similar paradigms and models in humans and animals.

Affective Bias Through the Lens of Signal Detection Theory

Affective bias – a propensity to focus on negative information at the expense of positive information – is a core feature of many mental health problems. However, it can be caused by wide range of possible underlying cognitive mechanisms. Here we illustrate this by focusing on one particular behavioural signature of affective bias – increased tendency of anxious/depressed individuals to predict lower rewards – in the context of the Signal Detection Theory (SDT) modelling framework. Specifically, we show how to apply this framework to measure affective bias and compare it to the behaviour of an optimal observer. We also show how to extend the framework to make predictions about bias when the individual holds incorrect assumptions about the decision context. Building on this theoretical foundation, we propose five experiments to test five hypothetical sources of this affective bias: beliefs about prior probabilities, beliefs about performance, subjective value of reward, learning differences, and need for accuracy differences. We argue that greater precision about the mechanisms driving affective bias may eventually enable us to better understand the mechanisms underlying mood and anxiety disorders.

Threat of shock promotes passive avoidance, but not active avoidance

Anxiety and stress are adaptive responses to threat that promote harm avoidance. In particular, prior work has shown that anxiety induced in humans using threat of unpredictable shock promotes behavioral inhibition in the face of harm. This is consistent with the idea that anxiety promotes passive avoidance-that is, withholding approach actions that could lead to harm. However, harm can also be avoided through active avoidance, where a (withdrawal) action is taken to avoid harm. Here, we provide the first direct within-study comparison of the effects of threat of shock on active and passive avoidance. We operationalize passive avoidance as withholding a button press response in the face of negative outcomes, and active avoidance as lifting/releasing a button press in the face of negative outcomes. We explore the impact of threat of unpredictable shock on the learning of these behavioral responses (alongside matched responses to rewards) within a single cognitive task. We predicted that threat of shock would promote both active and passive avoidance, and that this would be driven by increased reliance on Pavlovian bias, as parameterized within reinforcement-learning models. Consistent with our predictions, we provide evidence that threat of shock promotes passive avoidance as conceptualized by our task. However, inconsistent with predictions, we found no evidence that threat of shock promoted active avoidance, nor evidence of elevated Pavlovian bias in any condition. One hypothetical framework with which to understand these findings is that anxiety promotes passive over active harm avoidance strategies in order to conserve energy while avoiding harm.

Anxiety disorders

Anxiety disorders form the most common group of mental disorders and generally start before or in early adulthood. Core features include excessive fear and anxiety or avoidance of perceived threats that are persistent and impairing. Anxiety disorders involve dysfunction in brain circuits that respond to danger. Risk for anxiety disorders is influenced by genetic factors, environmental factors, and their epigenetic relations. Anxiety disorders are often comorbid with one another and with other mental disorders, especially depression, as well as with somatic disorders. Such comorbidity generally signifies more severe symptoms, greater clinical burden, and greater treatment difficulty. Reducing the large burden of disease from anxiety disorders in individuals and worldwide can be best achieved by timely, accurate disease detection and adequate treatment administration, scaling up of treatments when needed. Evidence-based psychotherapy (particularly cognitive behavioural therapy) and psychoactive medications (particularly serotonergic compounds) are both effective, facilitating patients' choices in therapeutic decisions. Although promising, no enduring preventive measures are available, and, along with frequent therapy resistance, clinical needs remain unaddressed. Ongoing research efforts tackle these problems, and future efforts should seek individualised, more effective approaches for treatment with precision medicine.

Elevated costly avoidance in anxiety disorders: Patients show little downregulation of acquired avoidance in face of competing rewards for approach

BACKGROUND

Pathological avoidance is a transdiagnostic characteristic of anxiety disorders. Avoidance conditioning re-emerged as a translational model to examine mechanisms and treatment of avoidance. However, its validity for anxiety disorders remains unclear.

METHODS

This study tested for altered avoidance in patients with anxiety disorders compared to matched controls (n = 40/group) using instrumental conditioning assessing low-cost avoidance (avoiding a single aversive outcome) and costly avoidance (avoidance conflicted with gaining rewards). Autonomic arousal and threat expectancy were assessed as indicators of conditioned fear. Associations with dimensional symptom severity were examined.

RESULTS

Patients and controls showed frequent low-cost avoidance without group differences. Controls subsequently inhibited avoidance to gain rewards, which was amplified when aversive outcomes discontinued. In contrast, patients failed to reduce avoidance when aversive and positive outcomes competed (elevated costly avoidance) and showed limited reduction when aversive outcomes discontinued (persistent costly avoidance). Interestingly, elevated costly avoidance was not linked to higher conditioned fear in patients. Moreover, individual data revealed a bimodal distribution of costly avoidance: Some patients showed persistent avoidance, others showed little to no avoidance. Persistent versus low avoiders did not differ in other task-related variables, response to gains and losses in absence of threat, sociodemographic data, or clinical characteristics.

CONCLUSIONS

Findings suggest that anxious psychopathology is associated with a deficit to inhibit avoidance in presence of competing positive outcomes. This offers novel perspectives for research on mechanisms and treatment of anxiety disorders.

Increased Anxiety is Associated with Better Learning from Negative Feedback

Anxiety is one of the most prevalent mental health problems; it is known to impede cognitive functioning. It is believed to alter preferences for feedback-based learning in anxious and non-anxious learners. Thus, the present study measured feedback processing in adults ( N = 30) with and without anxiety symptoms using a probabilistic learning task. Event-related potential (ERP) measures were used to assess how the bias for either positive or negative feedback learning is reflected by the feedback-related negativity component (FRN), an ERP extracted from the electroencephalogram. Anxious individuals, identified by means of the Penn State Worry Questionnaire, showed a diminished FRN and increased accuracy after negative compared to positive feedback. Non-anxious individuals exhibited the reversed pattern with better learning from positive feedback, highlighting their preference for positive feedback. Our ERP results imply that impairments with feedback-based learning in anxious individuals are due to alterations in the mesolimbic dopaminergic system. Our finding that anxious individuals seem to favor negative as opposed to positive feedback has important implications for teacher–student feedback communication.

Overcoming Pavlovian bias in semantic space

Action is invigorated in the presence of reward-predicting stimuli and inhibited in the presence of punishment-predicting stimuli. Although valuable as a heuristic, this Pavlovian bias can also lead to maladaptive behaviour and is implicated in addiction. Here we explore whether Pavlovian bias can be overcome through training. Across five experiments, we find that Pavlovian bias is resistant to unlearning under most task configurations. However, we demonstrate that when subjects engage in instrumental learning in a verbal semantic space, as opposed to a motoric space, not only do they exhibit the typical Pavlovian bias, but this Pavlovian bias diminishes with training. Our results suggest that learning within the semantic space is necessary, but not sufficient, for subjects to unlearn their Pavlovian bias, and that other task features, such as gamification and spaced stimulus presentation may also be necessary. In summary, we show that Pavlovian bias, whilst robust, is susceptible to change with experience, but only under specific environmental conditions.

Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry

Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI.

Social reinforcement learning as a predictor of real-life experiences in individuals with high and low depressive symptomatology

BACKGROUND

Several studies have reported diminished learning from non-social outcomes in depressed individuals. However, it is not clear how depression impacts learning from social feedback. Notably, mood disorders are commonly associated with deficits in social functioning, which raises the possibility that potential impairments in social learning may negatively affect real-life social experiences in depressed subjects.

METHODS

Ninety-two participants with high (HD; N = 40) and low (LD; N = 52) depression scores were recruited. Subjects performed a learning task, during which they received monetary outcomes or social feedback which they were told came from other people. Additionally, participants answered questions about their everyday social experiences. Computational models were fit to the data and model parameters were related to social experience measures.

RESULTS

HD subjects reported a reduced quality and quantity of social experiences compared to LD controls, including an increase in the amount of time spent in negative social situations. Moreover, HD participants showed lower learning rates than LD subjects in the social condition of the task. Interestingly, across all participants, reduced social learning rates predicted higher amounts of time spent in negative social situations, even when depression scores were controlled for.

CONCLUSION

These findings indicate that deficits in social learning may affect the quality of everyday social experiences. Specifically, the impaired ability to use social feedback to appropriately update future actions, which was observed in HD subjects, may lead to suboptimal interpersonal behavior in real life. This, in turn, may evoke negative feedback from others, thus bringing about more unpleasant social encounters.

The Overlapping Neurobiology of Induced and Pathological Anxiety: A Meta-Analysis of Functional Neural Activation

OBJECTIVE

Although anxiety can be an adaptive response to unpredictable threats, pathological anxiety disorders occur when symptoms adversely affect daily life. Whether or not adaptive and pathological anxiety share mechanisms remains unknown, but if they do, induced (adaptive) anxiety could be used as an intermediate translational model of pathological anxiety to improve drug development pipelines. The authors therefore compared meta-analyses of functional neuroimaging studies of induced and pathological anxiety.

METHODS

A systematic search of the PubMed database was conducted in June 2019 for whole-brain functional MRI articles. Eligible articles contrasted either anxious patients to control subjects or an unpredictable-threat condition to a safe condition in healthy participants. Five anxiety disorders were included: posttraumatic stress disorder, social anxiety disorder, generalized anxiety disorder, panic disorder, and specific phobia. A total of 3,433 records were identified, 181 articles met selection criteria, and the largest subset of task type was emotional (N=138). Seed-based d-mapping software was used for all analyses.

RESULTS

Induced anxiety (N=693 participants) and pathological anxiety (N=2,554 patients and 2,348 control subjects) both showed increased activation in the left and right insula (coordinates, 44, 14, -14 and -38, 20, -8; k=2,102 and k=1,305, respectively) and cingulate cortex/medial prefrontal cortex (-12, -8, 68; k=2,217). When the analyses were split by disorder, specific phobia appeared the most, and generalized anxiety disorder the least, similar to induced anxiety.

CONCLUSIONS

This meta-analysis indicates a consistent pattern of activation across induced and pathological anxiety, supporting the proposition that some neurobiological mechanisms overlap and that the former may be used as a model for the latter. Induced anxiety might nevertheless be a better model for some anxiety disorders than others.

Great Expectations: A Critical Review of and Suggestions for the Study of Reward Processing as a Cause and Predictor of Depression

Both human and animal studies support the relationship between depression and reward processing abnormalities, giving rise to the expectation that neural signals of these processes may serve as biomarkers or mechanistic treatment targets. Given the great promise of this research line, we scrutinized those findings and the theoretical claims that underlie them. To achieve this, we applied the framework provided by classical work on causality as well as contemporary approaches to prediction. We identified a number of conceptual, practical, and analytical challenges to this line of research and used a preregistered meta-analysis to quantify the longitudinal associations between reward processing abnormalities and depression. We also investigated the impact of measurement error on reported data. We found that reward processing abnormalities do not reach levels that would be useful for clinical prediction, yet the available evidence does not preclude a possible causal role in depression.

Advances in the computational understanding of mental illness

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

Impaired adaptation of learning to contingency volatility in internalizing psychopathology

Using a contingency volatility manipulation, we tested the hypothesis that difficulty adapting probabilistic decision-making to second-order uncertainty might reflect a core deficit that cuts across anxiety and depression and holds regardless of whether outcomes are aversive or involve reward gain or loss. We used bifactor modeling of internalizing symptoms to separate symptom variance common to both anxiety and depression from that unique to each. Across two experiments, we modeled performance on a probabilistic decision-making under volatility task using a hierarchical Bayesian framework. Elevated scores on the common internalizing factor, with high loadings across anxiety and depression items, were linked to impoverished adjustment of learning to volatility regardless of whether outcomes involved reward gain, electrical stimulation, or reward loss. In particular, high common factor scores were linked to dampened learning following better-than-expected outcomes in volatile environments. No such relationships were observed for anxiety- or depression-specific symptom factors.

Individual differences in costly fearful avoidance and the relation to psychophysiology

Excessive avoidance behaviour is a cardinal symptom of anxiety disorders. Avoidance is not only associated with the benefits of avoiding threats, but also with the costs of missing out on rewards upon exploration. Psychological and psychophysiological mechanisms contributing to these costly avoidance decisions in prospect of mixed outcomes remain unclear. We developed a novel Fearful Avoidance Task (FAT) that resembles characteristics of real-life approach-avoidance conflicts, enabling to disentangle reward and threat effects. Using the FAT, we investigated individual differences in avoidance behaviour and anticipatory psychophysiological states (i.e. startle reflex and skin conductance) in a relatively large sample of 343 (78 females) participants. Avoidance under acute threat of shock depends on a trade-off between perceived reward and threat. Both increased startle and skin conductance in the absence of threat of shock emerged as predictors of increased avoidance (potentially indicative of fear generalization). Increased avoidance was also associated with female sex and trait anxiety, dependent on reward and threat levels. Our findings highlight distinct possible predictors of heightened avoidance and add to mechanistic understanding of how individual propensity for costly avoidance may emerge. Distinct avoidance typologies based on differential reward and threat sensitivities may have different mechanistic origins and thereby could benefit from different treatment strategies.

Adolescents exhibit reduced Pavlovian biases on instrumental learning

Multiple learning systems allow individuals to flexibly respond to opportunities and challenges present in the environment. An evolutionarily conserved "Pavlovian" learning mechanism couples valence and action, promoting a tendency to approach cues associated with reward and to inhibit action in the face of anticipated punishment. Although this default response system may be adaptive, these hard-wired reactions can hinder the ability to learn flexible "instrumental" actions in pursuit of a goal. Such constraints on behavioral flexibility have been studied extensively in adults. However, the extent to which these valence-specific response tendencies bias instrumental learning across development remains poorly characterized. Here, we show that while Pavlovian response biases constrain flexible action learning in children and adults, these biases are attenuated in adolescents. This adolescent-specific reduction in Pavlovian bias may promote unbiased exploration of approach and avoidance responses, facilitating the discovery of rewarding behavior in the many novel contexts that adolescents encounter.

Abnormal night sleep duration and inappropriate sleep initiation time are associated with elevated anxiety symptoms in Chinese rural adults: the Henan Rural Cohort

Sleep problems have been demonstrated to cause mental symptoms, such as anxiety. However, research on the association of the night sleep duration and sleep initiation time on anxiety symptoms in rural China is still lacking. The current study, therefore, explored the independent and combined association of the night sleep duration and sleep initiation time on anxiety symptoms. This study included 28, 054 participants from the Henan Rural Cohort. Sleep was measured using the Pittsburgh Sleep Quality Index (PSQI). Anxiety was assessed by the two-item Generalized Anxiety Disorder scale (GAD-2). Multivariable logistic regression models and restricted cubic spline with anxiety symptoms as a dependent variable were fitted. Among the participants in this study, 11, 209 (39.96%) were males, and 16,845 (60.04%) were females, 1574 (5.61%) had anxiety symptoms. Both shorter and longer night sleep duration were significantly related to elevated prevalence of anxiety symptoms. Extreme sleep initiation time was also significantly associated with elevated anxiety symptoms. Additionally, night sleep duration and sleep initiation time had a combined effect on the prevalent anxiety symptoms. In conclusion, there was a dose-response association of night sleep duration and sleep initiation time with anxiety among Chinese rural population. Moreover, they might jointly increase the odds of prevalent anxiety.