Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy

Using Drift Diffusion and RL Models to Disentangle Effects of Depression On Decision-Making vs. Learning in the Probabilistic Reward Task

The Probabilistic Reward Task (PRT) is widely used to investigate the impact of Major Depressive Disorder (MDD) on reinforcement learning (RL), and recent studies have used it to provide insight into decision-making mechanisms affected by MDD. The current project used PRT data from unmedicated, treatment-seeking adults with MDD to extend these efforts by: (1) providing a more detailed analysis of standard PRT metrics-response bias and discriminability-to better understand how the task is performed; (2) analyzing the data with two computational models and providing psychometric analyses of both; and (3) determining whether response bias, discriminability, or model parameters predicted responses to treatment with placebo or the atypical antidepressant bupropion. Analysis of standard metrics replicated recent work by demonstrating a dependency between response bias and response time (RT), and by showing that reward totals in the PRT are governed by discriminability. Behavior was well-captured by the Hierarchical Drift Diffusion Model (HDDM), which models decision-making processes; the HDDM showed excellent internal consistency and acceptable retest reliability. A separate "belief" model reproduced the evolution of response bias over time better than the HDDM, but its psychometric properties were weaker. Finally, the predictive utility of the PRT was limited by small samples; nevertheless, depressed adults who responded to bupropion showed larger pre-treatment starting point biases in the HDDM than non-responders, indicating greater sensitivity to the PRT's asymmetric reinforcement contingencies. Together, these findings enhance our understanding of reward and decision-making mechanisms that are implicated in MDD and probed by the PRT.

Depression Severity Moderates Reward Learning Among Smokers With Current or Past Major Depressive Disorder in a Smoking Cessation Randomized Clinical Trial

INTRODUCTION

Behavioral and pharmacological smoking cessation treatments are hypothesized to increase patients' reward learning to reduce craving. Identifying changes in reward learning processes that support effective tobacco-dependence interventions among smokers who experience depression may guide patients toward efficient treatment strategies. The objective was to investigate the extent to which adult daily cigarette smokers with current or past major depressive disorder (MDD) learned to seek reward during 12 weeks of treatment combining behavioral activation and varenicline. We hypothesized that a decline in reward learning would be attenuated (least to most) in the following order: (1) behavioral activation integrated with ST (BASC) + varenicline, (2) BASC + placebo, (3) standard behavioral cessation treatment (ST) + varenicline, (4) ST + placebo.

METHODS

We ran a phase IV, placebo-controlled, randomized clinical trial with 300 participants receiving 12 weeks of one of four conditions across two urban medical centers. Depressive symptoms were measured using the Beck Depression Inventory-II (BDI). Reward learning was ascertained at weeks 1, 7, and 14 using the Probabilistic Reward Task (PRT), a laboratory task that uses an asymmetric reinforcement schedule to assess (a) learning to seek reward (response bias), (b) differentiate between stimuli, and (c) time to react to cues.

RESULTS

There was a significant interaction of BDI group × PRT response bias. Response bias declined from weeks 7 to 14 among participants with high baseline depression symptoms. The other two BDI groups showed no change in response bias.

CONCLUSIONS

Controlling for baseline depression, participants showed a decrease in response bias from weeks 1 to 14, and from weeks 7 to 14. Treatment condition and abstinence status were unassociated with change in reward learning.

IMPLICATIONS

Smokers who report greater depression severity show a decline in reward learning despite their participation in smoking cessation treatments, suggesting that depressed populations pose unique challenges with standard smoking cessation approaches.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02378714.

Chronic Ethanol Exposure Produces Persistent Impairment in Cognitive Flexibility and Decision Signals in the Striatum

Lack of cognitive flexibility is a hallmark of substance use disorders and has been associated with drug-induced synaptic plasticity in the dorsomedial striatum (DMS). Yet the possible impact of altered plasticity on real-time striatal neural dynamics during decision-making is unclear. Here, we identified persistent impairments induced by chronic ethanol (EtOH) exposure on cognitive flexibility and striatal decision signals. After a substantial withdrawal period from prior EtOH vapor exposure, male, but not female, rats exhibited reduced adaptability and exploratory behavior during a dynamic decision-making task. Reinforcement learning models showed that prior EtOH exposure enhanced learning from rewards over omissions. Notably, neural signals in the DMS related to the decision outcome were enhanced, while those related to choice and choice-outcome conjunction were reduced, in EtOH-treated rats compared to the controls. These findings highlight the profound impact of chronic EtOH exposure on adaptive decision-making, pinpointing specific changes in striatal representations of actions and outcomes as underlying mechanisms for cognitive deficits.

Impact of AI-Powered Solutions in Rehabilitation Process: Recent Improvements and Future Trends

Rehabilitation is an important and necessary part of local and global healthcare services along with treatment and palliative care, prevention of disease, and promotion of good health. The rehabilitation process helps older and young adults even children to become as independent as possible in activities of daily life and enables participation in useful living activities, recreation, work, and education. The technology of Artificial Intelligence (AI) has evolved significantly in recent years. Many activities related to rehabilitation have been getting benefits from using AI techniques. The objective of this review study is to explore the advantages of AI for rehabilitation and how AI is impacting the rehabilitation process. This study aims at the most critical aspects of the rehabilitation process that could potentially take advantage of AI techniques including personalized rehabilitation apps, rehabilitation through assistance, rehabilitation for neurological disorders, rehabilitation for developmental disorders, virtual reality rehabilitation, rehabilitation of neurodegenerative diseases and Telerehabilitation of Cardiovascular. We presented a survey on the newest empirical studies available in the literature including the AI-based technology helpful in the Rehabilitation process. The novelty feature included but was not limited to an overview of the technological solutions useful in rehabilitation. Seven different categories were identified. Illustrative examples of practical applications were detailed. Implications of the findings for both research and practice were critically discussed. Most of the AI applications in these rehabilitation types are in their infancy and continue to grow while exploring new opportunities. Therefore, we investigate the role of AI technology in rehabilitation processes. In addition, we do statistical analysis of the selected studies to highlight the significance of this review work. In the end, we also present a discussion on some challenges, and future research directions.

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

A diffusion decision model analysis of the cognitive effects of neurofeedback for ADHD

OBJECTIVE

To examine cognitive effects of neurofeedback (NF) for attention-deficit hyperactivity disorder (ADHD) as a secondary outcome of a randomized clinical trial.

METHOD

In a double-blind randomized clinical trial (NCT02251743), 133 7-10-year olds with ADHD received either 38 sessions of NF (n = 78) or control treatment (n = 55) and performed an integrated visual and auditory continuous performance test at baseline, mid- and end-treatment. We used the diffusion decision model to decompose integrated visual and auditory continuous performance test performance at each assessment into cognitive components: efficiency of integrating stimulus information (v), context sensitivity (cv), response cautiousness (a), response bias (z/a), and nondecision time for perceptual encoding and response execution (Ter). Based on prior findings, we tested whether the components known to be deficient improved with NF and explored whether other cognitive components improved using linear mixed modeling.

RESULTS

Before NF, children with ADHD showed main deficits in integrating stimulus information (v), which led to less accurate and slower responses than healthy controls (p = .008). The NF group showed significantly more improvement in integrating auditory stimulus information (v) than control treatment (significant group-by-time-by-modality effect: p = .044).

CONCLUSIONS

NF seems to improve v, deficient in ADHD. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Depression is associated with reduced outcome sensitivity in a dual valence, magnitude learning task

BACKGROUND

Learning from rewarded and punished choices is perturbed in depressed patients, suggesting that abnormal reinforcement learning may be a cognitive mechanism of the illness. However, previous studies have disagreed about whether this behavior is produced by alterations in the rate of learning or sensitivity to experienced outcomes. This previous work has generally assessed learning in response to binary outcomes of one valence, rather than to both rewarding and punishing continuous outcomes.

METHODS

A novel drifting reward and punishment magnitude reinforcement-learning task was administered to patients with current (n = 40) and remitted depression (n = 39), and healthy volunteers (n = 40) to capture potential differences in learning behavior. Standard questionnaires were administered to measure self-reported depressive symptom severity, trait and state anxiety and level of anhedonic symptoms.

RESULTS

Our findings demonstrate that patients with current depression adjust their learning behaviors to a lesser degree in response to trial-by-trial variations in reward and loss magnitudes than the other groups. Computational modeling revealed that this behavioral signature of current depressive state is better accounted for by reduced reward and punishment sensitivity (all p < 0.031), rather than a change in learning rate (p = 0.708). However, between-group differences were not related to self-reported symptom severity or comorbid anxiety disorders in the current depression group.

CONCLUSION

These findings suggest that current depression is associated with reduced outcome sensitivity rather than altered learning rate. Previous findings reported in this domain mainly from binary learning tasks seem to generalize to learning from continuous outcomes.

Transdiagnostic inflexible learning dynamics explain deficits in depression and schizophrenia

Deficits in reward learning are core symptoms across many mental disorders. Recent work suggests that such learning impairments arise by a diminished ability to use reward history to guide behaviour, but the neuro-computational mechanisms through which these impairments emerge remain unclear. Moreover, limited work has taken a transdiagnostic approach to investigate whether the psychological and neural mechanisms that give rise to learning deficits are shared across forms of psychopathology. To provide insight into this issue, we explored probabilistic reward learning in patients diagnosed with major depressive disorder (n = 33) or schizophrenia (n = 24) and 33 matched healthy controls by combining computational modelling and single-trial EEG regression. In our task, participants had to integrate the reward history of a stimulus to decide whether it is worthwhile to gamble on it. Adaptive learning in this task is achieved through dynamic learning rates that are maximal on the first encounters with a given stimulus and decay with increasing stimulus repetitions. Hence, over the course of learning, choice preferences would ideally stabilize and be less susceptible to misleading information. We show evidence of reduced learning dynamics, whereby both patient groups demonstrated hypersensitive learning (i.e. less decaying learning rates), rendering their choices more susceptible to misleading feedback. Moreover, there was a schizophrenia-specific approach bias and a depression-specific heightened sensitivity to disconfirmational feedback (factual losses and counterfactual wins). The inflexible learning in both patient groups was accompanied by altered neural processing, including no tracking of expected values in either patient group. Taken together, our results thus provide evidence that reduced trial-by-trial learning dynamics reflect a convergent deficit across depression and schizophrenia. Moreover, we identified disorder distinct learning deficits.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

Reinforcement learning and working memory in mood disorders: A computational analysis in a developmental transdiagnostic sample

BACKGROUND

Mood disorders commonly onset during adolescence and young adulthood and are conceptually and empirically related to reinforcement learning abnormalities. However, the nature of abnormalities associated with acute symptom severity versus lifetime diagnosis remains unclear, and prior research has often failed to disentangle working memory from reward processes.

METHODS

The present sample (N = 220) included adolescents and young adults with a lifetime history of unipolar disorders (n = 127), bipolar disorders (n = 28), or no history of psychopathology (n = 62), and varying severity of mood symptoms. Analyses fitted a reinforcement learning and working memory model to an instrumental learning task that varied working memory load, and tested associations between model parameters and diagnoses or current symptoms.

RESULTS

Current severity of manic or anhedonic symptoms negatively correlated with task performance. Participants reporting higher severity of current anhedonia, or with lifetime unipolar or bipolar disorders, showed lower reward learning rates. Participants reporting higher severity of current manic symptoms showed faster working memory decay and reduced use of working memory.

LIMITATIONS

Computational parameters should be interpreted in the task environment (a deterministic reward learning paradigm), and developmental population. Future work should test replication in other paradigms and populations.

CONCLUSIONS

Results indicate abnormalities in reinforcement learning processes that either scale with current symptom severity, or correspond with lifetime mood diagnoses. Findings may have implications for understanding reward processing anomalies related to state-like (current symptom) or trait-like (lifetime diagnosis) aspects of mood disorders.

A core component of psychological therapy causes adaptive changes in computational learning mechanisms

BACKGROUND

Cognitive distancing is an emotion regulation strategy commonly used in psychological treatment of various mental health disorders, but its therapeutic mechanisms are unknown.

METHODS

935 participants completed an online reinforcement learning task involving choices between pairs of symbols with differing reward contingencies. Half (49.1%) of the sample was randomised to a cognitive self-distancing intervention and were trained to regulate or 'take a step back' from their emotional response to feedback throughout. Established computational (Q-learning) models were then fit to individuals' choices to derive reinforcement learning parameters capturing clarity of choice values (inverse temperature) and their sensitivity to positive and negative feedback (learning rates).

RESULTS

Cognitive distancing improved task performance, including when participants were later tested on novel combinations of symbols without feedback. Group differences in computational model-derived parameters revealed that cognitive distancing resulted in clearer representations of option values (estimated 0.17 higher inverse temperatures). Simultaneously, distancing caused increased sensitivity to negative feedback (estimated 19% higher loss learning rates). Exploratory analyses suggested this resulted from an evolving shift in strategy by distanced participants: initially, choices were more determined by expected value differences between symbols, but as the task progressed, they became more sensitive to negative feedback, with evidence for a difference strongest by the end of training.

CONCLUSIONS

Adaptive effects on the computations that underlie learning from reward and loss may explain the therapeutic benefits of cognitive distancing. Over time and with practice, cognitive distancing may improve symptoms of mental health disorders by promoting more effective engagement with negative information.

Leveraging Decision Science to Characterize Depression

This brief review examines the potential to use decision science to objectively characterize depression. We provide a brief overview of the existing literature examining different domains of decision-making in depression. Because this overview highlights the specific role of reinforcement learning as an important decision process affected in the disorder, we then introduce reinforcement learning modeling and explain how this approach has identified specific reinforcement learning deficits in depression. We conclude with ideas for future research at the intersection of decision science and depression, emphasizing the potential for decision science to help uncover underlying mechanisms and targets for the treatment of depression.

Tonic dopamine and biases in value learning linked through a biologically inspired reinforcement learning model

A hallmark of various psychiatric disorders is biased future predictions. Here we examined the mechanisms for biased value learning using reinforcement learning models incorporating recent findings on synaptic plasticity and opponent circuit mechanisms in the basal ganglia. We show that variations in tonic dopamine can alter the balance between learning from positive and negative reward prediction errors, leading to biased value predictions. This bias arises from the sigmoidal shapes of the dose-occupancy curves and distinct affinities of D1- and D2-type dopamine receptors: changes in tonic dopamine differentially alters the slope of the dose-occupancy curves of these receptors, thus sensitivities, at baseline dopamine concentrations. We show that this mechanism can explain biased value learning in both mice and humans and may also contribute to symptoms observed in psychiatric disorders. Our model provides a foundation for understanding the basal ganglia circuit and underscores the significance of tonic dopamine in modulating learning processes.

Electrical brain activations in young children during a probabilistic reward-learning task are associated with behavioral strategy

Both adults and children learn through feedback which events and choices in the environment are associated with higher probability of reward. This probability reward-learning ability is thought to be supported by the development of fronto-striatal reward circuits. Recent developmental studies have applied computational models of reward learning to investigate such learning in children. However, there has been limited development of task tools capable of measuring the cascade of neural reward-learning processes in children. Using a child-version of a probabilistic reward-learning task while recording event-related-potential (ERP) measures of electrical brain activity, this study examined key processes of reward learning in preadolescents (n=30), namely: (1) reward-feedback sensitivity, as measured by the early reward-related frontal ERP positivity, (2) rapid attentional shifting of processing toward favored visual stimuli, as measured by the N2pc component, and (3) longer-latency attention-related responses to reward feedback as a function of behavior strategies (i.e., Win-Stay-Lose-Shift), as measured by the central-parietal P300. Consistent with our prior work in adults, the behavioral findings indicate that preadolescents could learn stimulus-reward outcome associations, but at varying levels of performance. Neurally, poor preadolescent learners (those with slower learning rates) showed greater reward-related positivity amplitudes relative to good learners, suggesting greater reward sensitivity. We also found attention shifting towards to-be-chosen stimuli, as evidenced by the N2pc, but not to more highly rewarded stimuli. Lastly, we found an effect of behavioral learning strategies (i.e., Win-Stay-Lose-Shift) on the feedback-locked P300 over the parietal cortex. These findings provide novel insights into the key neural processes underlying reinforcement learning in preadolescents.

A Computational Model Reveals Learning Dynamics During Interpretation Bias Training With Clinical Applications

BACKGROUND

Some psychopathologies, including anxiety and irritability, are associated with biases when judging ambiguous social stimuli. Interventions targeting these biases, or interpretation bias training (IBT), are amenable to computational modeling to describe their associative learning mechanisms. Here, we translated ALCOVE (attention learning covering map), a model of category learning, to describe learning in youths with affective psychopathology when training on more positive judgments of ambiguous face emotions.

METHODS

A predominantly clinical sample comprised 71 youths (age range, 8-22 years) representing broad distributions of irritability and anxiety symptoms. Of these, 63 youths were included in the test sample by completing an IBT task with acceptable performance for computational modeling. We used a separate sample of 28 youths to translate ALCOVE for individual estimates of learning rate and generalization. In the test sample, we assessed associations between model learning estimates and irritability, anxiety, their shared variance (negative affectivity), and age.

RESULTS

Age and affective symptoms were associated with category learning during IBT. Lower learning rates were associated with higher negative affectivity common in anxiety and irritability. Lower generalization, or improved discrimination between face emotions, was associated with increasing age.

CONCLUSIONS

This work demonstrates a functional consequence of age- and symptom-related learning during interpretation bias. Learning measured by ALCOVE also revealed learning types not accounted for in the prior literature on IBT. This work more broadly demonstrates the utility of measurement models for understanding trial-by-trial processes and identifying individual learning styles.

Prolonged Physiological Stress Is Associated With a Lower Rate of Exploratory Learning That Is Compounded by Depression

BACKGROUND

Stress is a major risk factor for depression, and both are associated with important changes in decision-making patterns. However, decades of research have only weakly connected physiological measurements of stress to the subjective experience of depression. Here, we examined the relationship between prolonged physiological stress, mood, and explore-exploit decision making in a population navigating a dynamic environment under stress: health care workers during the COVID-19 pandemic.

METHODS

We measured hair cortisol levels in health care workers who completed symptom surveys and performed an explore-exploit restless-bandit decision-making task; 32 participants were included in the final analysis. Hidden Markov and reinforcement learning models assessed task behavior.

RESULTS

Participants with higher hair cortisol exhibited less exploration (r = -0.36, p = .046). Higher cortisol levels predicted less learning during exploration (β = -0.42, false discovery rate [FDR]-corrected p [pFDR] = .022). Importantly, mood did not independently correlate with cortisol concentration, but rather explained additional variance (β = 0.46, pFDR = .022) and strengthened the relationship between higher cortisol and lower levels of exploratory learning (β = -0.47, pFDR = .022) in a joint model. These results were corroborated by a reinforcement learning model, which revealed less learning with higher hair cortisol and low mood (β = -0.67, pFDR = .002).

CONCLUSIONS

These results imply that prolonged physiological stress may limit learning from new information and lead to cognitive rigidity, potentially contributing to burnout. Decision-making measures link subjective mood states to measured physiological stress, suggesting that they should be incorporated into future biomarker studies of mood and stress conditions.

The shadowing effect of initial expectation on learning asymmetry

Evidence for positivity and optimism bias abounds in high-level belief updates. However, no consensus has been reached regarding whether learning asymmetries exist in more elementary forms of updates such as reinforcement learning (RL). In RL, the learning asymmetry concerns the sensitivity difference in incorporating positive and negative prediction errors (PE) into value estimation, namely the asymmetry of learning rates associated with positive and negative PEs. Although RL has been established as a canonical framework in characterizing interactions between agent and environment, the direction of learning asymmetry remains controversial. Here, we propose that part of the controversy stems from the fact that people may have different value expectations before entering the learning environment. Such a default value expectation influences how PEs are calculated and consequently biases subjects' choices. We test this hypothesis in two learning experiments with stable or varying reinforcement probabilities, across monetary gains, losses, and gain-loss mixed environments. Our results consistently support the model incorporating both asymmetric learning rates and the initial value expectation, highlighting the role of initial expectation in value updating and choice preference. Further simulation and model parameter recovery analyses confirm the unique contribution of initial value expectation in accessing learning rate asymmetry.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Most psychiatric disorders do not occur in isolation, and most psychiatric symptom dimensions are not uniquely expressed within a single diagnostic category. Current treatments fail to work for around 25% to 40% of individuals, perhaps due at least in part to an overreliance on diagnostic categories in treatment development and allocation. In this review, we describe ongoing efforts in the field to surmount these challenges and precisely characterize psychiatric symptom dimensions using large-scale studies of unselected samples via remote, online, and "citizen science" efforts that take a dimensional, mechanistic approach. We discuss the importance that efforts to identify meaningful psychiatric dimensions be coupled with careful computational modeling to formally specify, test, and potentially falsify candidate mechanisms that underlie transdiagnostic symptom dimensions. We refer to this approach, i.e., where symptom dimensions are identified and validated against computationally well-defined neurocognitive processes, as computational factor modeling. We describe in detail some recent applications of this method to understand transdiagnostic cognitive processes that include model-based planning, metacognition, appetitive processing, and uncertainty estimation. In this context, we highlight how computational factor modeling has been used to identify specific associations between cognition and symptom dimensions and reveal previously obscured relationships, how findings generalize to smaller in-person clinical and nonclinical samples, and how the method is being adapted and optimized beyond its original instantiation. Crucially, we discuss next steps for this area of research, highlighting the value of more direct investigations of treatment response that bridge the gap between basic research and the clinic.

Belief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches

Alterations in belief updating are proposed to underpin symptoms of psychiatric illness, including psychosis, depression, and anxiety. Key parameters underlying belief updating can be captured using computational modelling techniques, aiding the identification of unique and shared deficits, and improving diagnosis and treatment. We systematically reviewed research that applied computational modelling to probabilistic tasks measuring belief updating in stable and volatile (changing) environments, across clinical and subclinical psychosis (n = 17), anxiety (n = 9), depression (n = 9) and transdiagnostic samples (n = 9). Depression disorders related to abnormal belief updating in response to the valence of rewards, evidenced in both stable and volatile environments. Whereas psychosis and anxiety disorders were associated with difficulties adapting to changing contingencies specifically, indicating an inflexibility and/or insensitivity to environmental volatility. Higher-order learning models revealed additional difficulties in the estimation of overall environmental volatility across psychosis disorders, showing increased updating to irrelevant information. These findings stress the importance of investigating belief updating in transdiagnostic samples, using homogeneous experimental and computational modelling approaches.

Troubleshooting Bayesian cognitive models

Using Bayesian methods to apply computational models of cognitive processes, or Bayesian cognitive modeling, is an important new trend in psychological research. The rise of Bayesian cognitive modeling has been accelerated by the introduction of software that efficiently automates the Markov chain Monte Carlo sampling used for Bayesian model fitting-including the popular Stan and PyMC packages, which automate the dynamic Hamiltonian Monte Carlo and No-U-Turn Sampler (HMC/NUTS) algorithms that we spotlight here. Unfortunately, Bayesian cognitive models can struggle to pass the growing number of diagnostic checks required of Bayesian models. If any failures are left undetected, inferences about cognition based on the model's output may be biased or incorrect. As such, Bayesian cognitive models almost always require troubleshooting before being used for inference. Here, we present a deep treatment of the diagnostic checks and procedures that are critical for effective troubleshooting, but are often left underspecified by tutorial papers. After a conceptual introduction to Bayesian cognitive modeling and HMC/NUTS sampling, we outline the diagnostic metrics, procedures, and plots necessary to detect problems in model output with an emphasis on how these requirements have recently been changed and extended. Throughout, we explain how uncovering the exact nature of the problem is often the key to identifying solutions. We also demonstrate the troubleshooting process for an example hierarchical Bayesian model of reinforcement learning, including supplementary code. With this comprehensive guide to techniques for detecting, identifying, and overcoming problems in fitting Bayesian cognitive models, psychologists across subfields can more confidently build and use Bayesian cognitive models in their research. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Valence-partitioned learning signals drive choice behavior and phenomenal subjective experience in humans

How the human brain generates conscious phenomenal experience is a fundamental problem. In particular, it is unknown how variable and dynamic changes in subjective affect are driven by interactions with objective phenomena. We hypothesize a neurocomputational mechanism that generates valence-specific learning signals associated with 'what it is like' to be rewarded or punished. Our hypothesized model maintains a partition between appetitive and aversive information while generating independent and parallel reward and punishment learning signals. This valence-partitioned reinforcement learning (VPRL) model and its associated learning signals are shown to predict dynamic changes in 1) human choice behavior, 2) phenomenal subjective experience, and 3) BOLD-imaging responses that implicate a network of regions that process appetitive and aversive information that converge on the ventral striatum and ventromedial prefrontal cortex during moments of introspection. Our results demonstrate the utility of valence-partitioned reinforcement learning as a neurocomputational basis for investigating mechanisms that may drive conscious experience.

Individual differences in computational psychiatry: a review of current challenges

Bringing precision to the understanding and treatment of mental disorders requires instruments for studying clinically relevant individual differences. One promising approach is the development of computational assays: integrating computational models with cognitive tasks to infer latent patient-specific disease processes in brain computations. While recent years have seen many methodological advancements in computational modelling and many cross-sectional patient studies, much less attention has been paid to basic psychometric properties (reliability and construct validity) of the computational measures provided by the assays. In this review, we assess the extent of this issue by examining emerging empirical evidence. We find that many computational measures suffer from poor psychometric properties, which poses a risk of invalidating previous findings and undermining ongoing research efforts using computational assays to study individual (and even group) differences. We provide recommendations for how to address these problems and, crucially, embed them within a broader perspective on key developments that are needed for translating computational assays to clinical practice.

Stimulating human prefrontal cortex increases reward learning

Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.

Angiotensin blockade enhances motivational reward learning via enhancing striatal prediction error signaling and frontostriatal communication

Adaptive human learning utilizes reward prediction errors (RPEs) that scale the differences between expected and actual outcomes to optimize future choices. Depression has been linked with biased RPE signaling and an exaggerated impact of negative outcomes on learning which may promote amotivation and anhedonia. The present proof-of-concept study combined computational modeling and multivariate decoding with neuroimaging to determine the influence of the selective competitive angiotensin II type 1 receptor antagonist losartan on learning from positive or negative outcomes and the underlying neural mechanisms in healthy humans. In a double-blind, between-subjects, placebo-controlled pharmaco-fMRI experiment, 61 healthy male participants (losartan, n = 30; placebo, n = 31) underwent a probabilistic selection reinforcement learning task incorporating a learning and transfer phase. Losartan improved choice accuracy for the hardest stimulus pair via increasing expected value sensitivity towards the rewarding stimulus relative to the placebo group during learning. Computational modeling revealed that losartan reduced the learning rate for negative outcomes and increased exploitatory choice behaviors while preserving learning for positive outcomes. These behavioral patterns were paralleled on the neural level by increased RPE signaling in orbitofrontal-striatal regions and enhanced positive outcome representations in the ventral striatum (VS) following losartan. In the transfer phase, losartan accelerated response times and enhanced VS functional connectivity with left dorsolateral prefrontal cortex when approaching maximum rewards. These findings elucidate the potential of losartan to reduce the impact of negative outcomes during learning and subsequently facilitate motivational approach towards maximum rewards in the transfer of learning. This may indicate a promising therapeutic mechanism to normalize distorted reward learning and fronto-striatal functioning in depression.

Self-judgment dissected: A computational modeling analysis of self-referential processing and its relationship to trait mindfulness facets and depression symptoms

Cognitive theories of depression, and mindfulness theories of well-being, converge on the notion that self-judgment plays a critical role in mental health. However, these theories have rarely been tested via tasks and computational modeling analyses that can disentangle the information processes operative in self-judgments. We applied a drift-diffusion computational model to the self-referential encoding task (SRET) collected before and after an 8-week mindfulness intervention (n = 96). A drift-rate regression parameter representing positive-relative to negative-self-referential judgment strength positively related to mindful awareness and inversely related to depression, both at baseline and over time; however, this parameter did not significantly relate to the interaction between mindful awareness and nonjudgmentalness. At the level of individual depression symptoms, at baseline, a spectrum of symptoms (inversely) correlated with the drift-rate regression parameter, suggesting that many distinct depression symptoms relate to valenced self-judgment between subjects. By contrast, over the intervention, changes in only a smaller subset of anhedonia-related depression symptoms showed substantial relationships with this parameter. Both behavioral and model-derived measures showed modest split-half and test-retest correlations. Results support cognitive theories that implicate self-judgment in depression and mindfulness theories, which imply that mindful awareness should lead to more positive self-views.

The computational psychiatry of antisocial behaviour and psychopathy

Antisocial behaviours such as disobedience, lying, stealing, destruction of property, and aggression towards others are common to multiple disorders of childhood and adulthood, including conduct disorder, oppositional defiant disorder, psychopathy, and antisocial personality disorder. These disorders have a significant negative impact for individuals and for society, but whether they represent clinically different phenomena, or simply different approaches to diagnosing the same underlying psychopathology is highly debated. Computational psychiatry, with its dual focus on identifying different classes of disorder and health (data-driven) and latent cognitive and neurobiological mechanisms (theory-driven), is well placed to address these questions. The elucidation of mechanisms that might characterise latent processes across different disorders of antisocial behaviour can also provide important advances. In this review, we critically discuss the contribution of computational research to our understanding of various antisocial behaviour disorders, and highlight suggestions for how computational psychiatry can address important clinical and scientific questions about these disorders in the future.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

Functional MRI study of feedback-based reinforcement learning in depression

Reinforcement learning depends upon the integrity of emotional circuitry to establish associations between environmental cues, decisions, and positive or negative outcomes in order to guide behavior through experience. The emotional dysregulation characteristic of major depressive disorder (MDD) may alter activity in frontal and limbic structures that are key to learning. Although reward and decision-making have been examined in MDD, the effects of depression on associative learning is less well studied. We investigated whether depressive symptoms would be related to abnormalities in learning-related brain activity as measured by functional magnetic resonance imaging (fMRI). Also, we explored whether melancholic and atypical features were associated with altered brain activity. We conducted MRI scans on a 4T Varian MRI system in 10 individuals with MDD and 10 healthy subjects. We examined event-related brain activation during feedback-based learning task using Analysis of Functional NeuroImages (AFNI) for image processing and statistical analysis. We observed that MDD patients exhibited reduced activation in visual cortex but increased activation in cingulate and insular regions compared to healthy participants. Also, in relation to features of depressive subtypes, we observed that levels of activation in striatal, thalamic, and precuneus regions were negatively correlated with atypical characteristics. These results suggest that the effects of MDD change the neural circuitry underlying associative learning, and these effects may depend upon subtype features of MDD.

Translating Big Data to Clinical Outcomes in Anxiety: Potential for Multimodal Integration

PURPOSE OF THE REVIEW

This review describes approaches to research on anxiety that attempt to link neural correlates to treatment response and novel therapies. The review emphasizes pediatric anxiety disorders since most anxiety disorders begin before adulthood.

RECENT FINDINGS

Recent literature illustrates how current treatments for anxiety manifest diverse relations with a range of neural markers. While some studies demonstrate post-treatment normalization of markers in anxious individuals, others find persistence of group differences. For other markers, which show no pretreatment association with anxiety, the markers nevertheless distinguish treatment-responders from non-responders. Heightened error related negativity represents the risk marker discussed in the most depth; however, limitations in measures related to error responding necessitate multimodal and big-data approaches. Single risk markers show limits as correlates of treatment response. Large-scale, multimodal data analyzed with predictive models may illuminate additional risk markers related to anxiety disorder treatment outcomes. Such work may identify novel targets and eventually guide improvements in treatment response/outcomes.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

Even in favorable circumstances, we often find it hard to remain happy with what we have. One might enjoy a newly bought car for a season, but over time it brings fewer positive feelings and one eventually begins dreaming of the next rewarding thing to pursue. Here, we present a series of computational simulations that suggest these presumable “flaws” might play an important role in promoting adaptive behavior. We explore the value of prior expectations and relative comparisons as a useful reward signal and find that across a wide range of environments, these features help an agent learn faster and adapt better to changes in the environment. Our simulations also highlight scenarios when these relative features can be harmful to decision-making and happiness. Together, our results help explain why we have the propensity to keep wanting more, even if it contributes to depression, materialism, and overconsumption.

What Can Reinforcement Learning Models of Dopamine and Serotonin Tell Us about the Action of Antidepressants?

Although evidence suggests that antidepressants are effective at treating depression, the mechanisms behind antidepressant action remain unclear, especially at the cognitive/computational level. In recent years, reinforcement learning (RL) models have increasingly been used to characterise the roles of neurotransmitters and to probe the computations that might be altered in psychiatric disorders like depression. Hence, RL models might present an opportunity for us to better understand the computational mechanisms underlying antidepressant effects. Moreover, RL models may also help us shed light on how these computations may be implemented in the brain (e.g., in midbrain, striatal, and prefrontal regions) and how these neural mechanisms may be altered in depression and remediated by antidepressant treatments. In this paper, we evaluate the ability of RL models to help us understand the processes underlying antidepressant action. To do this, we review the preclinical literature on the roles of dopamine and serotonin in RL, draw links between these findings and clinical work investigating computations altered in depression, and appraise the evidence linking modification of RL processes to antidepressant function. Overall, while there is no shortage of promising ideas about the computational mechanisms underlying antidepressant effects, there is insufficient evidence directly implicating these mechanisms in the response of depressed patients to antidepressant treatment. Consequently, future studies should investigate these mechanisms in samples of depressed patients and assess whether modifications in RL processes mediate the clinical effect of antidepressant treatments.

Recentering neuroscience on behavior: The interface between brain and environment is a privileged level of control of neural activity

Despite the huge and constant progress in the molecular and cellular neuroscience fields, our capability to understand brain alterations and treat mental illness is still limited. Therefore, a paradigm shift able to overcome such limitation is warranted. Behavior and the associated mental states are the interface between the central nervous system and the living environment. Since, in any system, the interface is a key regulator of system organization, behavior is proposed here as a unique and privileged level of control and orchestration of brain structure and activity. This view has relevant scientific and clinical implications. First, the study of behavior represents a singular starting point for the investigation of neural activity in an integrated and comprehensive fashion. Second, behavioral changes, accomplished through psychotherapy or environmental interventions, are expected to have the highest impact to specifically reorganize the complexity of the human mind and thus achieve a solid and long-lasting improvement in mental health.

Explanatory personality science in the neuroimaging era: The map is not the territory

The recent rise of magnetic resonance imaging (MRI) has led to a proliferation of studies that seek to link individual differences in personality directly to their neural correlates. These studies function to describe personality at a lower level of analysis, but they do little to advance the field's understanding of the causal mechanisms that give rise to personality traits. To transition to a more explanatory personality neuroscience, researchers should strive to conduct theory-driven empirical studies that bridge multiple levels of analysis. Effectively doing so will require a continued reliance on rich description, strong theories, large samples, and careful behavioral experimentation. Integrating these components will lead to more robust and informative studies of personality neuroscience that help to move the field closer to explaining the causal sources of individual differences.

Pleasure, Reward Value, Prediction Error and Anhedonia

In order to develop effective treatments for anhedonia we need to understand its underlying neurobiological mechanisms. Anhedonia is conceptually strongly linked to reward processing, which involves a variety of cognitive and neural operations. This chapter reviews the evidence for impairments in experiencing hedonic response (pleasure), reward valuation and reward learning based on outcomes (commonly conceptualised in terms of "reward prediction error"). Synthesising behavioural and neuroimaging findings, we examine case-control studies of patients with depression and schizophrenia, including those focusing specifically on anhedonia. Overall, there is reliable evidence that depression and schizophrenia are associated with disrupted reward processing. In contrast to the historical definition of anhedonia, there is surprisingly limited evidence for impairment in the ability to experience pleasure in depression and schizophrenia. There is some evidence that learning about reward and reward prediction error signals are impaired in depression and schizophrenia, but the literature is inconsistent. The strongest evidence is for impairments in the representation of reward value and how this is used to guide action. Future studies would benefit from focusing on impairments in reward processing specifically in anhedonic samples, including transdiagnostically, and from using designs separating different components of reward processing, formulating them in computational terms, and moving beyond cross-sectional designs to provide an assessment of causality.

Probabilistic Reinforcement Learning and Anhedonia

Despite the prominence of anhedonic symptoms associated with diverse neuropsychiatric conditions, there are currently no approved therapeutics designed to attenuate the loss of responsivity to previously rewarding stimuli. However, the search for improved treatment options for anhedonia has been reinvigorated by a recent reconceptualization of the very construct of anhedonia, including within the Research Domain Criteria (RDoC) initiative. This chapter will focus on the RDoC Positive Valence Systems construct of reward learning generally and sub-construct of probabilistic reinforcement learning specifically. The general framework emphasizes objective measurement of a subject's responsivity to reward via reinforcement learning under asymmetrical probabilistic contingencies as a means to quantify reward learning. Indeed, blunted reward responsiveness and reward learning are central features of anhedonia and have been repeatedly described in major depression. Moreover, these probabilistic reinforcement techniques can also reveal neurobiological mechanisms to aid development of innovative treatment approaches. In this chapter, we describe how investigating reward learning can improve our understanding of anhedonia via the four RDoC-recommended tasks that have been used to probe sensitivity to probabilistic reinforcement contingencies and how such task performance is disrupted in various neuropsychiatric conditions. We also illustrate how reverse translational approaches of probabilistic reinforcement assays in laboratory animals can inform understanding of pharmacological and physiological mechanisms. Next, we briefly summarize the neurobiology of probabilistic reinforcement learning, with a focus on the prefrontal cortex, anterior cingulate cortex, striatum, and amygdala. Finally, we discuss treatment implications and future directions in this burgeoning area.

Beyond Description and Deficits: How Computational Psychiatry Can Enhance an Understanding of Decision-Making in Anorexia Nervosa

PURPOSE OF REVIEW

Despite decades of research, knowledge of the mechanisms maintaining anorexia nervosa (AN) remains incomplete and clearly effective treatments elusive. Novel theoretical frameworks are needed to advance mechanistic and treatment research for this disorder. Here, we argue the utility of engaging a novel lens that differs from existing perspectives in psychiatry. Specifically, we argue the necessity of expanding beyond two historically common perspectives: (1) the descriptive perspective: the tendency to define mechanisms on the basis of surface characteristics and (2) the deficit perspective: the tendency to search for mechanisms associated with under-functioning of decision-making abilities and related circuity, rather than problems of over-functioning, in psychiatric disorders.

RECENT FINDINGS

Computational psychiatry can provide a novel framework for understanding AN because this approach emphasizes the role of computational misalignments (rather than absolute deficits or excesses) between decision-making strategies and environmental demands as the key factors promoting psychiatric illnesses. Informed by this approach, we argue that AN can be understood as a disorder of excess goal pursuit, maintained by over-engagement, rather than disengagement, of executive functioning strategies and circuits. Emerging evidence suggests that this same computational imbalance may constitute an under-investigated phenotype presenting transdiagnostically across psychiatric disorders. A variety of computational models can be used to further elucidate excess goal pursuit in AN. Most traditional psychiatric treatments do not target excess goal pursuit or associated neurocognitive mechanisms. Thus, targeting at the level of computational dysfunction may provide a new avenue for enhancing treatment for AN and related disorders.