Reinforcement learning: the good, the bad and the ugly

Reconceptualized Associative Learning

Research on the role of time in associative learning has changed our understanding of what an association is. It is a measurable fact about the distribution of events in time, not an altered activation-conducting connection in a mind, brain or net. Associative learning is the process of perceiving temporal associations and deciding to act on them. Informativeness- the ratio of a conditional rate to an unconditional rate-is the essential empirical variable, not the probability of reinforcement. The communicated information between temporally associated behavioral and reinforcing events is the log of informativeness. Because the time units in the rate estimates cancel, associative-learning is time-scale invariant: Perceivably associated events may be arbitrarily widely separated. There are no windows of associability nor decaying eligibility traces. The learning rate-operationally defined as the reciprocal of reinforcements prior to the appearance of a conditioned response-is an almost scalar function of relative temporal separation, as measured by informativeness. The central role of informativeness unites our understanding of Pavlovian and operant/instrumental phenomena, revealing unexpected quantitative and conceptual communalities.

Curiosity overpowers cognitive effort avoidance tendencies

Curiosity has been described as a desire to learn new information, and previous studies have demonstrated that curiosity drives peoples' decision to invest resources (e.g., time or tokens) to find out answers. It is commonly assumed that curiosity should also prompt people to invest more effort until they attain unknown answers. However, experimental evidence is lacking on whether people would be willing to exert cognitive effort - in addition to time investments - to find out answers. In three pre-registered experiments, we first asked participants to rate a set of 20 trivia questions regarding their curiosity about knowing the answers. Subsequently, participants had to perform a set of random-dot kinematograms (RDKs) to view the answer to each trivia question. We varied the motion coherence of the RDKs as a proxy for cognitive effort demands and tested whether curiosity overpowers cognitive effort avoidance tendencies. Our results provide converging evidence that curiosity outweighs peoples' tendencies to avoid cognitive effort. That is, participants avoided high-effort RDKs if they were not curious about information and when the exertion of cognitive effort did not affect the attainment of information. However, if participants were curious about questions and if no alternative low-effort option was available, they were willing to employ cognitive effort to find out answers.

Old Strategies, New Environments: Reinforcement Learning on Social Media

The rise of social media has profoundly altered the social world, introducing new behaviors that can satisfy our social needs. However, it is not yet known whether human social strategies, which are well adapted to the offline world we developed in, operate as effectively within this new social environment. Here, we describe how the computational framework of reinforcement learning (RL) can help us to precisely frame this problem and diagnose where behavior-environment mismatches emerge. The RL framework describes a process by which an agent can learn to maximize their long-term reward. RL, which has proven to be successful in characterizing human social behavior, consists of 3 stages: updating expected reward, valuating expected reward by integrating subjective costs such as effort, and selecting an action. Specific social media affordances, such as the quantifiability of social feedback, may interact with the RL process at each of these stages. In some cases, affordances can exploit RL biases that are beneficial offline by violating the environmental conditions under which such biases are optimal, such as when algorithmic personalization of content interacts with confirmation bias. Characterizing the impact of specific aspects of social media through this lens can improve our understanding of how digital environments shape human behavior. Ultimately, this formal framework could help address pressing open questions about social media use, including its changing role across human development and its impact on outcomes such as mental health.

Feature identification learning both shapes and is shaped by spatial object-similarity representations

Object knowledge is bound together in semantic networks that can be spatially represented. How these knowledge representations shape and are in turn shaped by learning remains unclear. Here, we directly examined how object similarity representations impact implicit learning of feature dimensions and how learning, in turn, influences these representations. In a pre-experiment, 237 adult participants arranged object-pictures in a spatial arena, revealing semantic relatedness of everyday objects across categories: activity, fashion, and foods. The subsequent experiment assessed whether these semantic relationships played a role in implicitly learning specific object features in a separate adult participant group (N = 82). Participants inferred the meanings of two pseudo-words through feedback. Using computational modeling, we tested various learning strategies and established that learning was guided by semantic relationships quantified in the pre-experiment. Post-learning arrangements reflected object similarity representations as well as the learned feature. We directly show that similarity representations guide implicit learning and that learning in turn reshapes existing knowledge representations.

Adaptive circuits for action and value information in rodent operant learning

Operant learning is a behavioral paradigm where animals learn to associate their actions with consequences, adapting their behavior accordingly. This review delves into the neural circuits that underpin operant learning in rodents, emphasizing the dynamic interplay between neural pathways, synaptic plasticity, and gene expression changes. We explore the cortico-basal ganglia circuits, highlighting the pivotal role of dopamine in modulating these pathways to reinforce behaviors that yield positive outcomes. We include insights from recent studies, which reveals the intricate roles of midbrain dopamine neurons in integrating action initiation and reward feedback, thereby enhancing movement-related activities in the dorsal striatum. Additionally, we discuss the molecular diversity of striatal neurons and their specific roles in reinforcement learning. The review also covers advances in transcriptome analysis techniques, such as single-cell RNA sequencing, which have provided deeper insights into the gene expression profiles associated with different neuronal populations during operant learning.

A computational framework to study the etiology of grandiose narcissism

Grandiose narcissism is characterized by ambivalent interaction behavior (i.e., grandiose self-presentation and rivalrous devaluation of others) and strong oscillations in self-esteem over time. In the light of emotional and social problems associated with these self-esteem regulation patterns and the increasing prevalence of narcissistic tendencies, causal and formalized models for prevention and intervention are needed. Here, we present a computational model of narcissistic self-esteem regulation implementing established, verbal theories of narcissism to identify key etiological and disorder-maintaining mechanisms. Across four studies, we show that parental praise and overvaluation lead to typical grandiose-narcissistic behavioral patterns (i.e., entitled self-presentation and rivalry) and strong self-esteem oscillations. Underlying these phenomena, we identify two maintaining mechanisms that offer targets for intervention and empirical falsification: tolerance development, characterized by an ever-increasing desire for social recognition, and a vicious cycle of rivalry, characterized by the frequent use of other-devaluing behavior and massive drops in self-esteem.

Distributed representations of temporally accumulated reward prediction errors in the mouse cortex

Reward prediction errors (RPEs) quantify the difference between expected and actual rewards, serving to refine future actions. Although reinforcement learning (RL) provides ample theoretical evidence suggesting that the long-term accumulation of these error signals improves learning efficiency, it remains unclear whether the brain uses similar mechanisms. To explore this, we constructed RL-based theoretical models and used multiregional two-photon calcium imaging in the mouse dorsal cortex. We identified a population of neurons whose activity was modulated by varying degrees of RPE accumulation. Consequently, RPE-encoding neurons were sequentially activated within each trial, forming a distributed assembly. RPE representations in mice aligned with theoretical predictions of RL, emerging during learning and being subject to manipulations of the reward function. Interareal comparisons revealed a region-specific code, with higher-order cortical regions exhibiting long-term encoding of RPE accumulation. These results present an additional layer of complexity in cortical RPE computation, potentially augmenting learning efficiency in animals.

A neurocognitive mechanism for increased cooperation during group formation

How do group size changes influence cooperation within groups? To examine this question, we performed a dynamic, network-based prisoner's dilemma experiment with fMRI. Across 83 human participants, we observed increased cooperation as group size increased. However, our computational modeling analysis of behavior and fMRI revealed that groups size itself did not increase cooperation. Rather, interaction between (1) participants' stable prosocial tendencies, and (2) dynamic reciprocal strategy weighed by memory confidence, underlies the group size-modulated increase in cooperation because the balance between them shifts towards the prosocial tendency with higher memory demands in larger groups. We found that memory confidence was encoded in fusiform gyrus and precuneus, whereas its integration with prosocial tendencies was reflected in the left DLPFC and dACC. Therefore, interaction between recall uncertainty during reciprocal interaction (i.e., forgetting) and one's individual prosocial preference is a core pillar of emergent cooperation in more naturalistic and dynamic group formation.

Early-Life Adversities Are Associated With Lower Expected Value Signaling in the Adult Brain

BACKGROUND

Early adverse experiences are assumed to affect fundamental processes of reward learning and decision making. However, computational neuroimaging studies investigating these circuits in the context of adversity are sparse and limited to studies conducted in adolescent samples, leaving the long-term effects unexplored.

METHODS

Using data from a longitudinal birth cohort study (n = 156; 87 female), we investigated associations between adversities and computational markers of reward learning (i.e., expected value, prediction errors). At age 33 years, all participants completed a functional magnetic resonance imaging-based passive avoidance task. Psychopathology measures were collected at the time of functional magnetic resonance imaging investigation and during the COVID-19 pandemic. We applied a principal component analysis to capture common variations across 7 adversity measures. The resulting adversity factors (factor 1: postnatal psychosocial adversities and prenatal maternal smoking; factor 2: prenatal maternal stress and obstetric adversity; factor 3: lower maternal stimulation) were linked with psychopathology and neural responses in the core reward network using multiple regression analysis.

RESULTS

We found that the adversity dimension primarily informed by lower maternal stimulation was linked to lower expected value representation in the right putamen, right nucleus accumbens, and anterior cingulate cortex. Expected value encoding in the right nucleus accumbens further mediated the relationship between this adversity dimension and psychopathology and predicted higher withdrawn symptoms during the COVID-19 pandemic.

CONCLUSIONS

Our results suggested that early adverse experiences in caregiver context might have a long-term disruptive effect on reward learning in reward-related brain regions, which can be associated with suboptimal decision making and thereby may increase the vulnerability of developing psychopathology.

Global neural encoding of behavioral strategies in mice during perceptual decision-making task with two different sensory patterns

When a simple model-free strategy does not provide sufficient outcomes, an inference-based strategy estimating a hidden task structure becomes essential for optimizing choices. However, the neural circuitry involved in inference-based strategies is still unclear. We developed a tone frequency discrimination task in head-fixed mice in which the tone category of the current trial depended on the category of the previous trial. When the tone category was repeated, the mice continued using the default model-free strategy, as well as when the tone was randomly presented, to bias choices. In contrast, when the tone was alternated, the default strategy gradually shifted to a hybrid of model-free and inference-based strategies, although we did not observe distinct strategy changes. Brain-wide electrophysiological recording suggested that the neural activity of the frontal and sensory cortices, hippocampus, and striatum was correlated with the reward expectation in different task conditions, suggesting the global encoding of multiple strategies in the brain.

Cortical norepinephrine-astrocyte signaling critically mediates learned behavior

Updating behavior based on feedback from the environment is a crucial means by which organisms learn and develop optimal behavioral strategies1-3. Norepinephrine (NE) release from the locus coeruleus (LC) has been shown to mediate learned behaviors4-6 such that in a task with graded stimulus uncertainty and performance, a high level of NE released after an unexpected outcome causes improvement in subsequent behavior7. Yet, how the transient activity of LC-NE neurons, lasting tens of milliseconds, influences behavior several seconds later, is unclear. Here, we show that NE acts directly on cortical astrocytes via Adra1a adrenergic receptors to elicit sustained increases in intracellular calcium. Chemogenetic blockade of astrocytic calcium elevation prevents the improvement in behavioral performance. NE-activated calcium invokes purinergic pathways in cortical astrocytes that signal to neurons; pathway-specific astrocyte gene expression is altered in mice trained on the task, and blocking neuronal adenosine-sensitive A1 receptors also prevents post-reinforcement behavioral gain. Finally, blocking either astrocyte calcium dynamics or A1 receptors alters encoding of the task in prefrontal cortex neurons, preventing the post-reinforcement change in discriminability of rewarded and unrewarded stimuli underlying behavioral improvement. Together, these data demonstrate that astrocytes, rather than indirectly reflecting neuronal drive, play a direct, instrumental role in representing task-relevant information and signaling to neurons to mediate a fundamental component of learning in the brain.

The neuroscience of mental illness: Building toward the future

Mental illnesses arise from dysfunction in the brain. Although numerous extraneural factors influence these illnesses, ultimately, it is the science of the brain that will lead to novel therapies. Meanwhile, our understanding of this complex organ is incomplete, leading to the oft-repeated trope that neuroscience has yet to make significant contributions to the care of individuals with mental illnesses. This review seeks to counter this narrative, using specific examples of how neuroscientific advances have contributed to progress in mental health care in the past and how current achievements set the stage for further progress in the future.

On the effects of impulsivity and compulsivity on neural correlates of model-based performance

Impaired goal-directed behavior is associated with a range of mental disorders, implicating underlying transdiagnostic factors. While compulsivity has been linked to reduced model-based (MB) control, impulsivity has rarely been studied in the context of reinforcement learning despite its links to reward processing and cognitive control. This study investigated the neural mechanisms underlying MB control and the influence of impulsivity and compulsivity, using EEG data from 238 individuals during a two-step decision making task. Single-trial analyses revealed a modulation of the feedback-related negativity (FRN), where amplitudes were higher after common transitions and positive reward prediction error (RPE), indicating a valence effect. Meanwhile, enhanced P3 amplitudes after rare transitions and both positive and negative RPE possibly reflect surprise. In a second step, we regressed the mean b values of the effect of RPE on the EEG signals onto self-reported impulsivity and compulsivity and behavioral MB control (w). The effect of RPE on FRN-related activity was mainly associated with higher w scores, linking the FRN to MB control. Crucially, the modulation of the P3 by RPE was negatively associated with compulsivity, pointing to a deficient mental model in highly compulsive individuals.

Learning depends on the information conveyed by temporal relationships between events and is reflected in the dopamine response to cues

Contemporary theories guiding the search for neural mechanisms of learning and memory assume that associative learning results from the temporal pairing of cues and reinforcers resulting in coincident activation of associated neurons, strengthening their synaptic connection. While enduring, this framework has limitations: Temporal pairing-based models of learning do not fit with many experimental observations and cannot be used to make quantitative predictions about behavior. Here, we present behavioral data that support an alternative, information-theoretic conception: The amount of information that cues provide about the timing of reward delivery predicts behavior. Furthermore, this approach accounts for the rate and depth of both inhibitory and excitatory learning across paradigms and species. We also show that dopamine release in the ventral striatum reflects cue-predicted changes in reinforcement rates consistent with subjects understanding temporal relationships between task events. Our results reshape the conceptual and biological framework for understanding associative learning.

In Search for a Pathogenesis of Major Depression and Suicide-A Joint Investigation of Dopamine and Fiber Tract Anatomy Focusing on the Human Ventral Mesencephalic Tegmentum: Description of a Workflow

Major depressive disorder (MDD) is prevalent with a high subjective and socio-economic burden. Despite the effectiveness of classical treatment methods, 20-30% of patients stay treatment-resistant. Deep Brain Stimulation of the superolateral branch of the medial forebrain bundle is emerging as a clinical treatment. The stimulation region (ventral tegmental area, VTA), supported by experimental data, points to the role of dopaminergic (DA) transmission in disease pathology. This work sets out to develop a workflow that will allow the performance of analyses on midbrain DA-ergic neurons and projections in subjects who have committed suicide. Human midbrains were retrieved during autopsy, formalin-fixed, and scanned in a Bruker MRI scanner (7T). Sections were sliced, stained for tyrosine hydroxylase (TH), digitized, and integrated into the Montreal Neurological Institute (MNI) brain space together with a high-resolution fiber tract atlas. Subnuclei of the VTA region were identified. TH-positive neurons and fibers were semi-quantitatively evaluated. The study established a rigorous protocol allowing for parallel histological assessments and fiber tractographic analysis in a common space. Semi-quantitative readings are feasible and allow the detection of cell loss in VTA subnuclei. This work describes the intricate workflow and first results of an investigation of DA anatomy in VTA subnuclei in a growing naturalistic database.

Optimal Dynamic Regimes for CO Oxidation Discovered by Reinforcement Learning

Metal nanoparticles are widely used as heterogeneous catalysts to activate adsorbed molecules and reduce the energy barrier of the reaction. Reaction product yield depends on the interplay between elementary processes: adsorption, activation, desorption, and reaction. These processes, in turn, depend on the inlet gas composition, temperature, and pressure. At a steady state, the active surface sites may be inaccessible due to adsorbed reagents. Periodic regime may thus improve the yield, but the appropriate period and waveform are not known in advance. Dynamic control should account for surface and atmospheric modifications and adjust reaction parameters according to the current state of the system and its history. In this work, we applied a reinforcement learning algorithm to control CO oxidation on a palladium catalyst. The policy gradient algorithm was trained in the theoretical environment, parametrized from experimental data. The algorithm learned to maximize the CO2 formation rate based on CO and O2 partial pressures for several successive time steps. Within a unified approach, we found optimal stationary, periodic, and nonperiodic regimes for different problem formulations and gained insight into why the dynamic regime can be preferential. In general, this work contributes to the task of popularizing the reinforcement learning approach in the field of catalytic science.

In praise of folly: flexible goals and human cognition

Humans often pursue idiosyncratic goals that appear remote from functional ends, including information gain. We suggest that this is valuable because goals (even prima facie foolish or unachievable ones) contain structured information that scaffolds thinking and planning. By evaluating hypotheses and plans with respect to their goals, humans can discover new ideas that go beyond prior knowledge and observable evidence. These hypotheses and plans can be transmitted independently of their original motivations, adapted across generations, and serve as an engine of cultural evolution. Here, we review recent empirical and computational research underlying goal generation and planning and discuss the ways that the flexibility of our motivational system supports cognitive gains for both individuals and societies.

Contributions of Basal Ganglia Circuits to Perception, Attention, and Consciousness

Research into ascending sensory pathways and cortical networks has generated detailed models of perception. These same cortical regions are strongly connected to subcortical structures, such as the basal ganglia (BG), which have been conceptualized as playing key roles in reinforcement learning and action selection. However, because the BG amasses experiential evidence from higher and lower levels of cortical hierarchies, as well as higher-order thalamus, it is well positioned to dynamically influence perception. Here, we review anatomical, functional, and clinical evidence to demonstrate how the BG can influence perceptual processing and conscious states. This depends on the integrative relationship between cortex, BG, and thalamus, which allows contributions to sensory gating, predictive processing, selective attention, and representation of the temporal structure of events.

Surprise-minimization as a solution to the structural credit assignment problem

The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration.

A scoping review of methodologies for applying artificial intelligence to physical activity interventions

PURPOSE

This scoping review aimed to offer researchers and practitioners an understanding of artificial intelligence (AI) applications in physical activity (PA) interventions; introduce them to prevalent machine learning (ML), deep learning (DL), and reinforcement learning (RL) algorithms; and encourage the adoption of AI methodologies.

METHODS

A scoping review was performed in PubMed, Web of Science, Cochrane Library, and EBSCO focusing on AI applications for promoting PA or predicting related behavioral or health outcomes. AI methodologies were summarized and categorized to identify synergies, patterns, and trends informing future research. Additionally, a concise primer on predominant AI methodologies within the realm of PA was provided to bolster understanding and broader application.

RESULTS

The review included 24 studies that met the predetermined eligibility criteria. AI models were found effective in detecting significant patterns of PA behavior and associations between specific factors and intervention outcomes. Most studies comparing AI models to traditional statistical approaches reported higher prediction accuracy for AI models on test data. Comparisons of different AI models yielded mixed results, likely due to model performance being highly dependent on the dataset and task. An increasing trend of adopting state-of-the-art DL and RL models over standard ML was observed, addressing complex human-machine communication, behavior modification, and decision-making tasks. Six key areas for future AI adoption in PA interventions emerged: personalized PA interventions, real-time monitoring and adaptation, integration of multimodal data sources, evaluation of intervention effectiveness, expanding access to PA interventions, and predicting and preventing injuries.

CONCLUSION

The scoping review highlights the potential of AI methodologies for advancing PA interventions. As the field progresses, staying informed and exploring emerging AI-driven strategies is essential for achieving significant improvements in PA interventions and fostering overall well-being.

Bayesian Reinforcement Learning With Limited Cognitive Load

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Neurophysiological and Autonomic Correlates of Metacognitive Control of and Resistance to Distractors in Ecological Setting: A Pilot Study

In organisational contexts, professionals are required to decide dynamically and prioritise unexpected external inputs deriving from multiple sources. In the present study, we applied a multimethodological neuroscientific approach to investigate the ability to resist and control ecological distractors during decision-making and to explore whether a specific behavioural, neurophysiological (i.e., delta, theta, alpha and beta EEG band), or autonomic (i.e., heart rate-HR, and skin conductance response-SCR) pattern is correlated with specific personality profiles, collected with the 10-item Big Five Inventory. Twenty-four participants performed a novel Resistance to Ecological Distractors (RED) task aimed at exploring the ability to resist and control distractors and the level of coherence and awareness of behaviour (metacognition ability), while neurophysiological and autonomic measures were collected. The behavioural results highlighted that effectiveness in performance did not require self-control and metacognition behaviour and that being proficient in metacognition can have an impact on performance. Moreover, it was shown that the ability to resist ecological distractors is related to a specific autonomic profile (HR and SCR decrease) and that the neurophysiological and autonomic activations during task execution correlate with specific personality profiles. The agreeableness profile was negatively correlated with the EEG theta band and positively with the EEG beta band, the conscientiousness profile was negatively correlated with the EEG alpha band, and the extroversion profile was positively correlated with the EEG beta band. Taken together, these findings describe and disentangle the hidden relationship that lies beneath individuals' decision to inhibit or activate intentionally a specific behaviour, such as responding, or not, to an external stimulus, in ecological conditions.

The successor representation subserves hierarchical abstraction for goal-directed behavior

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making

Computational cognitive modeling is an important tool for understanding the processes supporting human and animal decision-making. Choice data in decision-making tasks are inherently noisy, and separating noise from signal can improve the quality of computational modeling. Common approaches to model decision noise often assume constant levels of noise or exploration throughout learning (e.g., the ϵ -softmax policy). However, this assumption is not guaranteed to hold - for example, a subject might disengage and lapse into an inattentive phase for a series of trials in the middle of otherwise low-noise performance. Here, we introduce a new, computationally inexpensive method to dynamically infer the levels of noise in choice behavior, under a model assumption that agents can transition between two discrete latent states (e.g., fully engaged and random). Using simulations, we show that modeling noise levels dynamically instead of statically can substantially improve model fit and parameter estimation, especially in the presence of long periods of noisy behavior, such as prolonged attentional lapses. We further demonstrate the empirical benefits of dynamic noise estimation at the individual and group levels by validating it on four published datasets featuring diverse populations, tasks, and models. Based on the theoretical and empirical evaluation of the method reported in the current work, we expect that dynamic noise estimation will improve modeling in many decision-making paradigms over the static noise estimation method currently used in the modeling literature, while keeping additional model complexity and assumptions minimal.

Apathy and Motivation: Biological Basis and Drug Treatment

Apathy is a disabling syndrome associated with poor functional outcomes that is common across a broad range of neurological and psychiatric conditions. Currently, there are no established therapies specifically for the condition, and safe and effective treatments are urgently needed. Advances in the understanding of motivation and goal-directed behavior in humans and animals have shed light on the cognitive and neurobiological mechanisms contributing to apathy, providing an important foundation for the development of new treatments. Here, we review the cognitive components, neural circuitry, and pharmacology of apathy and motivation, highlighting converging evidence of shared transdiagnostic mechanisms. Though no pharmacological treatments have yet been licensed, we summarize trials of existing and novel compounds to date, identifying several promising candidates for clinical use and avenues of future drug development.

Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder

Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.

Informational feedback accelerates learning in multi-alternative perceptual judgements of orientation

Experience or training can substantially improve perceptual performance through perceptual learning, and the extent and rate of these improvements may be affected by feedback. In this paper, we first developed a neural network model based on the integrated reweighting theory (Dosher et al., 2013) to account for perceptual learning and performance in n-alternative identification tasks and the dependence of learning on different forms of feedback. We then report an experiment comparing the effectiveness of response feedback (RF) versus accuracy feedback (AF) or no feedback (NF) (full versus partial versus no supervision) in learning a challenging eight-alternative visual orientation identification (8AFC) task. Although learning sometimes occurred in the absence of feedback (NF), RF had a clear advantage above AF or NF in this task. Using hybrid supervision learning rules, a new n-alternative identification integrated reweighting theory (I-IRT) explained both the differences in learning curves given different feedback and the dynamic changes in identification confusion data. This study shows that training with more informational feedback (RF) is more effective, though not necessary, in these challenging n-alternative tasks, a result that has implications for developing training paradigms in realistic tasks.

Fruit bats adjust their decision-making process according to environmental dynamics

One of the main functions of behavioral plasticity lies in the ability to contend with dynamic environments. Indeed, while numerous studies have shown that animals adapt their behavior to the environment, how they adapt their latent learning and decision strategies to changes in the environment is less understood. Here, we used a controlled experiment to examine the bats' ability to adjust their decision strategy according to the environmental dynamics. Twenty-five Egyptian fruit bats were placed individually in either a stable or a volatile environment for four consecutive nights. In the stable environment, two feeders offered food, each with a different reward probability (0.2 vs. 0.8) that remained fixed over two nights and were then switched, while in the volatile environment, the positions of the more and the less rewarding feeders were changed every hour. We then fit two alternative commonly used models namely, reinforcement learning and win-stay-lose-shift strategies to the bats' behavior. We found that while the bats adapted their decision-making strategy to the environmental dynamics, they seemed to be limited in their responses based on natural priors. Namely, when the environment had changed slowly, at a rate that is natural for these bats, they seemed to rely on reinforcement learning and their performance was nearly optimal, but when the experimental environment changed much faster than in the natural environment, the bats stopped learning and switched to a random decision-making strategy. Together, these findings exemplify both the bats' decision-making plasticity as well as its natural limitations.

Listening to the Data: Computational Approaches to Addiction and Learning

Computational approaches hold great promise for identifying novel treatment targets and creating translational therapeutics for substance use disorders. From circuitries underlying decision-making to computationally derived neural markers of drug-cue reactivity, this review is a summary of the approaches to data presented at our 2023 Society for Neuroscience Mini-Symposium. Here, we highlight data- and hypothesis-driven computational approaches that recently afforded advancements in addiction and learning neuroscience. First, we discuss the value of hypothesis-driven algorithmic modeling approaches, which integrate behavioral, neural, and cognitive outputs to refine hypothesis testing. Then, we review the advantages of data-driven dimensionality reduction and machine learning methods for uncovering novel predictor variables and elucidating relationships in high-dimensional data. Overall, this review highlights recent breakthroughs in cognitive mapping, model-based analysis of behavior/risky decision-making, patterns of drug taking, relapse, and neuromarker discovery, and showcases the benefits of novel modeling techniques, across both preclinical and clinical data.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Rational Simplification and Rigidity in Human Planning

Planning underpins the impressive flexibility of goal-directed behavior. However, even when planning, people can display surprising rigidity in how they think about problems (e.g., "functional fixedness") that lead them astray. How can our capacity for behavioral flexibility be reconciled with our susceptibility to conceptual inflexibility? We propose that these tendencies reflect avoidance of two cognitive costs: the cost of representing task details and the cost of switching between representations. To test this hypothesis, we developed a novel paradigm that affords participants opportunities to choose different families of simplified representations to plan. In two preregistered, online studies (Ns = 377 and 294 adults), we found that participants' optimal behavior, suboptimal behavior, and reaction time were explained by a computational model that formalized people's avoidance of representational complexity and switching. These results demonstrate how the selection of simplified, rigid representations leads to the otherwise puzzling combination of flexibility and inflexibility observed in problem solving.

The emergence of division of labour through decentralized social sanctioning

Human ecological success relies on our characteristic ability to flexibly self-organize into cooperative social groups, the most successful of which employ substantial specialization and division of labour. Unlike most other animals, humans learn by trial and error during their lives what role to take on. However, when some critical roles are more attractive than others, and individuals are self-interested, then there is a social dilemma: each individual would prefer others take on the critical but unremunerative roles so they may remain free to take one that pays better. But disaster occurs if all act thus and a critical role goes unfilled. In such situations learning an optimum role distribution may not be possible. Consequently, a fundamental question is: how can division of labour emerge in groups of self-interested lifetime-learning individuals? Here, we show that by introducing a model of social norms, which we regard as emergent patterns of decentralized social sanctioning, it becomes possible for groups of self-interested individuals to learn a productive division of labour involving all critical roles. Such social norms work by redistributing rewards within the population to disincentivize antisocial roles while incentivizing prosocial roles that do not intrinsically pay as well as others.

The interactive effect of external rewards and self-determined choice on memory

Both external motivational incentives (e.g., monetary reward) and internal motivational incentives (e.g., self-determined choice) have been found to promote memory, but much less is known about how these two types of incentives interact with each other to affect memory. The current study (N = 108) examined how performance-dependent monetary rewards affected the role of self-determined choice in memory performance, also known as the choice effect. Using a modified and better controlled version of the choice paradigm and manipulating levels of reward, we demonstrated an interactive effect between monetary reward and self-determined choice on 1-day delayed memory performance. Specifically, the choice effect on memory decreased when we introduced the performance-dependent external rewards. These results are discussed in terms of understanding how external and internal motivators interact to impact learning and memory.

The indirect pathway of the basal ganglia promotes transient punishment but not motor suppression

Optogenetic stimulation of Adora2a receptor-expressing spiny projection neurons (A2A-SPNs) in the striatum drives locomotor suppression and transient punishment, results attributed to activation of the indirect pathway. The sole long-range projection target of A2A-SPNs is the external globus pallidus (GPe). Unexpectedly, we found that inhibition of the GPe drove transient punishment but not suppression of movement. Within the striatum, A2A-SPNs inhibit other SPNs through a short-range inhibitory collateral network, and we found that optogenetic stimuli that drove motor suppression shared a common mechanism of recruiting this inhibitory collateral network. Our results suggest that the indirect pathway plays a more prominent role in transient punishment than in motor control and challenges the assumption that activity of A2A-SPNs is synonymous with indirect pathway activity.

Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance

Humans and animals develop learning-to-learn strategies throughout their lives to accelerate learning. One theory suggests that this is achieved by a metacognitive process of controlling and monitoring learning. Although such learning-to-learn is also observed in motor learning, the metacognitive aspect of learning regulation has not been considered in classical theories of motor learning. Here, we formulated a minimal mechanism of this process as reinforcement learning of motor learning properties, which regulates a policy for memory update in response to sensory prediction error while monitoring its performance. This theory was confirmed in human motor learning experiments, in which the subjective sense of learning-outcome association determined the direction of up- and down-regulation of both learning speed and memory retention. Thus, it provides a simple, unifying account for variations in learning speeds, where the reinforcement learning mechanism monitors and controls the motor learning process.

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches.

A neuro-computational social learning framework to facilitate transdiagnostic classification and treatment across psychiatric disorders

Social deficits are among the core and most striking psychiatric symptoms, present in most psychiatric disorders. Here, we introduce a novel social learning framework, which consists of neuro-computational models that combine reinforcement learning with various types of social knowledge structures. We outline how this social learning framework can help specify and quantify social psychopathology across disorders and provide an overview of the brain regions that may be involved in this type of social learning. We highlight how this framework can specify commonalities and differences in the social psychopathology of individuals with autism spectrum disorder (ASD), personality disorders (PD), and major depressive disorder (MDD) and improve treatments on an individual basis. We conjecture that individuals with psychiatric disorders rely on rigid social knowledge representations when learning about others, albeit the nature of their rigidity and the behavioral consequences can greatly differ. While non-clinical cohorts tend to efficiently adapt social knowledge representations to relevant environmental constraints, psychiatric cohorts may rigidly stick to their preconceived notions or overly coarse knowledge representations during learning.

Computational models of episodic-like memory in food-caching birds

Birds of the crow family adapt food-caching strategies to anticipated needs at the time of cache recovery and rely on memory of the what, where and when of previous caching events to recover their hidden food. It is unclear if this behavior can be explained by simple associative learning or if it relies on higher cognitive processes like mental time-travel. We present a computational model and propose a neural implementation of food-caching behavior. The model has hunger variables for motivational control, reward-modulated update of retrieval and caching policies and an associative neural network for remembering caching events with a memory consolidation mechanism for flexible decoding of the age of a memory. Our methodology of formalizing experimental protocols is transferable to other domains and facilitates model evaluation and experiment design. Here, we show that memory-augmented, associative reinforcement learning without mental time-travel is sufficient to explain the results of 28 behavioral experiments with food-caching birds.

Neural correlates underlying preference changes induced by food Go/No-Go training

Consistently not responding to appetitive foods during food go/no-go training could change individuals' food choices and sometimes even body weight, however, fewer studies have explored the neural pathways underlying the effects of food go/no-go training. In this study, we scanned eighty-six female participants using functional magnetic resonance imaging and investigated the neural bases of preference changes in a binary food choice task following action (e.g., go) or inaction (e.g., no-go) toward distinct foods within a food go/no-go training paradigm. In line with prior behavioral work, we found that participants' food preferences changed as a function of food go/no-go training, with participants choosing more "go" over "no-go" foods for consumption following training. At a neural level, preference changes were inversely associated with frontoparietal and salience network activity when choosing go (vs. no-go) foods. Additionally, task-related functional connectivities from the inferior parietal lobule to the pre-supplementary motor cortex, dorsolateral prefrontal cortex, and dorsal anterior cingulate cortex were related to these preference changes. Together, current work supports that food go/no-go training reliably changes people's preferences. More importantly, our findings suggest that a neural pathway centered on areas traditionally associated with selective attention may interface with prefrontal regions to guide preference changes induced by food go/no-go training, though future studies using other tasks (e.g., passive viewing tasks) are still needed to test this potential neural mechanism.

Cortical Grey Matter Mediates Increases in Model-Based Control and Learning from Positive Feedback from Adolescence to Adulthood

Cognition and brain structure undergo significant maturation from adolescence into adulthood. Model-based (MB) control is known to increase across development, which is mediated by cognitive abilities. Here, we asked two questions unaddressed in previous developmental studies. First, what are the brain structural correlates of age-related increases in MB control? Second, how are age-related increases in MB control from adolescence to adulthood influenced by motivational context? A human developmental sample (n = 103; age, 12-50, male/female, 55:48) completed structural MRI and an established task to capture MB control. The task was modified with respect to outcome valence by including (1) reward and punishment blocks to manipulate the motivational context and (2) an additional choice test to assess learning from positive versus negative feedback. After replicating that an age-dependent increase in MB control is mediated by cognitive abilities, we demonstrate first-time evidence that gray matter density (GMD) in the parietal cortex mediates the increase of MB control with age. Although motivational context did not relate to age-related changes in MB control, learning from positive feedback improved with age. Meanwhile, negative feedback learning showed no age effects. We present a first report that an age-related increase in positive feedback learning was mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. Our findings indicate that brain maturation, putatively reflected in lower GMD, in distinct and partially overlapping brain regions could lead to a more efficient brain organization and might thus be a key developmental step toward age-related increases in planning and value-based choice.SIGNIFICANCE STATEMENT Changes in model-based decision-making are paralleled by extensive maturation in cognition and brain structure across development. Still, to date the neuroanatomical underpinnings of these changes remain unclear. Here, we demonstrate for the first time that parietal GMD mediates age-dependent increases in model-based control. Age-related increases in positive feedback learning were mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. A manipulation of motivational context did not have an impact on age-related changes in model-based control. These findings highlight that brain maturation in distinct and overlapping cortical regions constitutes a key developmental step toward improved value-based choices.

Valence-partitioned learning signals drive choice behavior and phenomenal subjective experience in humans

How the human brain generates conscious phenomenal experience is a fundamental problem. In particular, it is unknown how variable and dynamic changes in subjective affect are driven by interactions with objective phenomena. We hypothesize a neurocomputational mechanism that generates valence-specific learning signals associated with 'what it is like' to be rewarded or punished. Our hypothesized model maintains a partition between appetitive and aversive information while generating independent and parallel reward and punishment learning signals. This valence-partitioned reinforcement learning (VPRL) model and its associated learning signals are shown to predict dynamic changes in 1) human choice behavior, 2) phenomenal subjective experience, and 3) BOLD-imaging responses that implicate a network of regions that process appetitive and aversive information that converge on the ventral striatum and ventromedial prefrontal cortex during moments of introspection. Our results demonstrate the utility of valence-partitioned reinforcement learning as a neurocomputational basis for investigating mechanisms that may drive conscious experience.

All the Pringle ladies: Neural and behavioral responses to high-calorie food rewards in young adult women

Reward processing is vital for learning and survival, and can be indexed using the Reward Positivity (RewP), an event-related potential (ERP) component that is larger for rewards than losses. Prior work suggests that heightened motivation to obtain reward, as well as greater reward value, is associated with an enhanced RewP. However, the extent to which internal and external factors modulate neural responses to rewards, and whether such neural responses motivate reward-seeking behavior, remains unclear. The present study investigated whether the degree to which a reward is salient to an individual's current motivational state modulates the RewP, and whether the RewP predicts motivated behaviors, in a sample of 133 women. To elicit the RewP, participants completed a forced-choice food reward guessing task. Data were also collected on food-related behaviors (i.e., type of food chosen, consumption of the food reward) and motivational salience factors (i.e., self-reported hunger, time since last meal, and subjective "liking" of food reward). Results showed that hungrier participants displayed an enhanced RewP compared to less hungry individuals. Further, self-reported snack liking interacted with RewP magnitude to predict behavior, such that when participants reported low levels of snack liking, those with a smaller RewP were more likely to consume their snacks than those with a larger RewP. Our data suggest that food-related motivational state may increase neural sensitivity to food reward in young women, and that neural markers of reward sensitivity might interact with subjective reward liking to predict real-world eating behavior.

Diagnostic Performance Evaluation of Multiparametric Magnetic Resonance Imaging in the Detection of Prostate Cancer with Supervised Machine Learning Methods

Prostate cancer is the second leading cause of cancer-related death in men. Its early and correct diagnosis is of particular importance to controlling and preventing the disease from spreading to other tissues. Artificial intelligence and machine learning have effectively detected and graded several cancers, in particular prostate cancer. The purpose of this review is to show the diagnostic performance (accuracy and area under the curve) of supervised machine learning algorithms in detecting prostate cancer using multiparametric MRI. A comparison was made between the performances of different supervised machine-learning methods. This review study was performed on the recent literature sourced from scientific citation websites such as Google Scholar, PubMed, Scopus, and Web of Science up to the end of January 2023. The findings of this review reveal that supervised machine learning techniques have good performance with high accuracy and area under the curve for prostate cancer diagnosis and prediction using multiparametric MR imaging. Among supervised machine learning methods, deep learning, random forest, and logistic regression algorithms appear to have the best performance.

Digitally embodied lifespan neurocognitive development and Tactile Internet: Transdisciplinary challenges and opportunities

Mechanisms underlying perceptual processing and inference undergo substantial changes across the lifespan. If utilized properly, technologies could support and buffer the relatively more limited neurocognitive functions in the still developing or aging brains. Over the past decade, a new type of digital communication infrastructure, known as the "Tactile Internet (TI)," is emerging in the fields of telecommunication, sensor and actuator technologies and machine learning. A key aim of the TI is to enable humans to experience and interact with remote and virtual environments through digitalized multimodal sensory signals that also include the haptic (tactile and kinesthetic) sense. Besides their applied focus, such technologies may offer new opportunities for the research tapping into mechanisms of digitally embodied perception and cognition as well as how they may differ across age cohorts. However, there are challenges in translating empirical findings and theories about neurocognitive mechanisms of perception and lifespan development into the day-to-day practices of engineering research and technological development. On the one hand, the capacity and efficiency of digital communication are affected by signal transmission noise according to Shannon's (1949) Information Theory. On the other hand, neurotransmitters, which have been postulated as means that regulate the signal-to-noise ratio of neural information processing (e.g., Servan-Schreiber et al., 1990), decline substantially during aging. Thus, here we highlight neuronal gain control of perceptual processing and perceptual inference to illustrate potential interfaces for developing age-adjusted technologies to enable plausible multisensory digital embodiments for perceptual and cognitive interactions in remote or virtual environments.

Reliability of Decision-Making and Reinforcement Learning Computational Parameters

Computational models can offer mechanistic insight into cognition and therefore have the potential to transform our understanding of psychiatric disorders and their treatment. For translational efforts to be successful, it is imperative that computational measures capture individual characteristics reliably. Here we examine the reliability of reinforcement learning and economic models derived from two commonly used tasks. Healthy individuals (N = 50) completed a restless four-armed bandit and a calibrated gambling task twice, two weeks apart. Reward and punishment learning rates from the reinforcement learning model showed good reliability and reward and punishment sensitivity from the same model had fair reliability; while risk aversion and loss aversion parameters from a prospect theory model exhibited good and excellent reliability, respectively. Both models were further able to predict future behaviour above chance within individuals. This prediction was better when based on participants' own model parameters than other participants' parameter estimates. These results suggest that reinforcement learning, and particularly prospect theory parameters, as derived from a restless four-armed bandit and a calibrated gambling task, can be measured reliably to assess learning and decision-making mechanisms. Overall, these findings indicate the translational potential of clinically-relevant computational parameters for precision psychiatry.

Prefrontal transcranial magnetic stimulation boosts response vigour during reinforcement learning in healthy adults

A 10-Hz repetitive transcranial magnetic stimulation to the left dorsal lateral prefrontal cortex has been shown to increase dopaminergic activity in the dorsal striatum, a region strongly implicated in reinforcement learning. However, the behavioural influence of this effect remains largely unknown. We tested the causal effects of 10-Hz stimulation on behavioural and computational characteristics of reinforcement learning. A total of 40 healthy individuals were randomized into active and sham (placebo) stimulation groups. Each participant underwent one stimulation session (1500 pulses) in which stimulation was applied over the left dorsal lateral prefrontal cortex using a robotic arm. Participants then completed a reinforcement learning task sensitive to striatal dopamine functioning. Participants' choices were modelled using a reinforcement learning model (Q-learning) that calculates separate learning rates associated with positive and negative reward prediction errors. Subjects receiving active stimulation exhibited increased reward rate (number of correct responses per second of task activity) compared with those in sham. Computationally, although no group differences were observed, the active group displayed a higher learning rate for correct trials (αG) compared with incorrect trials (αL). Finally, when tested with novel pairs of stimuli, the active group displayed extremely fast reaction times, and a trend towards a higher reward rate. This study provided specific behavioural and computational accounts of altered striatal-mediated behaviour, particularly response vigour, induced by a proposed increase of dopamine activity by 10-Hz stimulation to the left dorsal lateral prefrontal cortex. Together, these findings bolster the use of repetitive transcranial magnetic stimulation to target neurocognitive disturbances attributed to the dysregulation of dopaminergic-striatal circuits.

Long-term memory, synaptic plasticity and dopamine in rodent medial prefrontal cortex: Role in executive functions

Mnemonic functions, supporting rodent behavior in complex tasks, include both long-term and (short-term) working memory components. While working memory is thought to rely on persistent activity states in an active neural network, long-term memory and synaptic plasticity contribute to the formation of the underlying synaptic structure, determining the range of possible states. Whereas, the implication of working memory in executive functions, mediated by the prefrontal cortex (PFC) in primates and rodents, has been extensively studied, the contribution of long-term memory component to these tasks received little attention. This review summarizes available experimental data and theoretical work concerning cellular mechanisms of synaptic plasticity in the medial region of rodent PFC and the link between plasticity, memory and behavior in PFC-dependent tasks. A special attention is devoted to unique properties of dopaminergic modulation of prefrontal synaptic plasticity and its contribution to executive functions.

Bringing Bayes and Shannon to the Study of Behavioural and Neurobiological Timing and Associative Learning

Bayesian parameter estimation and Shannon’s theory of information provide tools for analysing and understanding data from behavioural and neurobiological experiments on interval timing—and from experiments on Pavlovian and operant conditioning, because timing plays a fundamental role in associative learning. In this tutorial, we explain basic concepts behind these tools and show how to apply them to estimating, on a trial-by-trial, reinforcement-by-reinforcement and response-by-response basis, important parameters of timing behaviour and of the neurobiological manifestations of timing in the brain. These tools enable quantification of relevant variables in the trade-off between acting as an ideal observer should act and acting as an ideal agent should act, which is also known as the trade-off between exploration (information gathering) and exploitation (information utilization) in reinforcement learning. They enable comparing the strength of the evidence for a measurable association to the strength of the behavioural evidence that the association has been perceived. A GitHub site and an OSF site give public access to well-documented Matlab and Python code and to raw data to which these tools have been applied.