Active inference and learning

Setting the space for deliberation in decision-making

Decision-making models in the behavioral, cognitive, and neural sciences typically consist of forced-choice paradigms with two alternatives. While theoretically it is feasible to translate any decision situation to a sequence of binary choices, real-life decision-making is typically more complex and nonlinear, involving choices among multiple items, graded judgments, and deferments of decision-making. Here, we discuss how the complexity of real-life decision-making can be addressed using conventional decision-making models by focusing on the interactive dynamics between criteria settings and the collection of evidence. Decision-makers can engage in multi-stage, parallel decision-making by exploiting the space for deliberation, with non-binary readings of evidence available at any point in time. The interactive dynamics principally adhere to the speed-accuracy tradeoff, such that increasing the space for deliberation enables extended data collection. The setting of space for deliberation reflects a form of meta-decision-making that can, and should be, studied empirically as a value-based exercise that weighs the prior propensities, the economics of information seeking, and the potential outcomes. Importantly, the control of the space for deliberation raises a question of agency. Decision-makers may actively and explicitly set their own decision parameters, but these parameters may also be set by environmental pressures. Thus, decision-makers may be influenced-or nudged in a particular direction-by how decision problems are framed, with a sense of urgency or a binary definition of choice options. We argue that a proper understanding of these mechanisms has important practical implications toward the optimal usage of space for deliberation.

Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise

The free energy principle from neuroscience has recently gained traction as one of the most prominent brain theories that can emulate the brain's perception and action in a bio-inspired manner. This renders the theory with the potential to hold the key for general artificial intelligence. Leveraging this potential, this paper aims to bridge the gap between neuroscience and robotics by reformulating an FEP-based inference scheme-Dynamic Expectation Maximization-into an algorithm that can perform simultaneous state, input, parameter, and noise hyperparameter estimation of any stable linear state space system subjected to colored noises. The resulting estimator was proved to be of the form of an augmented coupled linear estimator. Using this mathematical formulation, we proved that the estimation steps have theoretical guarantees of convergence. The algorithm was rigorously tested in simulation on a wide variety of linear systems with colored noises. The paper concludes by demonstrating the superior performance of DEM for parameter estimation under colored noise in simulation, when compared to the state-of-the-art estimators like Sub Space method, Prediction Error Minimization (PEM), and Expectation Maximization (EM) algorithm. These results contribute to the applicability of DEM as a robust learning algorithm for safe robotic applications.

An empirical evaluation of active inference in multi-armed bandits

A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

Gut inference: A computational modelling approach

Neurocomputational theories have hypothesized that Bayesian inference underlies interoception, which has become a topic of recent experimental work in heartbeat perception. To extend this approach beyond cardiac interoception, we describe the application of a Bayesian computational model to a recently developed gastrointestinal interoception task completed by 40 healthy individuals undergoing simultaneous electroencephalogram (EEG) and peripheral physiological recording. We first present results that support the validity of this modelling approach. Second, we provide a test of, and confirmatory evidence supporting, the neural process theory associated with a particular Bayesian framework (active inference) that predicts specific relationships between computational parameters and event-related potentials in EEG. We also offer some exploratory evidence suggesting that computational parameters may influence the regulation of peripheral physiological states. We conclude that this computational approach offers promise as a tool for studying individual differences in gastrointestinal interoception.

Sophisticated Inference

Active inference offers a first principle account of sentient behavior, from which special and important cases-for example, reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design-can be derived. Active inference finesses the exploitation-exploration dilemma in relation to prior preferences by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this letter, we consider a sophisticated kind of active inference using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs and beliefs about those latent states. In other words, we move from simply considering beliefs about "what would happen if I did that" to "what I would believe about what would happen if I did that." The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states as opposed to states per se. We illustrate the competence of this scheme using numerical simulations of deep decision problems.

Emergent Goal-Anticipatory Gaze in Infants via Event-Predictive Learning and Inference

From about 7 months of age onward, infants start to reliably fixate the goal of an observed action, such as a grasp, before the action is complete. The available research has identified a variety of factors that influence such goal-anticipatory gaze shifts, including the experience with the shown action events and familiarity with the observed agents. However, the underlying cognitive processes are still heavily debated. We propose that our minds (i) tend to structure sensorimotor dynamics into probabilistic, generative event-predictive, and event boundary predictive models, and, meanwhile, (ii) choose actions with the objective to minimize predicted uncertainty. We implement this proposition by means of event-predictive learning and active inference. The implemented learning mechanism induces an inductive, event-predictive bias, thus developing schematic encodings of experienced events and event boundaries. The implemented active inference principle chooses actions by aiming at minimizing expected future uncertainty. We train our system on multiple object-manipulation events. As a result, the generation of goal-anticipatory gaze shifts emerges while learning about object manipulations: the model starts fixating the inferred goal already at the start of an observed event after having sampled some experience with possible events and when a familiar agent (i.e., a hand) is involved. Meanwhile, the model keeps reactively tracking an unfamiliar agent (i.e., a mechanical claw) that is performing the same movement. We qualitatively compare these modeling results to behavioral data of infants and conclude that event-predictive learning combined with active inference may be critical for eliciting goal-anticipatory gaze behavior in infants.

Bidirectional interaction between visual and motor generative models using Predictive Coding and Active Inference

In this work, we build upon the Active Inference (AIF) and Predictive Coding (PC) frameworks to propose a neural architecture comprising a generative model for sensory prediction, and a distinct generative model for motor trajectories. We highlight how sequences of sensory predictions can act as rails guiding learning, control and online adaptation of motor trajectories. We furthermore inquire the effects of bidirectional interactions between the motor and the visual modules. The architecture is tested on the control of a simulated robotic arm learning to reproduce handwritten letters.

A Predictive Processing Account of Card Sorting: Fast Proactive and Reactive Frontoparietal Cortical Dynamics during Inference and Learning of Perceptual Categories

For decades, a common assumption in cognitive neuroscience has been that prefrontal executive control is mainly engaged during target detection [Posner, M. I., & Petersen, S. E. The attention system of the human brain. Annual Review of Neuroscience, 13, 25-42, 1990]. More recently, predictive processing theories of frontal function under the Bayesian brain hypothesis emphasize a key role of proactive control for anticipatory action selection (i.e., planning as active inference). Here, we review evidence of fast and widespread EEG and magnetoencephalographic fronto-temporo-parietal cortical activations elicited by feedback cues and target cards in the Wisconsin Card Sorting Test. This evidence is best interpreted when considering negative and positive feedback as predictive cues (i.e., sensory outcomes) for proactively updating beliefs about unknown perceptual categories. Such predictive cues inform posterior beliefs about high-level hidden categories governing subsequent response selection at target onset. Quite remarkably, these new views concur with Don Stuss' early findings concerning two broad classes of P300 cortical responses evoked by feedback cues and target cards in a computerized Wisconsin Card Sorting Test analogue. Stuss' discussion of those P300 responses-in terms of the resolution of uncertainty about response (policy) selection as well as the participants' expectancies for future perceptual or motor activities and their timing-was prescient of current predictive processing and active (Bayesian) inference theories. From these new premises, a domain-general frontoparietal cortical network is rapidly engaged during two temporarily distinct stages of inference and learning of perceptual categories that underwrite goal-directed card sorting behavior, and they each engage prefrontal executive functions in fundamentally distinct ways.

Realizing Active Inference in Variational Message Passing: The Outcome-Blind Certainty Seeker

Active inference is a state-of-the-art framework in neuroscience that offers a unified theory of brain function. It is also proposed as a framework for planning in AI. Unfortunately, the complex mathematics required to create new models can impede application of active inference in neuroscience and AI research. This letter addresses this problem by providing a complete mathematical treatment of the active inference framework in discrete time and state spaces and the derivation of the update equations for any new model. We leverage the theoretical connection between active inference and variational message passing as described by John Winn and Christopher M. Bishop in 2005. Since variational message passing is a well-defined methodology for deriving Bayesian belief update equations, this letter opens the door to advanced generative models for active inference. We show that using a fully factorized variational distribution simplifies the expected free energy, which furnishes priors over policies so that agents seek unambiguous states. Finally, we consider future extensions that support deep tree searches for sequential policy optimization based on structure learning and belief propagation.

An Active Inference Model of Collective Intelligence

Collective intelligence, an emergent phenomenon in which a composite system of multiple interacting agents performs at levels greater than the sum of its parts, has long compelled research efforts in social and behavioral sciences. To date, however, formal models of collective intelligence have lacked a plausible mathematical description of the relationship between local-scale interactions between autonomous sub-system components (individuals) and global-scale behavior of the composite system (the collective). In this paper we use the Active Inference Formulation (AIF), a framework for explaining the behavior of any non-equilibrium steady state system at any scale, to posit a minimal agent-based model that simulates the relationship between local individual-level interaction and collective intelligence. We explore the effects of providing baseline AIF agents (Model 1) with specific cognitive capabilities: Theory of Mind (Model 2), Goal Alignment (Model 3), and Theory of Mind with Goal Alignment (Model 4). These stepwise transitions in sophistication of cognitive ability are motivated by the types of advancements plausibly required for an AIF agent to persist and flourish in an environment populated by other highly autonomous AIF agents, and have also recently been shown to map naturally to canonical steps in human cognitive ability. Illustrative results show that stepwise cognitive transitions increase system performance by providing complementary mechanisms for alignment between agents’ local and global optima. Alignment emerges endogenously from the dynamics of interacting AIF agents themselves, rather than being imposed exogenously by incentives to agents’ behaviors (contra existing computational models of collective intelligence) or top-down priors for collective behavior (contra existing multiscale simulations of AIF). These results shed light on the types of generic information-theoretic patterns conducive to collective intelligence in human and other complex adaptive systems.

Prefrontal Computation as Active Inference

The prefrontal cortex is vital for a range of cognitive processes, including working memory, attention, and decision-making. Notably, its absence impairs the performance of tasks requiring the maintenance of information through a delay period. In this paper, we formulate a rodent task—which requires maintenance of delay-period activity—as a Markov decision process and treat optimal task performance as an (active) inference problem. We simulate the behavior of a Bayes optimal mouse presented with 1 of 2 cues that instructs the selection of concurrent visual and auditory targets on a trial-by-trial basis. Formulating inference as message passing, we reproduce features of neuronal coupling within and between prefrontal regions engaged by this task. We focus on the micro-circuitry that underwrites delay-period activity and relate it to functional specialization within the prefrontal cortex in primates. Finally, we simulate the electrophysiological correlates of inference and demonstrate the consequences of lesions to each part of our in silico prefrontal cortex. In brief, this formulation suggests that recurrent excitatory connections—which support persistent neuronal activity—encode beliefs about transition probabilities over time. We argue that attentional modulation can be understood as the contextualization of sensory input by these persistent beliefs.

Why Depressed Mood is Adaptive: A Numerical Proof of Principle for an Evolutionary Systems Theory of Depression

We provide a proof of principle for an evolutionary systems theory (EST) of depression. This theory suggests that normative depressive symptoms counter socioenvironmental volatility by increasing interpersonal support via social signalling and that this response depends upon the encoding of uncertainty about social contingencies, which can be targeted by neuromodulatory antidepressants. We simulated agents that committed to a series of decisions in a social two-arm bandit task before and after social adversity, which precipitated depressive symptoms. Responses to social adversity were modelled under various combinations of social support and pharmacotherapy. The normative depressive phenotype responded positively to social support and simulated treatments with antidepressants. Attracting social support and administering antidepressants also alleviated anhedonia and social withdrawal, speaking to improvements in interpersonal relationships. These results support the EST of depression by demonstrating that following adversity, normative depressed mood preserved social inclusion with appropriate interpersonal support or pharmacotherapy.

Statistical Properties of Musical Creativity: Roles of Hierarchy and Uncertainty in Statistical Learning

Creativity is part of human nature and is commonly understood as a phenomenon whereby something original and worthwhile is formed. Owing to this ability, humans can produce innovative information that often facilitates growth in our society. Creativity also contributes to esthetic and artistic productions, such as music and art. However, the mechanism by which creativity emerges in the brain remains debatable. Recently, a growing body of evidence has suggested that statistical learning contributes to creativity. Statistical learning is an innate and implicit function of the human brain and is considered essential for brain development. Through statistical learning, humans can produce and comprehend structured information, such as music. It is thought that creativity is linked to acquired knowledge, but so-called "eureka" moments often occur unexpectedly under subconscious conditions, without the intention to use the acquired knowledge. Given that a creative moment is intrinsically implicit, we postulate that some types of creativity can be linked to implicit statistical knowledge in the brain. This article reviews neural and computational studies on how creativity emerges within the framework of statistical learning in the brain (i.e., statistical creativity). Here, we propose a hierarchical model of statistical learning: statistically chunking into a unit (hereafter and shallow statistical learning) and combining several units (hereafter and deep statistical learning). We suggest that deep statistical learning contributes dominantly to statistical creativity in music. Furthermore, the temporal dynamics of perceptual uncertainty can be another potential causal factor in statistical creativity. Considering that statistical learning is fundamental to brain development, we also discuss how typical versus atypical brain development modulates hierarchical statistical learning and statistical creativity. We believe that this review will shed light on the key roles of statistical learning in musical creativity and facilitate further investigation of how creativity emerges in the brain.

Meta-analytic evidence of differential prefrontal and early sensory cortex activity during non-social sensory perception in autism

To date, neuroimaging research has had a limited focus on non-social features of autism. As a result, neurobiological explanations for atypical sensory perception in autism are lacking. To address this, we quantitively condensed findings from the non-social autism fMRI literature in line with the current best practices for neuroimaging meta-analyses. Using activation likelihood estimation (ALE), we conducted a series of robust meta-analyses across 83 experiments from 52 fMRI studies investigating differences between autistic (n = 891) and typical (n = 967) participants. We found that typical controls, compared to autistic people, show greater activity in the prefrontal cortex (BA9, BA10) during perception tasks. More refined analyses revealed that, when compared to typical controls, autistic people show greater recruitment of the extrastriate V2 cortex (BA18) during visual processing. Taken together, these findings contribute to our understanding of current theories of autistic perception, and highlight some of the challenges of cognitive neuroscience research in autism.

Fictionalism of Anticipation

A promising recent approach for understanding complex phenomena is recognition of anticipatory behavior of living organisms and social organizations. The anticipatory, predictive action permits learning, novelty seeking, rich experiential existence. I argue that the established frameworks of anticipation, adaptation or learning imply overly passive roles of anticipatory agents, and that a fictionalist standpoint reflects the core of anticipatory behavior better than representational or future references. Cognizing beings enact not just their models of the world, but own make-believe existential agendas as well. Anticipators embody plausible scripts of living, and effectively assume neo-Kantian or pragmatist perspectives of cognition and action. It is instructive to see that anticipatory behavior is not without mundane or loathsome deficiencies. Appreciation of ferally fictionalist anticipation suggests an equivalence of semiosis and anticipation.

Simulating lesion-dependent functional recovery mechanisms

Functional recovery after brain damage varies widely and depends on many factors, including lesion site and extent. When a neuronal system is damaged, recovery may occur by engaging residual (e.g., perilesional) components. When damage is extensive, recovery depends on the availability of other intact neural structures that can reproduce the same functional output (i.e., degeneracy). A system's response to damage may occur rapidly, require learning or both. Here, we simulate functional recovery from four different types of lesions, using a generative model of word repetition that comprised a default premorbid system and a less used alternative system. The synthetic lesions (i) completely disengaged the premorbid system, leaving the alternative system intact, (ii) partially damaged both premorbid and alternative systems, and (iii) limited the experience-dependent plasticity of both. The results, across 1000 trials, demonstrate that (i) a complete disconnection of the premorbid system naturally invoked the engagement of the other, (ii) incomplete damage to both systems had a much more devastating long-term effect on model performance and (iii) the effect of reducing learning capacity within each system. These findings contribute to formal frameworks for interpreting the effect of different types of lesions.

Acoustics versus linguistics? Context is Part and Parcel to lateralized processing of the parts and parcels of speech

The purpose of this review is to provide an accessible exploration of key considerations of lateralization in speech and non-speech perception using clear and defined language. From these considerations, the primary arguments for each side of the linguistics versus acoustics debate are outlined and explored in context of emerging integrative theories. This theoretical approach entails a perspective that linguistic and acoustic features differentially contribute to leftward bias, depending on the given context. Such contextual factors include stimulus parameters and variables of stimulus presentation (e.g., noise/silence and monaural/binaural) and variances in individuals (sex, handedness, age, and behavioural ability). Discussion of these factors and their interaction is also aimed towards providing an outline of variables that require consideration when developing and reviewing methodology of acoustic and linguistic processing laterality studies. Thus, there are three primary aims in the present paper: (1) to provide the reader with key theoretical perspectives from the acoustics/linguistics debate and a synthesis of the two viewpoints, (2) to highlight key caveats for generalizing findings regarding predominant models of speech laterality, and (3) to provide a practical guide for methodological control using predominant behavioural measures (i.e., gap detection and dichotic listening tasks) and/or neurophysiological measures (i.e., mismatch negativity) of speech laterality.

Active Vision for Robot Manipulators Using the Free Energy Principle

Occlusions, restricted field of view and limited resolution all constrain a robot's ability to sense its environment from a single observation. In these cases, the robot first needs to actively query multiple observations and accumulate information before it can complete a task. In this paper, we cast this problem of active vision as active inference, which states that an intelligent agent maintains a generative model of its environment and acts in order to minimize its surprise, or expected free energy according to this model. We apply this to an object-reaching task for a 7-DOF robotic manipulator with an in-hand camera to scan the workspace. A novel generative model using deep neural networks is proposed that is able to fuse multiple views into an abstract representation and is trained from data by minimizing variational free energy. We validate our approach experimentally for a reaching task in simulation in which a robotic agent starts without any knowledge about its workspace. Each step, the next view pose is chosen by evaluating the expected free energy. We find that by minimizing the expected free energy, exploratory behavior emerges when the target object to reach is not in view, and the end effector is moved to the correct reach position once the target is located. Similar to an owl scavenging for prey, the robot naturally prefers higher ground for exploring, approaching its target once located.

The Thalamus as a Blackboard for Perception and Planning

It has been suggested that the thalamus acts as a blackboard, on which the computations of different cortical modules are composed, coordinated, and integrated. This article asks what blackboard role the thalamus might play, and whether that role is consistent with the neuroanatomy of the thalamus. It does so in a context of Bayesian belief updating, expressed as a Free Energy Principle. We suggest that the thalamus-as-a-blackboard offers important questions for research in spatial cognition. Several prominent features of the thalamus-including its lack of olfactory relay function, its lack of internal excitatory connections, its regular and conserved shape, its inhibitory interneurons, triadic synapses, and diffuse cortical connectivity-are consistent with a blackboard role.Different thalamic nuclei may play different blackboard roles: (1) the Pulvinar, through its reciprocal connections to posterior cortical regions, coordinates perceptual inference about "what is where" from multi-sense-data. (2) The Mediodorsal (MD) nucleus, through its connections to the prefrontal cortex, and the other thalamic nuclei linked to the motor cortex, uses the same generative model for planning and learning novel spatial movements. (3) The paraventricular nucleus may compute risk-reward trade-offs. We also propose that as any new movement is practiced a few times, cortico-thalamocortical (CTC) links entrain the corresponding cortico-cortical links, through a process akin to supervised learning. Subsequently, the movement becomes a fast unconscious habit, not requiring the MD nucleus or other thalamic nuclei, and bypassing the thalamic bottleneck.

Single-trial regression of spatial exploration behavior indicates posterior EEG alpha modulation to reflect egocentric coding

Learning to navigate uncharted terrain is a key cognitive ability that emerges as a deeply embodied process, with eye movements and locomotion proving most useful to sample the environment. We studied healthy human participants during active spatial learning of room-scale virtual reality (VR) mazes. In the invisible maze task, participants wearing a wireless electroencephalography (EEG) headset were free to explore their surroundings, only given the objective to build and foster a mental spatial representation of their environment. Spatial uncertainty was resolved by touching otherwise invisible walls that were briefly rendered visible inside VR, similar to finding your way in the dark. We showcase the capabilities of mobile brain/body imaging using VR, demonstrating several analysis approaches based on general linear models (GLMs) to reveal behavior-dependent brain dynamics. Confirming spatial learning via drawn sketch maps, we employed motion capture to image spatial exploration behavior describing a shift from initial exploration to subsequent exploitation of the mental representation. Using independent component analysis, the current work specifically targeted oscillations in response to wall touches reflecting isolated spatial learning events arising in deep posterior EEG sources located in the retrosplenial complex. Single-trial regression identified significant modulation of alpha oscillations by the immediate, egocentric, exploration behavior. When encountering novel walls, as well as with increasing walking distance between subsequent touches when encountering novel walls, alpha power decreased. We conclude that these oscillations play a prominent role during egocentric evidencing of allocentric spatial hypotheses.

Active Inference: Applicability to Different Types of Social Organization Explained through Reference to Industrial Engineering and Quality Management

Active inference is a physics of life process theory of perception, action and learning that is applicable to natural and artificial agents. In this paper, active inference theory is related to different types of practice in social organization. Here, the term social organization is used to clarify that this paper does not encompass organization in biological systems. Rather, the paper addresses active inference in social organization that utilizes industrial engineering, quality management, and artificial intelligence alongside human intelligence. Social organization referred to in this paper can be in private companies, public institutions, other for-profit or not-for-profit organizations, and any combination of them. The relevance of active inference theory is explained in terms of variational free energy, prediction errors, generative models, and Markov blankets. Active inference theory is most relevant to the social organization of work that is highly repetitive. By contrast, there are more challenges involved in applying active inference theory for social organization of less repetitive endeavors such as one-of-a-kind projects. These challenges need to be addressed in order for active inference to provide a unifying framework for different types of social organization employing human and artificial intelligence.

Deeply Felt Affect: The Emergence of Valence in Deep Active Inference

The positive-negative axis of emotional valence has long been recognized as fundamental to adaptive behavior, but its origin and underlying function have largely eluded formal theorizing and computational modeling. Using deep active inference, a hierarchical inference scheme that rests on inverting a model of how sensory data are generated, we develop a principled Bayesian model of emotional valence. This formulation asserts that agents infer their valence state based on the expected precision of their action model-an internal estimate of overall model fitness ("subjective fitness"). This index of subjective fitness can be estimated within any environment and exploits the domain generality of second-order beliefs (beliefs about beliefs). We show how maintaining internal valence representations allows the ensuing affective agent to optimize confidence in action selection preemptively. Valence representations can in turn be optimized by leveraging the (Bayes-optimal) updating term for subjective fitness, which we label affective charge (AC). AC tracks changes in fitness estimates and lends a sign to otherwise unsigned divergences between predictions and outcomes. We simulate the resulting affective inference by subjecting an in silico affective agent to a T-maze paradigm requiring context learning, followed by context reversal. This formulation of affective inference offers a principled account of the link between affect, (mental) action, and implicit metacognition. It characterizes how a deep biological system can infer its affective state and reduce uncertainty about such inferences through internal action (i.e., top-down modulation of priors that underwrite confidence). Thus, we demonstrate the potential of active inference to provide a formal and computationally tractable account of affect. Our demonstration of the face validity and potential utility of this formulation represents the first step within a larger research program. Next, this model can be leveraged to test the hypothesized role of valence by fitting the model to behavioral and neuronal responses.

Representation Wars: Enacting an Armistice Through Active Inference

Over the last 30 years, representationalist and dynamicist positions in the philosophy of cognitive science have argued over whether neurocognitive processes should be viewed as representational or not. Major scientific and technological developments over the years have furnished both parties with ever more sophisticated conceptual weaponry. In recent years, an enactive generalization of predictive processing – known as active inference – has been proposed as a unifying theory of brain functions. Since then, active inference has fueled both representationalist and dynamicist campaigns. However, we believe that when diving into the formal details of active inference, one should be able to find a solution to the war; if not a peace treaty, surely an armistice of a sort. Based on an analysis of these formal details, this paper shows how both representationalist and dynamicist sensibilities can peacefully coexist within the new territory of active inference.

Whence the Expected Free Energy?

The expected free energy (EFE) is a central quantity in the theory of active inference. It is the quantity that all active inference agents are mandated to minimize through action, and its decomposition into extrinsic and intrinsic value terms is key to the balance of exploration and exploitation that active inference agents evince. Despite its importance, the mathematical origins of this quantity and its relation to the variational free energy (VFE) remain unclear. In this letter, we investigate the origins of the EFE in detail and show that it is not simply "the free energy in the future." We present a functional that we argue is the natural extension of the VFE but actively discourages exploratory behavior, thus demonstrating that exploration does not directly follow from free energy minimization into the future. We then develop a novel objective, the free energy of the expected future (FEEF), which possesses both the epistemic component of the EFE and an intuitive mathematical grounding as the divergence between predicted and desired futures.

Active Inference: Demystified and Compared

Active inference is a first principle account of how autonomous agents operate in dynamic, nonstationary environments. This problem is also considered in reinforcement learning, but limited work exists on comparing the two approaches on the same discrete-state environments. In this letter, we provide (1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in reinforcement learning, and (2) an explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration-and account for uncertainty about their environment-in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in reinforcement learning is removed in active inference, where reward can simply be treated as another observation we have a preference over; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based reinforcement learning agents and by placing zero prior preferences over rewards and learning the prior preferences over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings (e.g., robotic arm movement, Atari games) if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Recent advances in the application of predictive coding and active inference models within clinical neuroscience

Research in clinical neuroscience is founded on the idea that a better understanding of brain (dys)function will improve our ability to diagnose and treat neurological and psychiatric disorders. In recent years, neuroscience has converged on the notion that the brain is a 'prediction machine,' in that it actively predicts the sensory input that it will receive if one or another course of action is chosen. These predictions are used to select actions that will (most often, and in the long run) maintain the body within the narrow range of physiological states consistent with survival. This insight has given rise to an area of clinical computational neuroscience research that focuses on characterizing neural circuit architectures that can accomplish these predictive functions, and on how the associated processes may break down or become aberrant within clinical conditions. Here, we provide a brief review of examples of recent work on the application of predictive processing models of brain function to study clinical (psychiatric) disorders, with the aim of highlighting current directions and their potential clinical utility. We offer examples of recent conceptual models, formal mathematical models, and applications of such models in empirical research in clinical populations, with a focus on making this material accessible to clinicians without expertise in computational neuroscience. In doing so, we aim to highlight the potential insights and opportunities that understanding the brain as a prediction machine may offer to clinical research and practice.

A Bayesian computational model reveals a failure to adapt interoceptive precision estimates across depression, anxiety, eating, and substance use disorders

Recent neurocomputational theories have hypothesized that abnormalities in prior beliefs and/or the precision-weighting of afferent interoceptive signals may facilitate the transdiagnostic emergence of psychopathology. Specifically, it has been suggested that, in certain psychiatric disorders, interoceptive processing mechanisms either over-weight prior beliefs or under-weight signals from the viscera (or both), leading to a failure to accurately update beliefs about the body. However, this has not been directly tested empirically. To evaluate the potential roles of prior beliefs and interoceptive precision in this context, we fit a Bayesian computational model to behavior in a transdiagnostic patient sample during an interoceptive awareness (heartbeat tapping) task. Modelling revealed that, during an interoceptive perturbation condition (inspiratory breath-holding during heartbeat tapping), healthy individuals (N = 52) assigned greater precision to ascending cardiac signals than individuals with symptoms of anxiety (N = 15), depression (N = 69), co-morbid depression/anxiety (N = 153), substance use disorders (N = 131), and eating disorders (N = 14)-who failed to increase their precision estimates from resting levels. In contrast, we did not find strong evidence for differences in prior beliefs. These results provide the first empirical computational modeling evidence of a selective dysfunction in adaptive interoceptive processing in psychiatric conditions, and lay the groundwork for future studies examining how reduced interoceptive precision influences visceral regulation and interoceptively-guided decision-making.

Parietal-Prefrontal Feedforward Connectivity in Association With Schizophrenia Genetic Risk and Delusions

OBJECTIVE

Conceptualizations of delusion formation implicate deficits in feedforward information updating across the posterior to prefrontal cortices, resulting in dysfunctional integration of new information about contexts in working memory and, ultimately, failure to update overfamiliar prior beliefs. The authors used functional MRI and machine learning models to address individual variability in feedforward parietal-prefrontal information updating in patients with schizophrenia. They examined relationships between feedforward connectivity, and delusional thinking and polygenic risk for schizophrenia.

METHODS

The authors studied 66 schizophrenia patients and 143 healthy control subjects during performance of context updating in working memory. Dynamic causal models of effective connectivity were focused on regions of the prefrontal and parietal cortex potentially implicated in delusion processes. The effect of polygenic risk for schizophrenia on connectivity was examined in healthy individuals. The authors then leveraged support vector regression models to define optimal normalized target connectivity tailored for each patient and tested the extent to which deviation from this target could predict individual variation in severity of delusions.

RESULTS

In schizophrenia patients, updating and manipulating context information was disproportionately less accurate than was working memory maintenance, with an interaction of task accuracy by diagnosis. Patients with delusions also tended to have relatively reduced parietal-prefrontal feedforward effective connectivity during context updating in working memory manipulation. The same connectivity was adversely influenced by polygenic risk for schizophrenia in healthy subjects. Individual patients' deviation from predicted "normal" feedforward connectivity based on the support vector regression models correlated with severity of delusions.

CONCLUSIONS

These computationally derived observations support a role for feedforward parietal-prefrontal information processing deficits in delusional psychopathology and in genetic risk for schizophrenia.

The two kinds of free energy and the Bayesian revolution

The concept of free energy has its origins in 19th century thermodynamics, but has recently found its way into the behavioral and neural sciences, where it has been promoted for its wide applicability and has even been suggested as a fundamental principle of understanding intelligent behavior and brain function. We argue that there are essentially two different notions of free energy in current models of intelligent agency, that can both be considered as applications of Bayesian inference to the problem of action selection: one that appears when trading off accuracy and uncertainty based on a general maximum entropy principle, and one that formulates action selection in terms of minimizing an error measure that quantifies deviations of beliefs and policies from given reference models. The first approach provides a normative rule for action selection in the face of model uncertainty or when information processing capabilities are limited. The second approach directly aims to formulate the action selection problem as an inference problem in the context of Bayesian brain theories, also known as Active Inference in the literature. We elucidate the main ideas and discuss critical technical and conceptual issues revolving around these two notions of free energy that both claim to apply at all levels of decision-making, from the high-level deliberation of reasoning down to the low-level information processing of perception.

Active inference on discrete state-spaces: A synthesis

Active inference is a normative principle underwriting perception, action, planning, decision-making and learning in biological or artificial agents. From its inception, its associated process theory has grown to incorporate complex generative models, enabling simulation of a wide range of complex behaviours. Due to successive developments in active inference, it is often difficult to see how its underlying principle relates to process theories and practical implementation. In this paper, we try to bridge this gap by providing a complete mathematical synthesis of active inference on discrete state-space models. This technical summary provides an overview of the theory, derives neuronal dynamics from first principles and relates this dynamics to biological processes. Furthermore, this paper provides a fundamental building block needed to understand active inference for mixed generative models; allowing continuous sensations to inform discrete representations. This paper may be used as follows: to guide research towards outstanding challenges, a practical guide on how to implement active inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions.

Variational Free Energy and Economics Optimizing With Biases and Bounded Rationality

The purpose of this paper is to offer a new framework for understanding action, optimization, and choice when applied to economic theory more generally. By drawing upon the concept known as the variational free energy principle, the paper will explore how this principle can be used to temper rational choice theory by re-formulating how agents optimize. The approach will result in agent behavior that encompasses a wide range of so-called cognitive biases, as seen in the scientific literature of behavioral economics, but instead of using these biases as further indications of market inefficiencies or market failures, the paper will likewise attempt to show the limits to which these biases can inform or critique standard economic theory. The paper therefore offers up a “middle of the road” approach, in which the neoclassical agent is not quite as “rational” as rational choice theory assumes, but at the same time, not quite as irrational as behavioral economics would often have us believe.

Learning Generative State Space Models for Active Inference

In this paper we investigate the active inference framework as a means to enable autonomous behavior in artificial agents. Active inference is a theoretical framework underpinning the way organisms act and observe in the real world. In active inference, agents act in order to minimize their so called free energy, or prediction error. Besides being biologically plausible, active inference has been shown to solve hard exploration problems in various simulated environments. However, these simulations typically require handcrafting a generative model for the agent. Therefore we propose to use recent advances in deep artificial neural networks to learn generative state space models from scratch, using only observation-action sequences. This way we are able to scale active inference to new and challenging problem domains, whilst still building on the theoretical backing of the free energy principle. We validate our approach on the mountain car problem to illustrate that our learnt models can indeed trade-off instrumental value and ambiguity. Furthermore, we show that generative models can also be learnt using high-dimensional pixel observations, both in the OpenAI Gym car racing environment and a real-world robotic navigation task. Finally we show that active inference based policies are an order of magnitude more sample efficient than Deep Q Networks on RL tasks.

Generative models, linguistic communication and active inference

This paper presents a biologically plausible generative model and inference scheme that is capable of simulating communication between synthetic subjects who talk to each other. Building on active inference formulations of dyadic interactions, we simulate linguistic exchange to explore generative models that support dialogues. These models employ high-order interactions among abstract (discrete) states in deep (hierarchical) models. The sequential nature of language processing mandates generative models with a particular factorial structure-necessary to accommodate the rich combinatorics of language. We illustrate linguistic communication by simulating a synthetic subject who can play the 'Twenty Questions' game. In this game, synthetic subjects take the role of the questioner or answerer, using the same generative model. This simulation setup is used to illustrate some key architectural points and demonstrate that many behavioural and neurophysiological correlates of linguistic communication emerge under variational (marginal) message passing, given the right kind of generative model. For example, we show that theta-gamma coupling is an emergent property of belief updating, when listening to another.

Tracking the Influence of Predictive Cues on the Evaluation of Food Images: Volatility Enables Nudging

In previous research on the evaluation of food images, we found that appetitive food images were rated higher following a positive prediction than following a negative prediction, and vice versa for aversive food images. The findings suggested an active confirmation bias. Here, we examine whether this influence from prediction depends on the evaluative polarization of the food images. Specifically, we divided the set of food images into “strong” and “mild” images by how polarized (i.e., extreme) their average ratings were across all conditions. With respect to the influence from prediction, we raise two alternative hypotheses. According to a predictive dissonance hypothesis, the larger the discrepancy between prediction and outcome, the stronger the active inference toward accommodating the outcome with the prediction; thus, the confirmation bias should obtain particularly with strong images. Conversely, according to a nudging-in-volatility hypothesis, the active confirmation bias operates only on images within a dynamic range, where the values of images are volatile, and not on the evaluation of images that are too obviously appetitive or aversive; accordingly, the effects from prediction should occur predominately with mild images. Across the data from two experiments, we found that the evaluation of mild images tended to exhibit the confirmation bias, with ratings that followed the direction given by the prediction. For strong images, there was no confirmation bias. Our findings corroborate the nudging-in-volatility hypothesis, suggesting that predictive cues may be able to tip the balance of evaluation particularly for food images that do not have a strongly polarized value.

The predictive global neuronal workspace: A formal active inference model of visual consciousness

The global neuronal workspace (GNW) model has inspired over two decades of hypothesis-driven research on the neural basis of consciousness. However, recent studies have reported findings that are at odds with empirical predictions of the model. Further, the macro-anatomical focus of current GNW research has limited the specificity of predictions afforded by the model. In this paper we present a neurocomputational model - based on Active Inference - that captures central architectural elements of the GNW and is able to address these limitations. The resulting 'predictive global workspace' casts neuronal dynamics as approximating Bayesian inference, allowing precise, testable predictions at both the behavioural and neural levels of description. We report simulations demonstrating the model's ability to reproduce: 1) the electrophysiological and behavioural results observed in previous studies of inattentional blindness; and 2) the previously introduced four-way taxonomy predicted by the GNW, which describes the relationship between consciousness, attention, and sensory signal strength. We then illustrate how our model can reconcile/explain (apparently) conflicting findings, extend the GNW taxonomy to include the influence of prior expectations, and inspire novel paradigms to test associated behavioural and neural predictions.

Better Safe Than Sorry: A Common Signature of General Vulnerability for Psychopathology

Several labels, such as neuroticism, negative emotionality, and dispositional negativity, indicate a broad dimension of psychopathology. However, largely separate, often disorder-specific research lines have developed that focus on different cognitive and affective characteristics that are associated with this dimension, such as perseverative cognition (worry, rumination), reduced autobiographical memory specificity, compromised fear learning, and enhanced somatic-symptom reporting. In this article, we present a theoretical perspective within a predictive-processing framework in which we trace these phenotypically different characteristics back to a common underlying “better-safe-than-sorry” processing strategy. This implies information processing that tends to be low in sensory-perceptual detail, which allows threat-related categorical priors to dominate conscious experience and for chronic uncertainty/surprise because of a stagnated error-reduction process. This common information-processing strategy has beneficial effects in the short term but important costs in the long term. From this perspective, we suggest that the phenomenally distinct cognitive and affective psychopathological characteristics mentioned above represent the same basic processing heuristic of the brain and are only different in relation to the particular type of information involved (e.g., in working memory, in autobiographical memory, in the external and internal world). Clinical implications of this view are discussed.

Cognitive Modeling of Automation Adaptation in a Time Critical Task

This paper presents a cognitive model that simulates an adaptation process to automation in a time-critical task. The paper uses a simple tracking task (which represents vehicle operation) to reveal how the reliance on automation changes as the success probabilities of the automatic and manual mode vary. The model was developed by using a cognitive architecture, ACT-R (Adaptive Control of Thought-Rational). We also introduce two methods of reinforcement learning: the summation of rewards over time and a gating mechanism. The model performs this task through productions that manage perception and motor control. The utility values of these productions are updated based on rewards in every perception-action cycle. A run of this model simulated the overall trends of the behavioral data such as the performance (tracking accuracy), the auto use ratio, and the number of switches between the two modes, suggesting some validity of the assumptions made in our model. This work shows how combining different paradigms of cognitive modeling can lead to practical representations and solutions to automation and trust in automation.

Degeneracy and Redundancy in Active Inference

The notions of degeneracy and redundancy are important constructs in many areas, ranging from genomics through to network science. Degeneracy finds a powerful role in neuroscience, explaining key aspects of distributed processing and structure-function relationships in the brain. For example, degeneracy accounts for the superadditive effect of lesions on functional deficits in terms of a "many-to-one" structure-function mapping. In this paper, we offer a principled account of degeneracy and redundancy, when function is operationalized in terms of active inference, namely, a formulation of perception and action as belief updating under generative models of the world. In brief, "degeneracy" is quantified by the "entropy" of posterior beliefs about the causes of sensations, while "redundancy" is the "complexity" cost incurred by forming those beliefs. From this perspective, degeneracy and redundancy are complementary: Active inference tries to minimize redundancy while maintaining degeneracy. This formulation is substantiated using statistical and mathematical notions of degenerate mappings and statistical efficiency. We then illustrate changes in degeneracy and redundancy during the learning of a word repetition task. Finally, we characterize the effects of lesions-to intrinsic and extrinsic connections-using in silico disconnections. These numerical analyses highlight the fundamental difference between degeneracy and redundancy-and how they score distinct imperatives for perceptual inference and structure learning that are relevant to synthetic and biological intelligence.

Paradoxical lesions, plasticity and active inference

Paradoxical lesions are secondary brain lesions that ameliorate functional deficits caused by the initial insult. This effect has been explained in several ways; particularly by the reduction of functional inhibition, or by increases in the excitatory-to-inhibitory synaptic balance within perilesional tissue. In this article, we simulate how and when a modification of the excitatory-inhibitory balance triggers the reversal of a functional deficit caused by a primary lesion. For this, we introduce in-silico lesions to an active inference model of auditory word repetition. The first in-silico lesion simulated damage to the extrinsic (between regions) connectivity causing a functional deficit that did not fully resolve over 100 trials of a word repetition task. The second lesion was implemented in the intrinsic (within region) connectivity, compromising the model's ability to rebalance excitatory-inhibitory connections during learning. We found that when the second lesion was mild, there was an increase in experience-dependent plasticity that enhanced performance relative to a single lesion. This paradoxical lesion effect disappeared when the second lesion was more severe because plasticity-related changes were disproportionately amplified in the intrinsic connectivity, relative to lesioned extrinsic connections. Finally, this framework was used to predict the physiological correlates of paradoxical lesions. This formal approach provides new insights into the computational and neurophysiological mechanisms that allow some patients to recover after large or multiple lesions.

Reverse-Engineering Neural Networks to Characterize Their Cost Functions

This letter considers a class of biologically plausible cost functions for neural networks, where the same cost function is minimized by both neural activity and plasticity. We show that such cost functions can be cast as a variational bound on model evidence under an implicit generative model. Using generative models based on partially observed Markov decision processes (POMDP), we show that neural activity and plasticity perform Bayesian inference and learning, respectively, by maximizing model evidence. Using mathematical and numerical analyses, we establish the formal equivalence between neural network cost functions and variational free energy under some prior beliefs about latent states that generate inputs. These prior beliefs are determined by particular constants (e.g., thresholds) that define the cost function. This means that the Bayes optimal encoding of latent or hidden states is achieved when the network's implicit priors match the process that generates its inputs. This equivalence is potentially important because it suggests that any hyperparameter of a neural network can itself be optimized-by minimization with respect to variational free energy. Furthermore, it enables one to characterize a neural network formally, in terms of its prior beliefs.

"Stopping for knowledge": The sense of beauty in the perception-action cycle

According to a millennial-old philosophical debate, aesthetic emotions have been connected to knowledge acquisition. Recent scientific evidence, collected across different disciplinary domains, confirms this link, but also reveals that motor inhibition plays a crucial role in the process. In this review, we discuss multidisciplinary results and propose an original account of aesthetic appreciation (the stopping for knowledge hypothesis) framed within the predictive coding theory. We discuss evidence showing that aesthetic emotions emerge in correspondence with an inhibition of motor behavior (i.e., minimizing action), promoting a simultaneous perceptual processing enhancement, at the level of sensory cortices (i.e., optimizing learning). Accordingly, we suggest that aesthetic appreciation may represent a hedonic feedback over learning progresses, motivating the individual to inhibit motor routines to seek further knowledge acquisition. Furthermore, the neuroimaging and neuropsychological studies we review reveal the presence of a strong association between aesthetic appreciation and the activation of the dopaminergic reward-related circuits. Finally, we propose a number of possible applications of the stopping for knowledge hypothesis in the clinical and education domains.

Wilding the predictive brain

The Predictive Processing (PP) framework casts the brain as a probabilistic prediction engine that continually generates predictions of the causal structure of the world in order to construct for itself, from the top down, incoming sensory signals. Conceiving of the brain in this way has yielded incredible explanatory power, offering what many believe to be our first glimpse at a unified theory of the mind. In this paper, the picture of the mind brought into view by predictive processing theories is shown to be embodied, deeply affective and nicely poised for cognitive extension. We begin by giving an overview of the main themes of the framework, and situating this approach within embodied cognitive science. We show perception, action, homeostatic regulation and emotion to be underpinned by the very same predictive machinery. We conclude by showing how predictive minds will increasingly be understood as deeply interwoven with, and perhaps extended into, the surrounding social, cultural and technological landscape. This article is categorized under: Philosophy > Foundations of Cognitive Science Psychology > Emotion and Motivation Philosophy > Action.

A Bayesian Account of Generalist and Specialist Formation Under the Active Inference Framework

This paper offers a formal account of policy learning, or habitual behavioral optimization, under the framework of Active Inference. In this setting, habit formation becomes an autodidactic, experience-dependent process, based upon what the agent sees itself doing. We focus on the effect of environmental volatility on habit formation by simulating artificial agents operating in a partially observable Markov decision process. Specifically, we used a "two-step" maze paradigm, in which the agent has to decide whether to go left or right to secure a reward. We observe that in volatile environments with numerous reward locations, the agents learn to adopt a generalist strategy, never forming a strong habitual behavior for any preferred maze direction. Conversely, in conservative or static environments, agents adopt a specialist strategy; forming strong preferences for policies that result in approach to a small number of previously-observed reward locations. The pros and cons of the two strategies are tested and discussed. In general, specialization offers greater benefits, but only when contingencies are conserved over time. We consider the implications of this formal (Active Inference) account of policy learning for understanding the relationship between specialization and habit formation.

An insula-driven network computes decision uncertainty and promotes abstinence in chronic cocaine users

The anterior insular cortex (AIC) and its interconnected brain regions have been associated with both addiction and decision-making under uncertainty. However, the causal interactions in this uncertainty-encoding neurocircuitry and how these neural dynamics impact relapse remain elusive. Here, we used model-based fMRI to measure choice uncertainty in a motor decision task in 61 individuals with cocaine use disorder (CUD) and 25 healthy controls. CUD participants were assessed before discharge from a residential treatment program and followed for up to 24 weeks. We found that choice uncertainty was tracked by the AIC, dorsal anterior cingulate cortex (dACC) and ventral striatum (VS), across participants. Stronger activations in these regions measured pre-discharge predicted longer abstinence after discharge in individuals with CUD. Dynamic causal modeling revealed an AIC-to-dACC-directed connectivity modulated by uncertainty in controls, but a dACC-to-AIC connectivity in CUD participants. This reversal was mostly driven by early relapsers (<30 days). Furthermore, CUD individuals who displayed a stronger AIC-to-dACC excitatory connection during uncertainty encoding remained abstinent for longer periods. These findings reveal a critical role of an AIC-driven, uncertainty-encoding neurocircuitry in protecting against relapse and promoting abstinence.