Active inference and learning

TTOM in action: Refining the variational approach to cognition and culture

The target article "Thinking Through Other Minds" (TTOM) offered an account of the distinctively human capacity to acquire cultural knowledge, norms, and practices. To this end, we leveraged recent ideas from theoretical neurobiology to understand the human mind in social and cultural contexts. Our aim was both synthetic - building an integrative model adequate to account for key features of cultural learning and adaptation; and prescriptive - showing how the tools developed to explain brain dynamics can be applied to the emergence of social and cultural ecologies of mind. In this reply to commentators, we address key issues, including: (1) refining the concept of culture to show how TTOM and the free-energy principle (FEP) can capture essential elements of human adaptation and functioning; (2) addressing cognition as an embodied, enactive, affective process involving cultural affordances; (3) clarifying the significance of the FEP formalism related to entropy minimization, Bayesian inference, Markov blankets, and enactivist views; (4) developing empirical tests and applications of the TTOM model; (5) incorporating cultural diversity and context at the level of intra-cultural variation, individual differences, and the transition to digital niches; and (6) considering some implications for psychiatry. The commentators' critiques and suggestions point to useful refinements and applications of the model. In ongoing collaborations, we are exploring how to augment the theory with affective valence, take into account individual differences and historicity, and apply the model to specific domains including epistemic bias.

The Active Sensing of Control Difference

In everyday life, people control objects in the world around them to varying degrees. The processes people actively use to establish their control, while interacting with an environment containing large ambiguity, remain unknown. This study examines how people explore their control over the environment and how they detect small differences in control among objects. In the experimental task, participants moved three dots on a screen and identified one dot over which the level of control is different from that of the other two. The results support a two-step behavior mechanism underlying the sensing of control difference: People first explore their overall control in the environment, and then the results of the initial exploration are used to selectively tune the direction (i.e., either more or less) of the detected control difference, ensuring efficient and rapid detection of the type of control difference that is potentially important for further action selections.

An Active Inference Approach to Modeling Structure Learning: Concept Learning as an Example Case

Within computational neuroscience, the algorithmic and neural basis of structure learning remains poorly understood. Concept learning is one primary example, which requires both a type of internal model expansion process (adding novel hidden states that explain new observations), and a model reduction process (merging different states into one underlying cause and thus reducing model complexity via meta-learning). Although various algorithmic models of concept learning have been proposed within machine learning and cognitive science, many are limited to various degrees by an inability to generalize, the need for very large amounts of training data, and/or insufficiently established biological plausibility. Using concept learning as an example case, we introduce a novel approach for modeling structure learning-and specifically state-space expansion and reduction-within the active inference framework and its accompanying neural process theory. Our aim is to demonstrate its potential to facilitate a novel line of active inference research in this area. The approach we lay out is based on the idea that a generative model can be equipped with extra (hidden state or cause) "slots" that can be engaged when an agent learns about novel concepts. This can be combined with a Bayesian model reduction process, in which any concept learning-associated with these slots-can be reset in favor of a simpler model with higher model evidence. We use simulations to illustrate this model's ability to add new concepts to its state space (with relatively few observations) and increase the granularity of the concepts it currently possesses. We also simulate the predicted neural basis of these processes. We further show that it can accomplish a simple form of "one-shot" generalization to new stimuli. Although deliberately simple, these simulation results highlight ways in which active inference could offer useful resources in developing neurocomputational models of structure learning. They provide a template for how future active inference research could apply this approach to real-world structure learning problems and assess the added utility it may offer.

Investigating functional changes in the brain to intermittently induced auditory illusions and its relevance to chronic tinnitus

Several studies have demonstrated the neural correlates of chronic tinnitus. However, we still do not understand what happens in the acute phase. Past studies have established Zwicker tone (ZT) illusions as a good human model for acute tinnitus. ZT illusions are perceived following the presentation of a notched noise stimulus, that is, broadband noise with a narrow band-stop filter (notch). In the current study, we compared the neural correlates of the reliable perception of a ZT illusion to that which is not. We observed changes in evoked and total theta power in wide-spread regions of the brain particularly in the temporal-parietal junction, pregenual anterior cingulate cortex/ventromedial prefrontal cortex (pgACC/vmPFC), parahippocampus during perception of the ZT illusion. Furthermore, we observe that increased theta power significantly predicts a gradual positive change in the intensity of the ZT illusion. Such changes may suggest a malfunction of the sensory gating system that enables habituation to redundant stimuli and suppresses hyperactivity. It could also suggest a successful retrieval of the memory of the missing frequencies, resulting in their conscious perception indicating the role of higher-order processing in the mechanism of action of ZT illusions. To establish a more concrete relationship between ZT illusion and chronic tinnitus, future longitudinal studies following up a much larger sample of participants who reliably perceive a ZT illusion to see if they develop tinnitus at a later stage is essential. This could inform us if the ZT illusion may be a precursor to chronic tinnitus.

An Investigation of the Free Energy Principle for Emotion Recognition

This paper offers a prospectus of what might be achievable in the development of emotional recognition devices. It provides a conceptual overview of the free energy principle; including Markov blankets, active inference, and-in particular-a discussion of selfhood and theory of mind, followed by a brief explanation of how these concepts can explain both neural and cultural models of emotional inference. The underlying hypothesis is that emotion recognition and inference devices will evolve from state-of-the-art deep learning models into active inference schemes that go beyond marketing applications and become adjunct to psychiatric practice. Specifically, this paper proposes that a second wave of emotion recognition devices will be equipped with an emotional lexicon (or the ability to epistemically search for one), allowing the device to resolve uncertainty about emotional states by actively eliciting responses from the user and learning from these responses. Following this, a third wave of emotional devices will converge upon the user's generative model, resulting in the machine and human engaging in a reciprocal, prosocial emotional interaction, i.e., sharing a generative model of emotional states.

The QBIT Theory of Consciousness

The QBIT theory is an attempt toward solving the problem of consciousness based on empirical evidence provided by various scientific disciplines including quantum mechanics, biology, information theory, and thermodynamics. This theory formulates the problem of consciousness in the following four questions, and provides preliminary answers for each question: Question 1: What is the nature of qualia?

ANSWER

A quale is a superdense pack of quantum information encoded in maximally entangled pure states. Question 2: How are qualia generated?

ANSWER

When a pack of quantum information is compressed beyond a certain threshold, a quale is generated. Question 3: Why are qualia subjective?

ANSWER

A quale is subjective because a pack of information encoded in maximally entangled pure states are essentially private and unshareable. Question 4: Why does a quale have a particular meaning?

ANSWER

A pack of information within a cognitive system gradually obtains a particular meaning as it undergoes a progressive process of interpretation performed by an internal model installed in the system. This paper introduces the QBIT theory of consciousness, and explains its basic assumptions and conjectures.

Learning action-oriented models through active inference

Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms.

Driven by Pain, Not Gain: Computational Approaches to Aversion-Related Decision Making in Psychiatry

Although it is well known that "losses loom larger than gains," computational approaches to aversion-related decision making (ARDM) for psychiatric disorders is an underdeveloped area. Computational models of ARDM have been implemented primarily as state-dependent reinforcement learning models with bias parameters to quantify Pavlovian associations, and differential learning rates to quantify instrumental updating have been shown to depend on context, involve complex cost calculations, and include the consideration of counterfactual outcomes. Little is known about how individual differences influence these models relevant to anxiety-related conditions or addiction-related dysfunction. It is argued that model parameters reflecting 1) Pavlovian biases in the context of reinforcement learning or 2) hyperprecise prior beliefs in the context of active inference play an important role in the emergence of dysfunctional avoidance behaviors. The neural implementation of ARDM includes brain areas that are important for valuation (ventromedial prefrontal cortex) and positive reinforcement-related prediction errors (ventral striatum), but also aversive processing (insular cortex and cerebellum). Computational models of ARDM will help to establish a quantitative explanatory account of the development of anxiety disorders and addiction, but such models also face several challenges, including limited evidence for stability of individual differences, relatively low reliability of tasks, and disorder heterogeneity. Thus, it will be necessary to develop robust, reliable, and model-based experimental probes; recruit larger sample sizes; and use single case experimental designs for better pragmatic and explanatory biological models of psychiatric disorders.

Neural Autopoiesis: Organizing Self-Boundaries by Stimulus Avoidance in Biological and Artificial Neural Networks

Living organisms must actively maintain themselves in order to continue existing. Autopoiesis is a key concept in the study of living organisms, where the boundaries of the organism are not static but dynamically regulated by the system itself. To study the autonomous regulation of a self-boundary, we focus on neural homeodynamic responses to environmental changes using both biological and artificial neural networks. Previous studies showed that embodied cultured neural networks and spiking neural networks with spike-timing dependent plasticity (STDP) learn an action as they avoid stimulation from outside. In this article, as a result of our experiments using embodied cultured neurons, we find that there is also a second property allowing the network to avoid stimulation: If the agent cannot learn an action to avoid the external stimuli, it tends to decrease the stimulus-evoked spikes, as if to ignore the uncontrollable input. We also show such a behavior is reproduced by spiking neural networks with asymmetric STDP. We consider that these properties are to be regarded as autonomous regulation of self and nonself for the network, in which a controllable neuron is regarded as self, and an uncontrollable neuron is regarded as nonself. Finally, we introduce neural autopoiesis by proposing the principle of stimulus avoidance.

Doubting what you already know: Uncertainty regarding state transitions is associated with obsessive compulsive symptoms

Obsessive compulsive (OC) symptoms involve excessive information gathering (e.g., checking, reassurance-seeking), and uncertainty about possible, often catastrophic, future events. Here we propose that these phenomena are the result of excessive uncertainty regarding state transitions (transition uncertainty): a computational impairment in Bayesian inference leading to a reduced ability to use the past to predict the present and future, and to oversensitivity to feedback (i.e. prediction errors). Using a computational model of Bayesian learning under uncertainty in a reversal learning task, we investigate the relationship between OC symptoms and transition uncertainty. Individuals high and low in OC symptoms performed a task in which they had to detect shifts (i.e. transitions) in cue-outcome contingencies. Modeling subjects' choices was used to estimate each individual participant's transition uncertainty and associated responses to feedback. We examined both an optimal observer model and an approximate Bayesian model in which participants were assumed to attend (and learn about) only one of several cues on each trial. Results suggested the participants were more likely to distribute attention across cues, in accordance with the optimal observer model. As hypothesized, participants with higher OC symptoms exhibited increased transition uncertainty, as well as a pattern of behavior potentially indicative of a difficulty in relying on learned contingencies, with no evidence for perseverative behavior. Increased transition uncertainty compromised these individuals' ability to predict ensuing feedback, rendering them more surprised by expected outcomes. However, no evidence for excessive belief updating was found. These results highlight a potential computational basis for OC symptoms and obsessive compulsive disorder (OCD). The fact the OC symptoms predicted a decreased reliance on the past rather than perseveration challenges preconceptions of OCD as a disorder of inflexibility. Our results have implications for the understanding of the neurocognitive processes leading to excessive uncertainty and distrust of past experiences in OCD.

The sense of should: A biologically-based framework for modeling social pressure

What is social pressure, and how could it be adaptive to conform to others' expectations? Existing accounts highlight the importance of reputation and social sanctions. Yet, conformist behavior is multiply determined: sometimes, a person desires social regard, but at other times she feels obligated to behave a certain way, regardless of any reputational benefit-i.e. she feels a sense of should. We develop a formal model of this sense of should, beginning from a minimal set of biological premises: that the brain is predictive, that prediction error has a metabolic cost, and that metabolic costs are prospectively avoided. It follows that unpredictable environments impose metabolic costs, and in social environments these costs can be reduced by conforming to others' expectations. We elaborate on a sense of should's benefits and subjective experience, its likely developmental trajectory, and its relation to embodied mental inference. From this individualistic metabolic strategy, the emergent dynamics unify social phenomenon ranging from status quo biases, to communication and motivated cognition. We offer new solutions to long-studied problems (e.g. altruistic behavior), and show how compliance with arbitrary social practices is compelled without explicit sanctions. Social pressure may provide a foundation in individuals on which societies can be built.

Simulating Emotions: An Active Inference Model of Emotional State Inference and Emotion Concept Learning

The ability to conceptualize and understand one's own affective states and responses - or "Emotional awareness" (EA) - is reduced in multiple psychiatric populations; it is also positively correlated with a range of adaptive cognitive and emotional traits. While a growing body of work has investigated the neurocognitive basis of EA, the neurocomputational processes underlying this ability have received limited attention. Here, we present a formal Active Inference (AI) model of emotion conceptualization that can simulate the neurocomputational (Bayesian) processes associated with learning about emotion concepts and inferring the emotions one is feeling in a given moment. We validate the model and inherent constructs by showing (i) it can successfully acquire a repertoire of emotion concepts in its "childhood", as well as (ii) acquire new emotion concepts in synthetic "adulthood," and (iii) that these learning processes depend on early experiences, environmental stability, and habitual patterns of selective attention. These results offer a proof of principle that cognitive-emotional processes can be modeled formally, and highlight the potential for both theoretical and empirical extensions of this line of research on emotion and emotional disorders.

Symptom Perception From a Predictive Processing Perspective

Bodily symptoms are highly prevalent in psychopathology, and in some specific disorders, such as somatic symptom disorder, they are a central feature. In general, the mechanisms underlying these symptoms are poorly understood. However, also in well-known physical diseases there seems to be a variable relationship between physiological dysfunction and self-reported symptoms challenging traditional assumptions of a biomedical disease model.

Recently, a new, predictive processing conceptualization of how the brain works has been used to understand this variable relationship. According to this predictive processing view, the experience of a symptom results from an integration of both interoceptive sensations as well as from predictions about these sensations from the brain.

In the present paper, we introduce the predictive processing perspective on perception (predictive coding) and action (active inference), and apply it to asthma in order to understand when and why asthma symptoms are sometimes strongly, moderately or weakly related to physiological disease parameters.

Our predictive processing view of symptom perception contributes to understanding under which conditions misperceptions and maladaptive action selection may arise.

There is a variable relationship between physiological dysfunction and self-reported symptoms.

We conceptualize symptom perception (and misperception) within a predictive processing perspective.

In this view, symptom perception integrates sensations and predictions about these sensations.

Failures of such integration can produce misperceptions and maladaptive action selection.

We use the perception (and misperception) of asthma symptoms as an example.

Aesthetic appreciation of musical intervals enhances behavioural and neurophysiological indexes of attentional engagement and motor inhibition

From Kant to current perspectives in neuroaesthetics, the experience of beauty has been described as disinterested, i.e. focusing on the stimulus perceptual features while neglecting self-referred concerns. At a neurophysiological level, some indirect evidence suggests that disinterested aesthetic appreciation might be associated with attentional enhancement and inhibition of motor behaviour. To test this hypothesis, we performed three auditory-evoked potential experiments, employing consonant and dissonant two-note musical intervals. Twenty-two volunteers judged the beauty of intervals (Aesthetic Judgement task) or responded to them as fast as possible (Detection task). In a third Go-NoGo task, a different group of twenty-two participants had to refrain from responding when hearing intervals. Individual aesthetic judgements positively correlated with response times in the Detection task, with slower motor responses for more appreciated intervals. Electrophysiological indexes of attentional engagement (N1/P2) and motor inhibition (N2/P3) were enhanced for more appreciated intervals. These findings represent the first experimental evidence confirming the disinterested interest hypothesis and may have important applications in research areas studying the effects of stimulus features on learning and motor behaviour.

Mental Effort and Information-Processing Costs Are Inversely Related to Global Brain Free Energy During Visual Categorization

Mental effort is a neurocognitive process that reflects the controlled expenditure of psychological information-processing resources during perception, cognition, and action. There is a practical need to operationalize and measure mental effort in order to minimize detrimental effects of mental fatigue on real-world human performance. Previous research has identified several neurocognitive indices of mental effort, but these indices are indirect measures that are also sensitive to experimental demands or general factors such as sympathetic arousal. The present study investigated a potential direct neurocognitive index of mental effort based in theories where bounded rational decision makers (realized as embodied brains) are modeled as generalized thermodynamic systems. This index is called free energy, an information-theoretic system property of the brain that reflects the difference between the brain's current and predicted states. Theory predicts that task-related differences in a decision makers' free energy are inversely related to information-processing costs related to task decisions. The present study tested this prediction by quantifying global brain free energy from electroencephalographic (EEG) measures of human brain function. EEG signals were recorded while participants engaged in two visual categorization tasks in which categorization decisions resulted from the allocation of different levels of mental information processing resources. A novel method was developed to quantify brain free energy from machine learning classification of EEG trials. Participant information-processing resource costs were estimated via computational analysis of behavior, whereas the subjective expression of mental effort was estimated via participant ratings of mental workload. Following theoretical predictions, task-related differences in brain free energy negatively correlated with increased allocation of information-processing resource costs. These brain free energy differences were smaller for the visual categorization task that required a greater versus lesser allocation of information-processing resources. Ratings of mental workload were positively correlated with information-processing resource costs, and negatively correlated with global brain free energy differences, only for the categorization task requiring the larger amount of information-processing resource costs. These findings support theoretical thermodynamic approaches to decision making and provide the first empirical evidence of a relationship between mental effort, brain free energy, and neurocognitive information-processing.

Variational ecology and the physics of sentient systems

This paper addresses the challenges faced by multiscale formulations of the variational (free energy) approach to dynamics that obtain for large-scale ensembles. We review a framework for modelling complex adaptive control systems for multiscale free energy bounding organism-niche dynamics, thereby integrating the modelling strategies and heuristics of variational neuroethology with a broader perspective on the ecological nestedness of biotic systems. We extend the multiscale variational formulation beyond the action-perception loops of individual organisms by appealing to the variational approach to niche construction to explain the dynamics of coupled systems constituted by organisms and their ecological niche. We suggest that the statistical robustness of living systems is inherited, in part, from their eco-niches, as niches help coordinate dynamical patterns across larger spatiotemporal scales. We call this approach variational ecology. We argue that, when applied to cultural animals such as humans, variational ecology enables us to formulate not just a physics of individual minds, but also a physics of interacting minds across spatial and temporal scales - a physics of sentient systems that range from cells to societies.

Generalised free energy and active inference

Active inference is an approach to understanding behaviour that rests upon the idea that the brain uses an internal generative model to predict incoming sensory data. The fit between this model and data may be improved in two ways. The brain could optimise probabilistic beliefs about the variables in the generative model (i.e. perceptual inference). Alternatively, by acting on the world, it could change the sensory data, such that they are more consistent with the model. This implies a common objective function (variational free energy) for action and perception that scores the fit between an internal model and the world. We compare two free energy functionals for active inference in the framework of Markov decision processes. One of these is a functional of beliefs (i.e. probability distributions) about states and policies, but a function of observations, while the second is a functional of beliefs about all three. In the former (expected free energy), prior beliefs about outcomes are not part of the generative model (because they are absorbed into the prior over policies). Conversely, in the second (generalised free energy), priors over outcomes become an explicit component of the generative model. When using the free energy function, which is blind to future observations, we equip the generative model with a prior over policies that ensure preferred (i.e. priors over) outcomes are realised. In other words, if we expect to encounter a particular kind of outcome, this lends plausibility to those policies for which this outcome is a consequence. In addition, this formulation ensures that selected policies minimise uncertainty about future outcomes by minimising the free energy expected in the future. When using the free energy functional-that effectively treats future observations as hidden states-we show that policies are inferred or selected that realise prior preferences by minimising the free energy of future expectations. Interestingly, the form of posterior beliefs about policies (and associated belief updating) turns out to be identical under both formulations, but the quantities used to compute them are not.

The hierarchically mechanistic mind: an evolutionary systems theory of the human brain, cognition, and behavior

The purpose of this review was to integrate leading paradigms in psychology and neuroscience with a theory of the embodied, situated human brain, called the Hierarchically Mechanistic Mind (HMM). The HMM describes the brain as a complex adaptive system that functions to minimize the entropy of our sensory and physical states via action-perception cycles generated by hierarchical neural dynamics. First, we review the extant literature on the hierarchical structure of the brain. Next, we derive the HMM from a broader evolutionary systems theory that explains neural structure and function in terms of dynamic interactions across four nested levels of biological causation (i.e., adaptation, phylogeny, ontogeny, and mechanism). We then describe how the HMM aligns with a global brain theory in neuroscience called the free-energy principle, leveraging this theory to mathematically formulate neural dynamics across hierarchical spatiotemporal scales. We conclude by exploring the implications of the HMM for psychological inquiry.

On How Definitions of Habits Can Complicate Habit Research

The core message of this paper is that many of the challenges of habit research can be traced back to the presence of causal elements within the definition of habits. For instance, the idea that habits are stimulus-driven implies that habitual behavior is not causally mediated by goal-representations. The presence of these causal elements in the definition of habits leads to difficulties in establishing empirically whether behavior is habitual. Some of these elements can also impoverish theoretical thinking about the mechanisms underlying habitual behavior. I argue that habit research would benefit from eliminating any reference to specific S-R association formation theories from the definition of habits. Which causal elements are retained in the definition of habits depends on the goals of researchers. However, regardless of the definition that is selected, it is good to be aware of the implications of the definition of habits for empirical and theoretical research on habits.

Prediction error and regularity detection underlie two dissociable mechanisms for computing the sense of agency

The sense of agency refers to the subjective feeling of controlling one's own actions, and through them, events in the outside world. According to computational motor control models, the prediction errors from comparison between the predicted sensory feedback and actual sensory feedback determine whether people feel agency over the corresponding outcome event, or not. This mechanism requires a model of the relation between action and outcome. However, in a novel environment, where this model has not yet been learned, the sense of agency must emerge during exploratory behaviours. In the present study, we designed a novel control detection task, in which participants explored the extent to which they could control the movement of three dots with a computer mouse, and then identified the dot that they felt they could control. Pre-recorded motions were applied for two dots, and the participants' real-time motion only influenced one dot's motion (i.e. the target dot). We disturbed participants' control over the motion of the target dot in one of two ways. In one case, we applied a fixed angular bias transformation between participant's movements and dot movements. In another condition, we mixed the participant's current movement with replay of another movement, and used the resulting hybrid signal to drive visual dot position. The former intervention changes the match between motor action and visual outcome, but maintains a regular relation between the two. In contrast, the latter alters both matching and motor-visual correlation. Crucially, we carefully selected the strength of these two perturbations so that they caused the same magnitude of impairment of motor performance in a simple reaching task, suggesting that both interventions produced comparable prediction errors. However, we found the visuomotor transformation had much less effect on the ability to detect which dot was under one's own control than did the nonlinear disturbance. This suggests a specific role of a correlation-like mechanism that detects ongoing visual-motor regularity in the human sense of agency. These regularity-detection mechanisms would remain intact under the linear, but not the nonlinear transformation. Human sense of agency may depend on monitoring ongoing motor-visual regularities, as well as on detecting prediction errors.

Modeling other minds: Bayesian inference explains human choices in group decision-making

To make decisions in a social context, humans have to predict the behavior of others, an ability that is thought to rely on having a model of other minds known as "theory of mind." Such a model becomes especially complex when the number of people one simultaneously interacts with is large and actions are anonymous. Here, we present results from a group decision-making task known as the volunteer's dilemma and demonstrate that a Bayesian model based on partially observable Markov decision processes outperforms existing models in quantitatively predicting human behavior and outcomes of group interactions. Our results suggest that in decision-making tasks involving large groups with anonymous members, humans use Bayesian inference to model the "mind of the group," making predictions of others' decisions while also simulating the effects of their own actions on the group's dynamics in the future.

Tonality Tunes the Statistical Characteristics in Music: Computational Approaches on Statistical Learning

Statistical learning is a learning mechanism based on transition probability in sequences such as music and language. Recent computational and neurophysiological studies suggest that the statistical learning contributes to production, action, and musical creativity as well as prediction and perception. The present study investigated how statistical structure interacts with tonalities in music based on various-order statistical models. To verify this in all 24 major and minor keys, the transition probabilities of the sequences containing the highest pitches in Bach's Well-Tempered Clavier, which is a collection of two series (No. 1 and No. 2) of preludes and fugues in all of the 24 major and minor keys, were calculated based on nth-order Markov models. The transition probabilities of each sequence were compared among tonalities (major and minor), two series (No. 1 and No. 2), and music types (prelude and fugue). The differences in statistical characteristics between major and minor keys were detected in lower- but not higher-order models. The results also showed that statistical knowledge in music might be modulated by tonalities and composition periods. Furthermore, the principal component analysis detected the shared components of related keys, suggesting that the tonalities modulate statistical characteristics in music. The present study may suggest that there are at least two types of statistical knowledge in music that are interdependent on and independent of tonality, respectively.

Working With the Predictable Life of Patients: The Importance of "Mentalizing Interoception" to Meaningful Change in Psychotherapy

To understand our patients and optimize their treatment, psychotherapists of all theoretical orientations may benefit from considering current scientific evidence alongside psychodynamic constructs. There is recent neuroscientific evidence that subjective awareness, feelings and emotions depend upon "interoception," defined as the neural signaling to the brain from all tissues of the body. Interoception is the obvious basis of homeostasis (in the brainstem) but some interoceptive signals rise above this level and contribute to inferential processes that substantiate intrapersonal and interpersonal experience. The focus of this paper is on the essential role that their "interoception" plays in our patients' emotional experience and subjective awareness, and how the process referred to as "mentalizing interoception" may be harnessed in therapy. This can best be understood in terms of "predictive processing," which describes how subjective states, and particularly emotion, are inferred from sensory inputs - both interoceptive and exteroceptive. Predictive processing assumes that the brain infers (probabilistically) the likely cause of sensation experienced through the sense organs, by testing this sensory data against its innate and learned "priors." This implies that any effort at changing heavily over-learned prior beliefs will require action upon the system that has generated that set of prior beliefs. This involves, quite literally, acting upon the world to alter inferential processes, or in the case of interoceptive priors, acting on the patient's body to alter habitual autonomic nervous system (ANS) reflexes. Focused attention to bodily sensations/reactions, in the safety of the therapeutic relationship, provides a route to "mentalizing interoception," by means of the bodily cues that may be the only conscious element of deeply hidden priors and thus the clearest way to access them. This can: update patients' characteristic, dysfunctional responses to emotion and feelings; increase emotional insight; decrease cognitive distortions; and engender a more acute awareness of the present moment. These important ideas are outlined below from the perspective of psychodynamic psychotherapeutic practice, in order to discuss how relevant information from neuroscientific theory and current research can best be applied in clinical treatment. A clinical case will be presented to illustrate how this argument or treatment relates directly to clinical practice.

Caching mechanisms for habit formation in Active Inference

A popular distinction in the human and animal learning literature is between deliberate (or willed) and habitual (or automatic) modes of control. Extensive evidence indicates that, after sufficient learning, living organisms develop behavioural habits that permit them saving computational resources. Furthermore, humans and other animals are able to transfer control from deliberate to habitual modes (and vice versa), trading off efficiently flexibility and parsimony - an ability that is currently unparalleled by artificial control systems. Here, we discuss a computational implementation of habit formation, and the transfer of control from deliberate to habitual modes (and vice versa) within Active Inference: a computational framework that merges aspects of cybernetic theory and of Bayesian inference. To model habit formation, we endow an Active Inference agent with a mechanism to "cache" (or memorize) policy probabilities from previous trials, and reuse them to skip - in part or in full - the inferential steps of deliberative processing. We exploit the fact that the relative quality of policies, conditioned upon hidden states, is constant over trials; provided that contingencies and prior preferences do not change. This means the only quantity that can change policy selection is the prior distribution over the initial state - where this prior is based upon the posterior beliefs from previous trials. Thus, an agent that caches the quality (or the probability) of policies can safely reuse cached values to save on cognitive and computational resources - unless contingencies change. Our simulations illustrate the computational benefits, but also the limits, of three caching schemes under Active Inference. They suggest that key aspects of habitual behaviour - such as perseveration - can be explained in terms of caching policy probabilities. Furthermore, they suggest that there may be many kinds (or stages) of habitual behaviour, each associated with a different caching scheme; for example, caching associated or not associated with contextual estimation. These schemes are more or less impervious to contextual and contingency changes.

Reframing PTSD for computational psychiatry with the active inference framework

Introduction: Recent advances in research on stress and, respectively, on disorders of perception, learning, and behaviour speak to a promising synthesis of current insights from (i) neurobiology, cognitive neuroscience and psychology of stress and post-traumatic stress disorder (PTSD), and (ii) computational psychiatry approaches to pathophysiology (e.g. of schizophrenia and autism). Methods: Specifically, we apply this synthesis to PTSD. The framework of active inference offers an embodied and embedded lens through which to understand neuronal mechanisms, structures, and processes of cognitive function and dysfunction. In turn, this offers an explanatory model of how healthy mental functioning can go awry due to psychopathological conditions that impair inference about our environment and our bodies. In this context, auditory phenomena-known to be especially relevant to studies of PTSD and schizophrenia-and traditional models of auditory function can be viewed from an evolutionary perspective based on active inference. Results: We assess and contextualise a range of evidence on audition, stress, psychosis, and PTSD, and bring some existing partial models of PTSD into multilevel alignment. Conclusions: The novel perspective on PTSD we present aims to serve as a basis for new experimental designs and therapeutic interventions that integrate fundamentally biological, cognitive, behavioural, and environmental factors.

Is there a prediction network? Meta-analytic evidence for a cortical-subcortical network likely subserving prediction

Predictive coding is an increasingly influential and ambitious concept in neuroscience viewing the brain as a 'hypothesis testing machine' that constantly strives to minimize prediction error, the gap between its predictions and the actual sensory input. Despite the invaluable contribution of this framework to the formulation of brain function, its neuroanatomical foundations have not been fully defined. To address this gap, we conducted activation likelihood estimation (ALE) meta-analysis of 39 neuroimaging studies of three functional domains (action perception, language and music) inherently involving prediction. The ALE analysis revealed a widely distributed brain network encompassing regions within the inferior and middle frontal gyri, anterior insula, premotor cortex, pre-supplementary motor area, temporoparietal junction, striatum, thalamus/subthalamus and the cerebellum. This network is proposed to subserve domain-general prediction and its relevance to motor control, attention, implicit learning and social cognition is discussed in light of the predictive coding scheme. Better understanding of the presented network may help advance treatments of neuropsychiatric conditions related to aberrant prediction processing and promote cognitive enhancement in healthy individuals.

Somatic multicellularity as a satisficing solution to the prediction-error minimization problem

Adaptive success in the biosphere requires the dynamic ability to adjust physiological, transcriptional, and behavioral responses to environmental conditions. From chemical networks to organisms to whole communities, biological entities at all levels of organization seek to optimize their predictive power. Here, we argue that this fundamental drive provides a novel perspective on the origin of multicellularity. One way for unicellular organisms to minimize surprise with respect to external inputs is to be surrounded by reproductively-disabled, i.e. somatic copies of themselves - highly predictable agents which in effect reduce uncertainty in their microenvironments. We show that the transition to multicellularity can be modeled as a phase transition driven by environmental threats. We present modeling results showing how multicellular bodies can arise if non-reproductive somatic cells protect their reproductive parents from environmental lethality. We discuss how a somatic body can be interpreted as a Markov blanket around one or more reproductive cells, and how the transition to somatic multicellularity can be represented as a transition from exposure of reproductive cells to a high-uncertainty environment to their protection from environmental uncertainty by this Markov blanket. This is, effectively, a transition by the Markov blanket from transparency to opacity for the variational free energy of the environment. We suggest that the ability to arrest the cell cycle of daughter cells and redirect their resource utilization from division to environmental threat amelioration is the key innovation of obligate multicellular eukaryotes, that the nervous system evolved to exercise this control over long distances, and that cancer is an escape by somatic cells from the control of reproductive cells. Our quantitative model illustrates the evolutionary dynamics of this system, provides a novel hypothesis for the origin of multicellular animal bodies, and suggests a fundamental link between the architectures of complex organisms and information processing in proto-cognitive cellular agents.

With an eye on uncertainty: Modelling pupillary responses to environmental volatility

Living creatures must accurately infer the nature of their environments. They do this despite being confronted by stochastic and context sensitive contingencies—and so must constantly update their beliefs regarding their uncertainty about what might come next. In this work, we examine how we deal with uncertainty that evolves over time. This prospective uncertainty (or imprecision) is referred to as volatility and has previously been linked to noradrenergic signals that originate in the locus coeruleus. Using pupillary dilatation as a measure of central noradrenergic signalling, we tested the hypothesis that changes in pupil diameter reflect inferences humans make about environmental volatility. To do so, we collected pupillometry data from participants presented with a stream of numbers. We generated these numbers from a process with varying degrees of volatility. By measuring pupillary dilatation in response to these stimuli—and simulating the inferences made by an ideal Bayesian observer of the same stimuli—we demonstrate that humans update their beliefs about environmental contingencies in a Bayes optimal way. We show this by comparing general linear (convolution) models that formalised competing hypotheses about the causes of pupillary changes. We found greater evidence for models that included Bayes optimal estimates of volatility than those without. We additionally explore the interaction between different causes of pupil dilation and suggest a quantitative approach to characterising a person’s prior beliefs about volatility.

Humans are constantly confronted with surprising events. To navigate such a world, we must understand the chances of an unexpected event occurring at any given point in time. We do this by creating a model of the world around us, in which we allow for these unexpected events to occur by holding beliefs about how volatile our environment is. In this work we explore the way in which we update our beliefs, demonstrating that this updating relies on the number of unexpected events in relation to the expected number. We do this by examining the pupil diameter, since—in controlled environments—changes in pupil diameter reflect our response to unexpected observations. Finally, we show that our methodology is appropriate for assessing the individual participant’s prior expectations about the amount of uncertainty in their environment.

The Human Default Consciousness and Its Disruption: Insights From an EEG Study of Buddhist Jhāna Meditation

The “neural correlates of consciousness” (NCC) is a familiar topic in neuroscience, overlapping with research on the brain’s “default mode network.” Task-based studies of NCC by their nature recruit one part of the cortical network to study another, and are therefore both limited and compromised in what they can reveal about consciousness itself. The form of consciousness explored in such research, we term the human default consciousness (DCs), our everyday waking consciousness. In contrast, studies of anesthesia, coma, deep sleep, or some extreme pathological states such as epilepsy, reveal very different cortical activity; all of which states are essentially involuntary, and generally regarded as “unconscious.” An exception to involuntary disruption of consciousness is Buddhist jhāna meditation, whose implicit aim is to intentionally withdraw from the default consciousness, to an inward-directed state of stillness referred to as jhāna consciousness, as a basis to develop insight. The default consciousness is sensorily-based, where information about, and our experience of, the outer world is evaluated against personal and organic needs and forms the basis of our ongoing self-experience. This view conforms both to Buddhist models, and to the emerging work on active inference and minimization of free energy in determining the network balance of the human default consciousness. This paper is a preliminary report on the first detailed EEG study of jhāna meditation, with findings radically different to studies of more familiar, less focused forms of meditation. While remaining highly alert and “present” in their subjective experience, a high proportion of subjects display “spindle” activity in their EEG, superficially similar to sleep spindles of stage 2 nREM sleep, while more-experienced subjects display high voltage slow-waves reminiscent, but significantly different, to the slow waves of deeper stage 4 nREM sleep, or even high-voltage delta coma. Some others show brief posterior spike-wave bursts, again similar, but with significant differences, to absence epilepsy. Some subjects also develop the ability to consciously evoke clonic seizure-like activity at will, under full control. We suggest that the remarkable nature of these observations reflects a profound disruption of the human DCs when the personal element is progressively withdrawn.

Tracking the Time Course of Bayesian Inference With Event-Related Potentials:A Study Using the Central Cue Posner Paradigm

In this study, we asked whether the event-related potentials associated to cue and target stimuli of a Central Cue Posner Paradigm (CCPP) may encode key parameters of Bayesian inference – prior expectation and surprise – on a trial-by-trial basis. Thirty-two EEG channel were recorded in a sample of 19 young adult subjects while performing a CCPP, in which a cue indicated (validly or invalidly) the position of an incoming auditory target. Three different types of blocks with validities of 50%, 64%, and 88%, respectively, were presented. Estimates of prior expectation and surprise were obtained on a trial-by-trial basis from participants’ responses, using a computational model implementing Bayesian learning. These two values were correlated on a trial-by-trial basis with the EEG values in all the electrodes and time bins. Therefore, a Spearman correlation metrics of the relationship between Bayesian parameters and the EEG was obtained. We report that the surprise parameter was able to classify the different validity blocks. Furthermore, the prior expectation parameter showed a significant correlation with the EEG in the cue-target period, in which the Contingent Negative Variation develops. Finally, in the post-target period the surprise parameter showed a significant correlation in the latencies and electrodes in which different event-related potentials are induced. Our results suggest that Bayesian parameters are coded in the EEG signals; and namely, the CNV would be related to prior expectation, while the post-target components P2a, P2, P3a, P3b, and SW would be related to surprise. This study thus provides novel support to the idea that human electrophysiological neural activity may implement a (Bayesian) predictive processing scheme.

Does delay in feedback diminish sense of agency? A review

Sense of agency refers to the subjective feeling of controlling one's own action, and through it, external events. Action-effect delay is widely used to disrupt this subjective feeling. Numerous studies have shown that self-reported sense of agency decreases along with the increase in delay. I discussed the distinction between body and external agency, and the possible different effects of delay on them. Furthermore, I reviewed literature that examined the influence of delay on self-reported sense of agency, implicit measures of sense of agency, and control-based action selection, and discussed possible reasons of the reported effects. Delay influences the measures of agency via multiple possible processes, such as graded response, task performance, sensory pre-activation, and temporal perceptual sensitivity. However, the causal relation between action and effect at higher-level of judgment may remain intact even for super-second delays. I conclude that the effects of delay on the sense of agency significantly differ between different levels, and researchers willing to use delay to disturb the sense of agency should carefully clarify which process it may affect.

Thinking through other minds: A variational approach to cognition and culture

The processes underwriting the acquisition of culture remain unclear. How are shared habits, norms, and expectations learned and maintained with precision and reliability across large-scale sociocultural ensembles? Is there a unifying account of the mechanisms involved in the acquisition of culture? Notions such as "shared expectations," the "selective patterning of attention and behaviour," "cultural evolution," "cultural inheritance," and "implicit learning" are the main candidates to underpin a unifying account of cognition and the acquisition of culture; however, their interactions require greater specification and clarification. In this article, we integrate these candidates using the variational (free-energy) approach to human cognition and culture in theoretical neuroscience. We describe the construction by humans of social niches that afford epistemic resources called cultural affordances. We argue that human agents learn the shared habits, norms, and expectations of their culture through immersive participation in patterned cultural practices that selectively pattern attention and behaviour. We call this process "thinking through other minds" (TTOM) - in effect, the process of inferring other agents' expectations about the world and how to behave in social context. We argue that for humans, information from and about other people's expectations constitutes the primary domain of statistical regularities that humans leverage to predict and organize behaviour. The integrative model we offer has implications that can advance theories of cognition, enculturation, adaptation, and psychopathology. Crucially, this formal (variational) treatment seeks to resolve key debates in current cognitive science, such as the distinction between internalist and externalist accounts of theory of mind abilities and the more fundamental distinction between dynamical and representational accounts of enactivism.

The emergence of synchrony in networks of mutually inferring neurons

This paper considers the emergence of a generalised synchrony in ensembles of coupled self-organising systems, such as neurons. We start from the premise that any self-organising system complies with the free energy principle, in virtue of placing an upper bound on its entropy. Crucially, the free energy principle allows one to interpret biological systems as inferring the state of their environment or external milieu. An emergent property of this inference is synchronisation among an ensemble of systems that infer each other. Here, we investigate the implications of neuronal dynamics by simulating neuronal networks, where each neuron minimises its free energy. We cast the ensuing ensemble dynamics in terms of inference and show that cardinal behaviours of neuronal networks - both in vivo and in vitro - can be explained by this framework. In particular, we test the hypotheses that (i) generalised synchrony is an emergent property of free energy minimisation; thereby explaining synchronisation in the resting brain: (ii) desynchronisation is induced by exogenous input; thereby explaining event-related desynchronisation and (iii) structure learning emerges in response to causal structure in exogenous input; thereby explaining functional segregation in real neuronal systems.

Parameters as Trait Indicators: Exploring a Complementary Neurocomputational Approach to Conceptualizing and Measuring Trait Differences in Emotional Intelligence

Current assessments of trait emotional intelligence (EI) rely on self-report inventories. While this approach has seen considerable success, a complementary approach allowing objective assessment of EI-relevant traits would provide some potential advantages. Among others, one potential advantage is that it would aid in emerging efforts to assess the brain basis of trait EI, where self-reported competency levels do not always match real-world behavior. In this paper, we review recent experimental paradigms in computational cognitive neuroscience (CCN), which allow behavioral estimates of individual differences in range of parameter values within computational models of neurocognitive processes. Based on this review, we illustrate how several of these parameters appear to correspond well to EI-relevant traits (i.e., differences in mood stability, stress vulnerability, self-control, and flexibility, among others). In contrast, although estimated objectively, these parameters do not correspond well to the optimal performance abilities assessed within competing “ability models” of EI. We suggest that adapting this approach from CCN—by treating parameter value estimates as objective trait EI measures—could (1) provide novel research directions, (2) aid in characterizing the neural basis of trait EI, and (3) offer a promising complementary assessment method.

The value of uncertainty: An active inference perspective

We discuss how uncertainty underwrites exploration and epistemic foraging from the perspective of active inference: a generic scheme that places pragmatic (utility maximization) and epistemic (uncertainty minimization) imperatives on an equal footing - as primary determinants of proximal behavior. This formulation contextualizes the complementary motivational incentives for reward-related stimuli and environmental uncertainty, offering a normative treatment of their trade-off.

Making the Environment an Informative Place: A Conceptual Analysis of Epistemic Policies and Sensorimotor Coordination

How do living organisms decide and act with limited and uncertain information? Here, we discuss two computational approaches to solving these challenging problems: a "cognitive" and a "sensorimotor" enrichment of stimuli, respectively. In both approaches, the key notion is that agents can strategically modulate their behavior in informative ways, e.g., to disambiguate amongst alternative hypotheses or to favor the perception of stimuli providing the information necessary to later act appropriately. We discuss how, despite their differences, both approaches appeal to the notion that actions must obey both epistemic (i.e., information-gathering or uncertainty-reducing) and pragmatic (i.e., goal- or reward-maximizing) imperatives and balance them. Our computationally-guided analysis reveals that epistemic behavior is fundamental to understanding several facets of cognitive processing, including perception, decision making, and social interaction.

Regimes of Expectations: An Active Inference Model of Social Conformity and Human Decision Making

How do humans come to acquire shared expectations about how they ought to behave in distinct normalized social settings? This paper offers a normative framework to answer this question. We introduce the computational construct of 'deontic value' - based on active inference and Markov decision processes - to formalize conceptions of social conformity and human decision-making. Deontic value is an attribute of choices, behaviors, or action sequences that inherit directly from deontic cues in our econiche (e.g., red traffic lights); namely, cues that denote an obligatory social rule. Crucially, the prosocial aspect of deontic value rests upon a particular form of circular causality: deontic cues exist in the environment in virtue of the environment being modified by repeated actions, while action itself is contingent upon the deontic value of environmental cues. We argue that this construction of deontic cues enables the epistemic (i.e., information-seeking) and pragmatic (i.e., goal- seeking) values of any behavior to be 'cached' or 'outsourced' to the environment, where the environment effectively 'learns' about the behavior of its denizens. We describe the process whereby this particular aspect of value enables learning of habitual behavior over neurodevelopmental and transgenerational timescales.

Simulating Active Inference Processes by Message Passing

The free energy principle (FEP) offers a variational calculus-based description for how biological agents persevere through interactions with their environment. Active inference (AI) is a corollary of the FEP, which states that biological agents act to fulfill prior beliefs about preferred future observations (target priors). Purposeful behavior then results from variational free energy minimization with respect to a generative model of the environment with included target priors. However, manual derivations for free energy minimizing algorithms on custom dynamic models can become tedious and error-prone. While probabilistic programming (PP) techniques enable automatic derivation of inference algorithms on free-form models, full automation of AI requires specialized tools for inference on dynamic models, together with the description of an experimental protocol that governs the interaction between the agent and its simulated environment. The contributions of the present paper are two-fold. Firstly, we illustrate how AI can be automated with the use of ForneyLab, a recent PP toolbox that specializes in variational inference on flexibly definable dynamic models. More specifically, we describe AI agents in a dynamic environment as probabilistic state space models (SSM) and perform inference for perception and control in these agents by message passing on a factor graph representation of the SSM. Secondly, we propose a formal experimental protocol for simulated AI. We exemplify how this protocol leads to goal-directed behavior for flexibly definable AI agents in two classical RL examples, namely the Bayesian thermostat and the mountain car parking problems.

Multiscale integration: beyond internalism and externalism

We present a multiscale integrationist interpretation of the boundaries of cognitive systems, using the Markov blanket formalism of the variational free energy principle. This interpretation is intended as a corrective for the philosophical debate over internalist and externalist interpretations of cognitive boundaries; we stake out a compromise position. We first survey key principles of new radical (extended, enactive, embodied) views of cognition. We then describe an internalist interpretation premised on the Markov blanket formalism. Having reviewed these accounts, we develop our positive multiscale account. We argue that the statistical seclusion of internal from external states of the system-entailed by the existence of a Markov boundary-can coexist happily with the multiscale integration of the system through its dynamics. Our approach does not privilege any given boundary (whether it be that of the brain, body, or world), nor does it argue that all boundaries are equally prescient. We argue that the relevant boundaries of cognition depend on the level being characterised and the explanatory interests that guide investigation. We approach the issue of how and where to draw the boundaries of cognitive systems through a multiscale ontology of cognitive systems, which offers a multidisciplinary research heuristic for cognitive science.

Toward a Neurobiologically Plausible Model of Language-Related, Negative Event-Related Potentials

Language-related event-related potential (ERP) components such as the N400 have traditionally been associated with linguistic or cognitive functional interpretations. By contrast, it has been considerably more difficult to relate these components to neurobiologically grounded accounts of language. Here, we propose a theoretical framework based on a predictive coding architecture, within which negative language-related ERP components such as the N400 can be accounted for in a neurobiologically plausible manner. Specifically, we posit that the amplitude of negative language-related ERP components reflects precision-weighted prediction error signals, i.e., prediction errors weighted by the relevance of the information source leading to the error. From this perspective, precision has a direct link to cue validity in a particular language and, thereby, to relevance of individual linguistic features for internal model updating. We view components such as the N400 and LAN as members of a family with similar functional characteristics and suggest that latency and topography differences between these components reflect the locus of prediction errors and model updating within a hierarchically organized cortical predictive coding architecture. This account has the potential to unify findings from the full range of the N400 literature, including word-level, sentence-, and discourse-level results as well as cross-linguistic differences.

Neuronal message passing using Mean-field, Bethe, and Marginal approximations

Neuronal computations rely upon local interactions across synapses. For a neuronal network to perform inference, it must integrate information from locally computed messages that are propagated among elements of that network. We review the form of two popular (Bayesian) message passing schemes and consider their plausibility as descriptions of inference in biological networks. These are variational message passing and belief propagation - each of which is derived from a free energy functional that relies upon different approximations (mean-field and Bethe respectively). We begin with an overview of these schemes and illustrate the form of the messages required to perform inference using Hidden Markov Models as generative models. Throughout, we use factor graphs to show the form of the generative models and of the messages they entail. We consider how these messages might manifest neuronally and simulate the inferences they perform. While variational message passing offers a simple and neuronally plausible architecture, it falls short of the inferential performance of belief propagation. In contrast, belief propagation allows exact computation of marginal posteriors at the expense of the architectural simplicity of variational message passing. As a compromise between these two extremes, we offer a third approach - marginal message passing - that features a simple architecture, while approximating the performance of belief propagation. Finally, we link formal considerations to accounts of neurological and psychiatric syndromes in terms of aberrant message passing.