Computational models of adaptive behavior and prefrontal cortex

Behavior engineering using quantitative reinforcement learning models

Effectively shaping human and animal behavior is of great practical and theoretical importance. Here we ask whether quantitative models of choice can be used to achieve this goal more effectively than qualitative psychological principles. We term this approach, which is motivated by the effectiveness of engineering in the natural sciences, 'choice engineering'. To address this question, we launched an academic competition, in which teams of academic competitors used either quantitative models or qualitative principles to design reward schedules that would maximally bias the choices of experimental participants in a repeated, two-alternative task. We found that a choice engineering approach is the most successful method for shaping behavior in our task. This is a proof of concept that quantitative models are ripe to be used in order to engineer behavior. Finally, we show that choice engineering can be effectively used to compare models in the cognitive sciences, thus providing an alternative to the standard statistical methods of model comparison that are based on likelihood or explained variance.

Theory Change in Cognitive Neurobiology: The Case of the Orbitofrontal Cortex

How do theories of the functions of parts of the brain change? I argue that computational hypotheses help explain the nature of theorizing in cognitive neurobiology. I will focus on the orbitofrontal cortex (OFC), a frontal region of the brain implicated in an array of cognitive functions. Different theories of OFC state different principles of OFC function and use different concepts to construct those principles. There are also differences in the patterns of use of evidence across different theories. I briefly survey several extant proposals for understanding theory change in science generally and cognitive neuroscience specifically, including paradigm shifts, tool innovation, mechanism discovery, conceptual innovation, exploratory experimentation, and changes in measurement techniques. While these extant approaches fall short at describing the nature of theory change illustrated by the case of OFC, they are compatible with my proposal that these theoretical changes and differences in the use of evidence result from different computational hypotheses about the region.

Development of a neural efficiency metric to assess human-exoskeleton adaptations

Passive exoskeletons have been introduced to alleviate loading on the lumbar spine while increasing the wearer's productivity. However, few studies have examined the neurocognitive effects of short-term human-exoskeleton adaptation. The objective of the study was to develop a novel neural efficiency metric to assess short-term human exoskeleton adaptation during repetitive lifting. Twelve participants (gender-balanced) performed simulated asymmetric lifting tasks for a short duration (phase: early, middle, late) with and without a passive low back exoskeleton on two separate days. Phase, exoskeleton condition, and their interaction effects on biomechanical parameters, neural activation, and the novel neural efficiency metric were examined. Peak L5/S1 superior lateral shear forces were found to be significantly lower in the exoskeleton condition than in the control condition. However, other biomechanical and neural activation measures were comparable between conditions. The temporal change of the neural efficiency metric was found to follow the motor adaptation process. Compared to the control condition, participants exhibited lower efficiency during the exoskeleton-assisted lifting condition over time. The neural efficiency metric was capable of tracking the short-term task adaptation process during a highly ambulatory exoskeleton-assisted manual handling task. The exoskeleton-assisted task was less efficient and demanded a longer adaptation period than the control condition, which may impact exoskeleton acceptance and/or intent to use.

Acetylcholine modulates prefrontal outcome coding during threat learning under uncertainty

Outcomes can vary even when choices are repeated. Such ambiguity necessitates adjusting how much to learn from each outcome by tracking its variability. The medial prefrontal cortex (mPFC) has been reported to signal the expected outcome and its discrepancy from the actual outcome (prediction error), two variables essential for controlling the learning rate. However, the source of signals that shape these coding properties remains unknown. Here, we investigated the contribution of cholinergic projections from the basal forebrain because they carry precisely timed signals about outcomes. One-photon calcium imaging revealed that as mice learned different probabilities of threat occurrence on two paths, some mPFC cells responded to threats on one of the paths, while other cells gained responses to threat omission. These threat- and omission-evoked responses were scaled to the unexpectedness of outcomes, some exhibiting a reversal in response direction when encountering surprising threats as opposed to surprising omissions. This selectivity for signed prediction errors was enhanced by optogenetic stimulation of local cholinergic terminals during threats. The enhanced threat-evoked cholinergic signals also made mice erroneously abandon the correct choice after a single threat that violated expectations, thereby decoupling their path choice from the history of threat occurrence on each path. Thus, acetylcholine modulates the encoding of surprising outcomes in the mPFC to control how much they dictate future decisions.

Dorsolateral prefrontal cortex drives strategic aborting by optimizing long-run policy extraction

Real world choices often involve balancing decisions that are optimized for the short-vs. long-term. Here, we reason that apparently sub-optimal single trial decisions in macaques may in fact reflect long-term, strategic planning. We demonstrate that macaques freely navigating in VR for sequentially presented targets will strategically abort offers, forgoing more immediate rewards on individual trials to maximize session-long returns. This behavior is highly specific to the individual, demonstrating that macaques reason about their own long-run performance. Reinforcement-learning (RL) models suggest this behavior is algorithmically supported by modular actor-critic networks with a policy module not only optimizing long-term value functions, but also informed of specific state-action values allowing for rapid policy optimization. The behavior of artificial networks suggests that changes in policy for a matched offer ought to be evident as soon as offers are made, even if the aborting behavior occurs much later. We confirm this prediction by demonstrating that single units and population dynamics in macaque dorsolateral prefrontal cortex (dlPFC), but not parietal area 7a or dorsomedial superior temporal area (MSTd), reflect the upcoming reward-maximizing aborting behavior upon offer presentation. These results cast dlPFC as a specialized policy module, and stand in contrast to recent work demonstrating the distributed and recurrent nature of belief-networks.

Orbitofrontal and Prelimbic Cortices Serve Complementary Roles in Adapting Reward Seeking to Learned Anxiety

BACKGROUND

Anxiety is a common symptom of several mental health disorders and adversely affects motivated behaviors. Anxiety can emerge from associating risk of future harm while engaged in goal-guided actions. Using a recently developed behavioral paradigm to model this aspect of anxiety, we investigated the role of 2 cortical subregions, the prelimbic medial frontal cortex (PL) and lateral orbitofrontal cortex (lOFC), which have been implicated in anxiety and outcome expectation, in flexible representation of actions associated with harm risk.

METHODS

A seek-take reward-guided instrumental task design was used to train animals (N = 8) to associate the seek action with a variable risk of punishment. After learning, animals underwent extinction training for this association. Fiber photometry was used to measure and compare neuronal activity in the PL and lOFC during learning and extinction.

RESULTS

Animals increased action suppression in response to punishment contingencies. This increase dissipated after extinction training. These behavioral changes were associated with region-specific changes in neuronal activity. PL neuronal activity preferentially adapted to the threat of punishment, whereas lOFC activity adapted to safe aspects of the task. Moreover, correlated activity between these regions was suppressed during actions associated with harm risk, suggesting that these regions may guide behavior independently under anxiety.

CONCLUSIONS

These findings suggest that the PL and lOFC serve distinct but complementary roles in the representation of learned anxiety. This dissociation may provide a mechanism to explain how overlapping cortical systems are implicated in reward-guided action execution during anxiety.

Lateral prefrontal cortex controls interplay between working memory and actions

Humans must often keep multiple task goals in mind, at different levels of priority and immediacy, while also interacting with the environment. We might need to remember information for an upcoming task while engaged in more immediate actions. Consequently, actively maintained working memory (WM) content may bleed into ongoing but unrelated motor behavior. Here, we experimentally test the impact of WM maintenance on action execution, and we transcranially stimulate lateral prefrontal cortex (PFC) to parse its functional contributions to WM-motor interactions. We first created a task scenario wherein human participants (both sexes) executed cued hand movements during WM maintenance. We manipulated the compatibility between WM and movement goals at the trial level and the statistical likelihood that the two would be compatible at the block level. We found that remembering directional words (e.g., 'left', 'down') biased the trajectory and speed of hand movements that occurred during the WM delay, but the bias was dampened in blocks when WM content predictably conflicted with movement goals. Then we targeted left lateral PFC with two different transcranial magnetic stimulation (TMS) protocols before participants completed the task. We found that an intermittent theta-burst protocol, which is thought to be excitatory, dampened sensitivity to block-level control demands (i.e., proactive control), while a continuous theta-burst protocol, which is thought to be inhibitory, dampened adaptation to trial-by-trial conflict (i.e., reactive control). Therefore, lateral PFC is involved in controlling the interplay between WM content and manual action, but different PFC mechanisms may support different time-scales of adaptive control.

Contribution of amygdala to dynamic model arbitration under uncertainty

Intrinsic uncertainty in the reward environment requires the brain to run multiple models simultaneously to predict outcomes based on preceding cues or actions, commonly referred to as stimulus- and action-based learning. Ultimately, the brain also must adopt appropriate choice behavior using reliability of these models. Here, we combined multiple experimental and computational approaches to quantify concurrent learning in monkeys performing tasks with different levels of uncertainty about the model of the environment. By comparing behavior in control monkeys and monkeys with bilateral lesions to the amygdala or ventral striatum, we found evidence for dynamic, competitive interaction between stimulus-based and action-based learning, and for a distinct role of the amygdala. Specifically, we demonstrate that the amygdala adjusts the initial balance between the two learning systems, thereby altering the interaction between arbitration and learning that shapes the time course of both learning and choice behaviors. This novel role of the amygdala can account for existing contradictory observations and provides testable predictions for future studies into circuit-level mechanisms of flexible learning and choice under uncertainty.

Effects of Phasic Activation of Locus Ceruleus on Cortical Neural Activity and Auditory Discrimination Behavior

Although the locus ceruleus (LC) is recognized as a crucial modulator for attention and perception by releasing norepinephrine into various cortical regions, the impact of LC-noradrenergic (LC-NE) modulation on auditory discrimination behavior remains elusive. In this study, we firstly recorded local field potential and single-unit activity in multiple cortical regions associated with auditory-motor processing, including the auditory cortex, posterior parietal cortex, secondary motor cortex, anterior cingulate cortex, prefrontal cortex, and orbitofrontal cortex (OFC), in response to optogenetic activation (40 Hz and 0.5 s) of the LC-NE neurons in awake mice (male). We found that phasic LC stimulation induced a persistent high gamma oscillation (50-80 Hz) in the OFC. Phasic activation of LC-NE neurons also resulted in a corresponding increase in norepinephrine levels in the OFC, accompanied by a pupillary dilation response. Furthermore, when mice were performing a go/no-go auditory discrimination task, we optogeneticaly activated the neural projections from LC to OFC and revealed a shortened latency in behavioral responses to sound stimuli and an increased false alarm rate. These impulsive behavioral responses may be associated with the gamma neural activity in the OFC. These findings have broadened our understanding of the neural mechanisms involved in the role of LC in auditory-motor processing.

Timescales of learning in prefrontal cortex

The lateral prefrontal cortex (PFC) in humans and other primates is critical for immediate, goal-directed behaviour and working memory, which are classically considered distinct from the cognitive and neural circuits that support long-term learning and memory. Over the past few years, a reconsideration of this textbook perspective has emerged, in that different timescales of memory-guided behaviour are in constant interaction during the pursuit of immediate goals. Here, we will first detail how neural activity related to the shortest timescales of goal-directed behaviour (which requires maintenance of current states and goals in working memory) is sculpted by long-term knowledge and learning - that is, how the past informs present behaviour. Then, we will outline how learning across different timescales (from seconds to years) drives plasticity in the primate lateral PFC, from single neuron firing rates to mesoscale neuroimaging activity patterns. Finally, we will review how, over days and months of learning, dense local and long-range connectivity patterns in PFC facilitate longer-lasting changes in population activity by changing synaptic weights and recruiting additional neural resources to inform future behaviour. Our Review sheds light on how the machinery of plasticity in PFC circuits facilitates the integration of learned experiences across time to best guide adaptive behaviour.

Bridging flexible goal-directed cognition and consciousness: The Goal-Aligning Representation Internal Manipulation theory

Goal-directed manipulation of internal representations is a key element of human flexible behaviour, while consciousness is commonly associated with higher-order cognition and human flexibility. Current perspectives have only partially linked these processes, thus preventing a clear understanding of how they jointly generate flexible cognition and behaviour. Moreover, these limitations prevent an effective exploitation of this knowledge for technological scopes. We propose a new theoretical perspective that extends our 'three-component theory of flexible cognition' toward higher-order cognition and consciousness, based on the systematic integration of key concepts from Cognitive Neuroscience and AI/Robotics. The theory proposes that the function of conscious processes is to support the alignment of representations with multi-level goals. This higher alignment leads to more flexible and effective behaviours. We analyse here our previous model of goal-directed flexible cognition (validated with more than 20 human populations) as a starting GARIM-inspired model. By bridging the main theories of consciousness and goal-directed behaviour, the theory has relevant implications for scientific and technological fields. In particular, it contributes to developing new experimental tasks and interpreting clinical evidence. Finally, it indicates directions for improving machine learning and robotics systems and for informing real-world applications (e.g., in digital-twin healthcare and roboethics).

The neurobiology of aesthetic chills: How bodily sensations shape emotional experiences

The phenomenon of aesthetic chills-shivers and goosebumps associated with either rewarding or threatening stimuli-offers a unique window into the brain basis of conscious reward because of their universal nature and simultaneous subjective and physical counterparts. Elucidating the neural mechanisms underlying aesthetic chills can reveal fundamental insights about emotion, consciousness, and the embodied mind. What is the precise timing and mechanism of bodily feedback in emotional experience? How are conscious feelings and motivations generated from interoceptive predictions? What is the role of uncertainty and precision signaling in shaping emotions? How does the brain distinguish and balance processing of rewards versus threats? We review neuroimaging evidence and highlight key questions for understanding how bodily sensations shape conscious feelings. This research stands to advance models of brain-body interactions shaping affect and may lead to novel nonpharmacological interventions for disorders of motivation and pleasure.

Evolution of a central dopamine circuit underlies adaptation of light-evoked sensorimotor response in the blind cavefish, Astyanax mexicanus

Adaptive behaviors emerge in novel environments through functional changes in neural circuits. While relationships between circuit function and behavior have been well studied, how evolution shapes those circuits and leads to behavioral adpation is poorly understood. The Mexican cavefish, Astyanax mexicanus, provides a unique genetically amendable model system, equipped with above ground eyed surface fish and multiple evolutionarily divergent populations of blind cavefish that have evolved in complete darkness. These differences in environment and vision provide an opprotunity to examine how a neural circuit is functionally influenced by the presence of light. Here, we examine differences in the detection, and behavioral response induced by non visual light reception. Both populations exhibit photokinetic behavior, with surface fish becoming hyperactive following sudden darkness and cavefish becoming hyperactive following sudden illumination. To define these photokinetic neural circuits, we integrated whole brain functional imaging with our Astyanax brain atlas for surface and cavefish responding to light changes. We identified the caudal posterior tuberculum as the central modulator for both light or dark stimulated photokinesis. To unconver how spatiotemporal neuronal activity differed between surface fish and cavefish, we used stable pan-neuronal GCaMP Astyanax transgenics to show that a subpopulation of darkness sensitve neurons in surface fish are now light senstive in cavefish. Further functional analysis revealed that this integrative switch is dependent on dopmane signaling, suggesting a key role for dopamine and a highly conserved dopamine circuit in modulating the evolution of a circuit driving an essential behavior. Together, these data shed light into how neural circuits evolved to adapte to novel settings, and reveal the power of Astyanax as a model to elucidate mechanistic ingiths underlying sensory adaptation.

Topological features of functional brain networks and subclinical impulsivity: an investigation in younger and older adults

Impulsive traits (i.e., the tendency to act without forethought regardless of negative outcomes) are frequently found in healthy populations. When exposed to risk factors, individuals may develop debilitating disorders of impulse control (addiction, substance abuse, gambling) characterized by behavioral and cognitive deficits, eventually leading to huge socioeconomic costs. With the far-reaching aim of preventing the onset of impulsive disorders, it is relevant to investigate the topological organization of functional brain networks associated with impulsivity in sub-clinical populations. Taking advantage of the open-source LEMON dataset, we investigated the topological features of resting-state functional brain networks associated with impulsivity in younger (n = 146, age: 20-35) and older (n = 61, age: 59-77) individuals, using a graph-theoretical approach. Specifically, we computed indices of segregation and integration at the level of specific circuits and nodes known to be involved in impulsivity (frontal, limbic, and striatal networks). In younger individuals, results revealed that impulsivity was associated with a more widespread, less clustered and less efficient functional organization, at all levels of analyses and in all selected networks. Conversely, impulsivity in older individuals was associated with reduced integration and increased segregation of striatal regions. Speculatively, such alterations of functional brain networks might underlie behavioral and cognitive abnormalities associated with impulsivity, a working hypothesis worth being tested in future research. Lastly, differences between younger and older individuals might reflect the implementation of age-specific adaptive strategies, possibly accounting for observed differences in behavioral manifestations. Potential interpretations, limitations and implications are discussed.

Motor Adaptation Deficits in Children with Developmental Coordination Disorder and/or Reading Disorder

Procedural learning has been mainly tested through motor sequence learning tasks in children with neurodevelopmental disorders, especially with isolated Developmental Coordination Disorder (DCD) and Reading Disorder (RD). Studies on motor adaptation are scarcer and more controversial. This study aimed to compare the performance of children with isolated and associated DCD and RD in a graphomotor adaptation task. In total, 23 children with RD, 16 children with DCD, 19 children with DCD-RD, and 21 typically developing (TD) children wrote trigrams both in the conventional (from left to right) and opposite (from right to left) writing directions. The results show that movement speed and accuracy were more impacted by the adaptation condition (opposite writing direction) in children with neurodevelopmental disorders than TD children. Our results also reveal that children with RD have less difficulty adapting their movement than children with DCD. Children with DCD-RD had the most difficulty, and analysis of their performance suggests a cumulative effect of the two neurodevelopmental disorders in motor adaptation.

A computationally informed distinction of interoception and exteroception

While interoception is of major neuroscientific interest, its precise definition and delineation from exteroception continue to be debated. Here, we propose a functional distinction between interoception and exteroception based on computational concepts of sensor-effector loops. Under this view, the classification of sensory inputs as serving interoception or exteroception depends on the sensor-effector loop they feed into, for the control of either bodily (physiological and biochemical) or environmental states. We explain the utility of this perspective by examining the perception of skin temperature, one of the most challenging cases for distinguishing between interoception and exteroception. Specifically, we propose conceptualising thermoception as inference about the thermal state of the body (including the skin), which is directly coupled to thermoregulatory processes. This functional view emphasises the coupling to regulation (control) as a defining property of perception (inference) and connects the definition of interoception to contemporary computational theories of brain-body interactions.

Proactive Versus Reactive Control Strategies Differentially Mediate Alcohol Drinking in Male Wistars and P Rats

Problematic alcohol consumption is associated with deficits in decision-making and alterations in prefrontal cortex neural activity likely contribute. We hypothesized that the differences in cognitive control would be evident between male Wistars and a model of genetic risk: alcohol-preferring P rats. Cognitive control is split into proactive and reactive components. Proactive control maintains goal-directed behavior independent of a stimulus, whereas reactive control elicits goal-directed behavior at the time of a stimulus. We hypothesized that Wistars would show proactive control over alcohol seeking whereas P rats would show reactive control over alcohol seeking. Neural activity was recorded from the prefrontal cortex during an alcohol seeking task with two session types. On congruent sessions, the conditioned stimulus (CS+) was on the same side as alcohol access. Incongruent sessions presented alcohol opposite the CS+. Wistars, but not P rats, made more incorrect approaches during incongruent sessions, suggesting that Wistars utilized the previously learned rule. This motivated the hypothesis that neural activity reflecting proactive control would be observable in Wistars but not P rats. While P rats showed differences in neural activity at times of alcohol access, Wistars showed differences prior to approaching the sipper. These results support our hypothesis that Wistars are more likely to engage in proactive cognitive control strategies whereas P rats are more likely to engage in reactive cognitive control strategies. Although P rats were bred to prefer alcohol, the differences in cognitive control may reflect a sequela of behaviors that mirror those in humans at risk for an AUD.

Controlling human causal inference through in silico task design

Learning causal relationships is crucial for survival. The human brain's functional flexibility allows for effective causal inference, underlying various learning processes. While past studies focused on environmental factors influencing causal inference, a fundamental question remains: can these factors be manipulated for strategic causal inference control? This paper presents a task control framework for orchestrating causal learning task design. It utilizes a two-player game setting where a neural network learns to manipulate task variables by interacting with a human causal inference model. Training the task controller to generate experimental designs, we confirm its ability to accommodate complexities of environmental causal structure. Experiments involving 126 human subjects successfully validate the impact of task control on performance and learning efficiency. Additionally, we find that task control policy reflects the intrinsic nature of human causal inference: one-shot learning. This framework holds promising potential for applications paving the way for targeted behavioral outcomes in humans.

Dissociable Contributions of Basolateral Amygdala and Ventrolateral Orbitofrontal Cortex to Flexible Learning Under Uncertainty

Reversal learning measures the ability to form flexible associations between choice outcomes with stimuli and actions that precede them. This type of learning is thought to rely on several cortical and subcortical areas, including the highly interconnected orbitofrontal cortex (OFC) and basolateral amygdala (BLA), and is often impaired in various neuropsychiatric and substance use disorders. However, the unique contributions of these regions to stimulus- and action-based reversal learning have not been systematically compared using a chemogenetic approach particularly before and after the first reversal that introduces new uncertainty. Here, we examined the roles of ventrolateral OFC (vlOFC) and BLA during reversal learning. Male and female rats were prepared with inhibitory designer receptors exclusively activated by designer drugs targeting projection neurons in these regions and tested on a series of deterministic and probabilistic reversals during which they learned about stimulus identity or side (left or right) associated with different reward probabilities. Using a counterbalanced within-subject design, we inhibited these regions prior to reversal sessions. We assessed initial and pre-/post-reversal changes in performance to measure learning and adjustments to reversals, respectively. We found that inhibition of the ventrolateral orbitofrontal cortex (vlOFC), but not BLA, eliminated adjustments to stimulus-based reversals. Inhibition of BLA, but not vlOFC, selectively impaired action-based probabilistic reversal learning, leaving deterministic reversal learning intact. vlOFC exhibited a sex-dependent role in early adjustment to action-based reversals, but not in overall learning. These results reveal dissociable roles for BLA and vlOFC in flexible learning and highlight a more crucial role for BLA in learning meaningful changes in the reward environment.

The cognitive (lateral) hypothalamus

Despite the physiological complexity of the hypothalamus, its role is typically restricted to initiation or cessation of innate behaviors. For example, theories of lateral hypothalamus argue that it is a switch to turn feeding 'on' and 'off' as dictated by higher-order structures that render when feeding is appropriate. However, recent data demonstrate that the lateral hypothalamus is critical for learning about food-related cues. Furthermore, the lateral hypothalamus opposes learning about information that is neutral or distal to food. This reveals the lateral hypothalamus as a unique arbitrator of learning capable of shifting behavior toward or away from important events. This has relevance for disorders characterized by changes in this balance, including addiction and schizophrenia. Generally, this suggests that hypothalamic function is more complex than increasing or decreasing innate behaviors.

Metabolic and functional substrates of impulsive decision-making in individuals with heroin addiction after prolonged methadone maintenance treatment

Elevated impulsivity has been frequently reported in individuals with opioid addiction receiving methadone maintenance therapy (MMT), but the underlying neural mechanisms and cognitive subprocesses are not fully understood. We acquired functional magnetic resonance imaging (fMRI) data from 37 subjects with heroin addiction receiving long-term MMT and 33 healthy controls who performed a probabilistic reversal learning task, and measured their resting-state brain glucose using fluorine-18-fluorodeoxyglucose positron emission tomography (18F-FDG PET). Subjects receiving MMT exhibited significantly elevated self-reported impulsivity, and computational modeling revealed a marked impulsive decision bias manifested as switching more frequently without available evidence. Moreover, this impulsive decision bias was associated with the dose and duration of methadone use, irrelevant to the duration of heroin use. During the task, the switch-related hypoactivation in the left rostral middle frontal gyrus was correlated with the impulsive decision bias while the function of reward sensitivity was intact in subjects receiving MMT. Using prior brain-wide receptor density data, we found that the highest variance of regional metabolic abnormalities was explained by the spatial distribution of μ-opioid receptors among 10 types of neurotransmitter receptors. Heightened impulsivity in individuals receiving prolonged MMT is manifested as atypical choice bias and noise in decision-making processes, which is further driven by deficits in top-down cognitive control, other than reward sensitivity. Our findings uncover multifaceted mechanisms underlying elevated impulsivity in subjects receiving MMT, which might provide insights for developing complementary therapies to improve retention during MMT.

The Willpower Paradox: Possible and Impossible Conceptions of Self-Control

Self-control denotes the ability to override current desires to render behavior consistent with long-term goals. A key assumption is that self-control is required when short-term desires are transiently stronger (more preferred) than long-term goals and people would yield to temptation without exerting self-control. We argue that this widely shared conception of self-control raises a fundamental yet rarely discussed conceptual paradox: How is it possible that a person most strongly desires to perform a behavior (e.g., eat chocolate) and at the same time desires to recruit self-control to prevent themselves from doing it? A detailed analysis reveals that three common assumptions about self-control cannot be true simultaneously. To avoid the paradox, any coherent theory of self-control must abandon either the assumption (a) that recruitment of self-control is an intentional process, or (b) that humans are unitary agents, or (c) that self-control consists in overriding the currently strongest desire. We propose a taxonomy of different kinds of self-control processes that helps organize current theories according to which of these assumptions they abandon. We conclude by outlining unresolved questions and future research perspectives raised by different conceptions of self-control and discuss implications for the question of whether self-control can be considered rational.

Dissociable contributions of basolateral amygdala and ventrolateral orbitofrontal cortex to flexible learning under uncertainty

Reversal learning measures the ability to form flexible associations between choice outcomes with stimuli and actions that precede them. This type of learning is thought to rely on several cortical and subcortical areas, including highly interconnected orbitofrontal cortex (OFC) and basolateral amygdala (BLA), and is often impaired in various neuropsychiatric and substance use disorders. However, unique contributions of these regions to stimulus- and action-based reversal learning have not been systematically compared using a chemogenetic approach and particularly before and after the first reversal that introduces new uncertainty. Here, we examined the roles of ventrolateral OFC (vlOFC) and BLA during reversal learning. Male and female rats were prepared with inhibitory DREADDs targeting projection neurons in these regions and tested on a series of deterministic and probabilistic reversals during which they learned about stimulus identity or side (left or right) associated with different reward probabilities. Using a counterbalanced within-subject design, we inhibited these regions prior to reversal sessions. We assessed initial and pre-post reversal changes in performance to measure learning and adjustments to reversals, respectively. We found that inhibition of vlOFC, but not BLA, eliminated adjustments to stimulus-based reversals. Inhibition of BLA, but not vlOFC, selectively impaired action-based probabilistic reversal learning, leaving deterministic reversal learning intact. vlOFC exhibited a sex-dependent role in early adjustment to action-based reversals, but not in overall learning. These results reveal dissociable roles for BLA and vlOFC in flexible learning and highlight a more crucial role for BLA in learning meaningful changes in the reward environment.

Emotion regulation from an action-control perspective

Despite increasing interest in emotional processes in cognitive science, theories on emotion regulation have remained rather isolated, predominantly focused on cognitive regulation strategies such as reappraisal. However, recent neurocognitive evidence suggests that early emotion regulation may involve sensorimotor control in addition to other emotion-regulation processes. We propose an action-oriented view of emotion regulation, in which feedforward predictions develop from action-selection mechanisms. Those can account for acute emotional-action control as well as more abstract instances of emotion regulation such as cognitive reappraisal. We argue the latter occurs in absence of overt motor output, yet in the presence of full-blown autonomic, visceral, and subjective changes. This provides an integrated framework with testable neuro-computational predictions and concrete starting points for intervention to improve emotion control in affective disorders.

Neural representations for multi-context visuomotor adaptation and the impact of common representation on multi-task performance: a multivariate decoding approach

The human brain's remarkable motor adaptability stems from the formation of context representations and the use of a common context representation (e.g., an invariant task structure across task contexts) derived from structural learning. However, direct evaluation of context representations and structural learning in sensorimotor tasks remains limited. This study aimed to rigorously distinguish neural representations of visual, movement, and context levels crucial for multi-context visuomotor adaptation and investigate the association between representation commonality across task contexts and adaptation performance using multivariate decoding analysis with fMRI data. Here, we focused on three distinct task contexts, two of which share a rotation structure (i.e., visuomotor rotation contexts with -90° and +90° rotations, in which the mouse cursor's movement was rotated 90 degrees counterclockwise and clockwise relative to the hand-movement direction, respectively) and the remaining one does not (i.e., mirror-reversal context where the horizontal movement of the computer mouse was inverted). This study found that visual representations (i.e., visual direction) were decoded in the occipital area, while movement representations (i.e., hand-movement direction) were decoded across various visuomotor-related regions. These findings are consistent with prior research and the widely recognized roles of those areas. Task-context representations (i.e., either -90° rotation, +90° rotation, or mirror-reversal) were also distinguishable in various brain regions. Notably, these regions largely overlapped with those encoding visual and movement representations. This overlap suggests a potential intricate dependency of encoding visual and movement directions on the context information. Moreover, we discovered that higher task performance is associated with task-context representation commonality, as evidenced by negative correlations between task performance and task-context-decoding accuracy in various brain regions, potentially supporting structural learning. Importantly, despite limited similarities between tasks (e.g., rotation and mirror-reversal contexts), such association was still observed, suggesting an efficient mechanism in the brain that extracts commonalities from different task contexts (such as visuomotor rotations or mirror-reversal) at multiple structural levels, from high-level abstractions to lower-level details. In summary, while illuminating the intricate interplay between visuomotor processing and context information, our study highlights the efficiency of learning mechanisms, thereby paving the way for future exploration of the brain's versatile motor ability.

The Computational and Neural Bases of Context-Dependent Learning

Flexible behavior requires the creation, updating, and expression of memories to depend on context. While the neural underpinnings of each of these processes have been intensively studied, recent advances in computational modeling revealed a key challenge in context-dependent learning that had been largely ignored previously: Under naturalistic conditions, context is typically uncertain, necessitating contextual inference. We review a theoretical approach to formalizing context-dependent learning in the face of contextual uncertainty and the core computations it requires. We show how this approach begins to organize a large body of disparate experimental observations, from multiple levels of brain organization (including circuits, systems, and behavior) and multiple brain regions (most prominently the prefrontal cortex, the hippocampus, and motor cortices), into a coherent framework. We argue that contextual inference may also be key to understanding continual learning in the brain. This theory-driven perspective places contextual inference as a core component of learning.

Proactive Versus Reactive Control Strategies Differentially Mediate Alcohol Drinking in Wistar and P rats

Problematic alcohol consumption is associated with deficits in decision-making, and alterations in prefrontal cortex neural activity likely contributes. We hypothesized that differences in cognitive control would be evident between male Wistar rats and a model for genetic risk for alcohol use disorder (alcohol-preferring P rats). Cognitive control can be split into proactive and reactive components. Proactive control maintains goal-directed behavior independent of a stimulus whereas reactive control elicits goal-directed behavior at the time of a stimulus. We hypothesized that Wistars would show proactive control over alcohol-seeking whereas P rats would show reactive control over alcohol-seeking. Neural ensembles were recorded from prefrontal cortex during an alcohol seeking task that utilized two session types. On congruent sessions the CS+ was on the same side as alcohol access. Incongruent sessions presented alcohol opposite the CS+. Wistars, but not P rats, exhibited an increase in incorrect approaches during incongruent sessions, suggesting that Wistars utilized the previously learned task-rule. This motivated the hypothesis that ensemble activity reflecting proactive control would be observable in Wistars but not P rats. While P rats showed differences in neural activity at times relevant for alcohol delivery, Wistars showed differences prior to approaching the sipper. These results support our hypothesis that Wistars are more likely to engage proactive cognitive-control strategies whereas P rats are more likely to engage reactive cognitive control strategies. Although P rats were bred to prefer alcohol, differences in cognitive control may reflect a sequela of behaviors that mirror those in humans at risk for an AUD.

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

Neurocomputational mechanisms underlying fear-biased adaptation learning in changing environments

Humans are able to adapt to the fast-changing world by estimating statistical regularities of the environment. Although fear can profoundly impact adaptive behaviors, the computational and neural mechanisms underlying this phenomenon remain elusive. Here, we conducted a behavioral experiment (n = 21) and a functional magnetic resonance imaging experiment (n = 37) with a novel cue-biased adaptation learning task, during which we simultaneously manipulated emotional valence (fearful/neutral expressions of the cue) and environmental volatility (frequent/infrequent reversals of reward probabilities). Across 2 experiments, computational modeling consistently revealed a higher learning rate for the environment with frequent versus infrequent reversals following neutral cues. In contrast, this flexible adjustment was absent in the environment with fearful cues, suggesting a suppressive role of fear in adaptation to environmental volatility. This suppressive effect was underpinned by activity of the ventral striatum, hippocampus, and dorsal anterior cingulate cortex (dACC) as well as increased functional connectivity between the dACC and temporal-parietal junction (TPJ) for fear with environmental volatility. Dynamic causal modeling identified that the driving effect was located in the TPJ and was associated with dACC activation, suggesting that the suppression of fear on adaptive behaviors occurs at the early stage of bottom-up processing. These findings provide a neuro-computational account of how fear interferes with adaptation to volatility during dynamic environments.

A rubric for human-like agents and NeuroAI

Researchers across cognitive, neuro- and computer sciences increasingly reference 'human-like' artificial intelligence and 'neuroAI'. However, the scope and use of the terms are often inconsistent. Contributed research ranges widely from mimicking behaviour, to testing machine learning methods as neurally plausible hypotheses at the cellular or functional levels, or solving engineering problems. However, it cannot be assumed nor expected that progress on one of these three goals will automatically translate to progress in others. Here, a simple rubric is proposed to clarify the scope of individual contributions, grounded in their commitments to human-like behaviour, neural plausibility or benchmark/engineering/computer science goals. This is clarified using examples of weak and strong neuroAI and human-like agents, and discussing the generative, corroborate and corrective ways in which the three dimensions interact with one another. The author maintains that future progress in artificial intelligence will need strong interactions across the disciplines, with iterative feedback loops and meticulous validity tests-leading to both known and yet-unknown advances that may span decades to come. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Long-term memory, synaptic plasticity and dopamine in rodent medial prefrontal cortex: Role in executive functions

Mnemonic functions, supporting rodent behavior in complex tasks, include both long-term and (short-term) working memory components. While working memory is thought to rely on persistent activity states in an active neural network, long-term memory and synaptic plasticity contribute to the formation of the underlying synaptic structure, determining the range of possible states. Whereas, the implication of working memory in executive functions, mediated by the prefrontal cortex (PFC) in primates and rodents, has been extensively studied, the contribution of long-term memory component to these tasks received little attention. This review summarizes available experimental data and theoretical work concerning cellular mechanisms of synaptic plasticity in the medial region of rodent PFC and the link between plasticity, memory and behavior in PFC-dependent tasks. A special attention is devoted to unique properties of dopaminergic modulation of prefrontal synaptic plasticity and its contribution to executive functions.

The thalamus in psychosis spectrum disorder

Psychosis spectrum disorder (PSD) affects 1% of the world population and results in a lifetime of chronic disability, causing devastating personal and economic consequences. Developing new treatments for PSD remains a challenge, particularly those that target its core cognitive deficits. A key barrier to progress is the tenuous link between the basic neurobiological understanding of PSD and its clinical phenomenology. In this perspective, we focus on a key opportunity that combines innovations in non-invasive human neuroimaging with basic insights into thalamic regulation of functional cortical connectivity. The thalamus is an evolutionary conserved region that forms forebrain-wide functional loops critical for the transmission of external inputs as well as the construction and update of internal models. We discuss our perspective across four lines of evidence: First, we articulate how PSD symptomatology may arise from a faulty network organization at the macroscopic circuit level with the thalamus playing a central coordinating role. Second, we discuss how recent animal work has mechanistically clarified the properties of thalamic circuits relevant to regulating cortical dynamics and cognitive function more generally. Third, we present human neuroimaging evidence in support of thalamic alterations in PSD, and propose that a similar "thalamocortical dysconnectivity" seen in pharmacological imaging (under ketamine, LSD and THC) in healthy individuals may link this circuit phenotype to the common set of symptoms in idiopathic and drug-induced psychosis. Lastly, we synthesize animal and human work, and lay out a translational path for biomarker and therapeutic development.

Frontal lobe fALFF measured from resting-state fMRI as a prognostic biomarker in first-episode psychosis

Clinical response to antipsychotic drug treatment is highly variable, yet prognostic biomarkers are lacking. The goal of the present study was to test whether the fractional amplitude of low-frequency fluctuations (fALFF), as measured from baseline resting-state fMRI data, can serve as a potential biomarker of treatment response to antipsychotics. Patients in the first episode of psychosis (n = 126) were enrolled in two prospective studies employing second-generation antipsychotics (risperidone or aripiprazole). Patients were scanned at the initiation of treatment on a 3T MRI scanner (Study 1, GE Signa HDx, n = 74; Study 2, Siemens Prisma, n = 52). Voxelwise fALFF derived from baseline resting-state fMRI scans served as the primary measure of interest, providing a hypothesis-free (as opposed to region-of-interest) search for regions of the brain that might be predictive of response. At baseline, patients who would later meet strict criteria for clinical response (defined as two consecutive ratings of much or very much improved on the CGI, as well as a rating of ≤3 on psychosis-related items of the BPRS-A) demonstrated significantly greater baseline fALFF in bilateral orbitofrontal cortex compared to non-responders. Thus, spontaneous activity in orbitofrontal cortex may serve as a prognostic biomarker of antipsychotic treatment.

Medial and orbital frontal cortex in decision-making and flexible behavior

The medial frontal cortex and adjacent orbitofrontal cortex have been the focus of investigations of decision-making, behavioral flexibility, and social behavior. We review studies conducted in humans, macaques, and rodents and argue that several regions with different functional roles can be identified in the dorsal anterior cingulate cortex, perigenual anterior cingulate cortex, anterior medial frontal cortex, ventromedial prefrontal cortex, and medial and lateral parts of the orbitofrontal cortex. There is increasing evidence that the manner in which these areas represent the value of the environment and specific choices is different from subcortical brain regions and more complex than previously thought. Although activity in some regions reflects distributions of reward and opportunities across the environment, in other cases, activity reflects the structural relationships between features of the environment that animals can use to infer what decision to take even if they have not encountered identical opportunities in the past.

Computational mechanisms of distributed value representations and mixed learning strategies

Learning appropriate representations of the reward environment is challenging in the real world where there are many options, each with multiple attributes or features. Despite existence of alternative solutions for this challenge, neural mechanisms underlying emergence and adoption of value representations and learning strategies remain unknown. To address this, we measure learning and choice during a multi-dimensional probabilistic learning task in humans and trained recurrent neural networks (RNNs) to capture our experimental observations. We find that human participants estimate stimulus-outcome associations by learning and combining estimates of reward probabilities associated with the informative feature followed by those of informative conjunctions. Through analyzing representations, connectivity, and lesioning of the RNNs, we demonstrate this mixed learning strategy relies on a distributed neural code and opponency between excitatory and inhibitory neurons through value-dependent disinhibition. Together, our results suggest computational and neural mechanisms underlying emergence of complex learning strategies in naturalistic settings.