Emergence of complex dynamics of choice due to repeated exposures to extinction learning

Extinction context is learned by pigeons, not given by the environment

The saying "context is everything" underscores the importance of interpreting things, be they quotes, events, actions, or stimuli, not in isolation but in the light of a bigger picture - their context. This is evident even in fundamental forms of learning such as extinction learning where, in contextual renewal, an extinguished response reoccurs if the context is changed. But what exactly is context? Is context given by stimuli with inherent properties making them context or, what are the circumstances that allow a stimulus to become "contextual"? Even though the answer may seem intuitively trivial, the literature only provides competing and vague definitions. Using a modified ABA paradigm, we assessed how competing stimuli induced contextual renewal during extinction learning in seven pigeons (Columba livia). Furthermore, we controlled the timing of these stimuli and found it to be crucial; with the right contiguity, even small local stimuli resulted in the strongest contextual renewal. This result challenges definitions of context as 'a backdrop where learning occurs'. Instead, we propose that context can be understood mechanistically as a learned stimulus property. Therefore, context truly is everything and anything.

Global remapping emerges as the mechanism for renewal of context-dependent behavior in a reinforcement learning model

Introduction

The hippocampal formation exhibits complex and context-dependent activity patterns and dynamics, e.g., place cell activity during spatial navigation in rodents or remapping of place fields when the animal switches between contexts. Furthermore, rodents show context-dependent renewal of extinguished behavior. However, the link between context-dependent neural codes and context-dependent renewal is not fully understood.

Methods

We use a deep neural network-based reinforcement learning agent to study the learning dynamics that occur during spatial learning and context switching in a simulated ABA extinction and renewal paradigm in a 3D virtual environment.

Results

Despite its simplicity, the network exhibits a number of features typically found in the CA1 and CA3 regions of the hippocampus. A significant proportion of neurons in deeper layers of the network are tuned to a specific spatial position of the agent in the environment-similar to place cells in the hippocampus. These complex spatial representations and dynamics occur spontaneously in the hidden layer of a deep network during learning. These spatial representations exhibit global remapping when the agent is exposed to a new context. The spatial maps are restored when the agent returns to the previous context, accompanied by renewal of the conditioned behavior. Remapping is facilitated by memory replay of experiences during training.

Discussion

Our results show that integrated codes that jointly represent spatial and task-relevant contextual variables are the mechanism underlying renewal in a simulated DQN agent.

Dynamics and development of interhemispheric conflict solving in pigeons

The dominance of one hemisphere for cognitive operations and decision making may be an efficient mechanism solving interhemispheric conflicts. To understand the ecological significance of the so-called metacontrol, we need better knowledge of its frequency and ontogenetic foundations. Since in pigeons, embryonic light experiences influence degree and direction of interhemispheric specialization and communication, it is conceivable that light affects metacontrol mechanisms. We therefore trained pigeons (Columba livia) with and without embryonic light stimulation in a colour discrimination task. Each eye/hemisphere learnt a different set of colours. After training, hemispheric-specific information was put into conflict and the analysis of conflict decision pattern allowed the identification of hemispheric dominance under binocular and monocular viewing conditions. A majority of pigeons displayed individual metacontrol independent of embryonic light experiences though not in the first test session. Reaction times indicate that interhemispheric mechanisms are critically involved in mediating the dominance of one hemisphere. The impact of interhemispheric components rises with increasing experience and even affects decision making under monocular seeing conditions. Overall results indicate that the hemispheres do not evaluate information independently and that interhemispheric communication in the pigeon brain is much stronger than previously thought and becomes more important with increasing experience.

ABA, AAB and ABC renewal with Pavlovian Conditioning of Tentacle Lowering procedure in the snail Cornu aspersum

This study assesses the recovery of the conditioned response (CR) due to a contextual change (renewal effect) in the Cornu aspersum, using the appetitive Pavlovian Conditioning of Tentacle Lowering procedure. Snails experienced an odorous conditioned stimulus (CS) paired with food (conditioning), followed by the exposition to the CS without any consequence (extinction). Then, they were exposed to the CS in a different context from the extinction one (renewal test). The contexts were three types of illumination. In Experiment 1a, the conditioning was performed in context A, the extinction was conducted in context B and the renewal test was performed in context A. For Experiment 1b, the conditioning and extinction were conducted in context A and renewal was performed in context B. In Experiment 1c, three dissimilar contexts were used for each experimental phase: context A for the conditioning, context B for the extinction and context C for the renewal. In Experiment 2, the renewal magnitude was compared among the three paradigms (ABA, AAB and ABC). Experiments 1a, 1b and 1c showed a recovery of the CR when subjects experienced a contextual change and Experiment 2 showed equivalent levels of renewal in the three paradigms. Learning processes and theories involved are discussed.

Characterisation of the neural basis underlying appetitive extinction & renewal in Cacna1c rats

Recent studies have revealed impairments in Cacna1c ± heterozygous animals (a gene that encodes the Cav 1.2 L-type voltage-gated calcium channels and is implicated in risk for multiple neuropsychiatric disorders) in aversive forms of learning, such as latent inhibition, reversal learning or context discrimination. However, the role of Cav 1.2 L-type voltage-gated calcium channels in extinction of appetitive associations remains under-investigated. Here, we used an appetitive Pavlovian conditioning task and evaluated extinction learning (EL) with a change of context from that of training and test (ABA) and without such a change (AAA) in Cacna1c ± male rats versus their wild-type (WT) littermates. In addition, we used fluorescence in situ hybridization of somatic immediate early genes (IEGs) Arc and Homer1a expression to scrutinize associated changes in the medial prefrontal cortex and the amygdala. Cacna1c ± animals successfully adapt their responses by engaging in appetitive EL and renewal. However, the regional IEG expression profile changed. For the EL occurring in the same context, Cacna1c ± animals presented higher IEG expression in the infralimbic cortex and the central amygdala than controls. The prelimbic region presented a larger neural ensemble in Cacna1c ± than WT animals, co-labelled for the time window of EL in the original context and prolonged exposure to the unrewarded context. With a context change, the Cacna1c ± infralimbic region displayed higher IEG expression during renewal than controls. Taken together, our findings provide novel evidence of distinct brain activation patterns occurring in Cacna1c ± rats after appetitive extinction and renewal despite preserved behavioral responses. This article is part of the Special Issue on "L-type calcium channel mechanisms in neuropsychiatric disorders".

Modeling the function of episodic memory in spatial learning

Episodic memory has been studied extensively in the past few decades, but so far little is understood about how it drives future behavior. Here we propose that episodic memory can facilitate learning in two fundamentally different modes: retrieval and replay, which is the reinstatement of hippocampal activity patterns during later sleep or awake quiescence. We study their properties by comparing three learning paradigms using computational modeling based on visually-driven reinforcement learning. Firstly, episodic memories are retrieved to learn from single experiences (one-shot learning); secondly, episodic memories are replayed to facilitate learning of statistical regularities (replay learning); and, thirdly, learning occurs online as experiences arise with no access to memories of past experiences (online learning). We found that episodic memory benefits spatial learning in a broad range of conditions, but the performance difference is meaningful only when the task is sufficiently complex and the number of learning trials is limited. Furthermore, the two modes of accessing episodic memory affect spatial learning differently. One-shot learning is typically faster than replay learning, but the latter may reach a better asymptotic performance. In the end, we also investigated the benefits of sequential replay and found that replaying stochastic sequences results in faster learning as compared to random replay when the number of replays is limited. Understanding how episodic memory drives future behavior is an important step toward elucidating the nature of episodic memory.

Learning shifts the preferred theta phase of gamma oscillations in CA1

Hippocampal neuronal oscillations reflect different cognitive processes and can therefore be used to dissect the role of hippocampal subfields in learning and memory. In particular, it has been suggested that encoding and retrieval is associated with slow gamma (25-55 Hz) and fast gamma (60-100 Hz) oscillations, respectively, which appear in a nested manner at specific phases of the ongoing theta oscillations (4-12 Hz). However, the relationship between memory demand and the theta phase of gamma oscillations remains unclear. Here, we assessed the theta phase preference of gamma oscillations in the CA1 region, at the starting and junction zones of a T-maze, while rats were learning an appetitive task. We found that the theta phase preference of slow gamma showed a ~180° phase shift when animals switched from novice to skilled performance during task acquisition. This phase-shift was not present at the junction zone, where animals chose a right or left turn within the T-maze, suggesting that a recall/decision process had already taken place at the starting zone. Our findings indicate that slow gamma oscillations support both encoding and retrieval, depending on the theta phase at which they occur. These properties are particularly evident prior to cognitive engagement in an acquired spatial task.