A Mechanistic Model for Reward Prediction and Extinction Learning in the Fruit Fly

Functional dissection of a neuronal brain circuit mediating higher-order associative learning

A central feature characterizing the neural architecture of many species' brains is their capacity to form associative chains through learning. In elementary forms of associative learning, stimuli coinciding with reward or punishment become attractive or repulsive. Notably, stimuli previously learned as attractive or repulsive can themselves serve as reinforcers, establishing a cascading effect whereby they become associated with additional stimuli. When this iterative process is perpetuated, it results in higher-order associations. Here, we use odor conditioning in Drosophila and computational modeling to dissect the architecture of neuronal networks underlying higher-order associative learning. We show that the responsible circuit, situated in the mushroom bodies of the brain, is characterized by parallel processing of odor information and by recurrent excitatory and inhibitory feedback loops that empower odors to gain control over the dopaminergic valence-signaling system. Our findings establish a paradigmatic framework of a neuronal circuit diagram enabling the acquisition of associative chains.

Dopamine-mediated interactions between short- and long-term memory dynamics

In dynamic environments, animals make behavioural decisions on the basis of the innate valences of sensory cues and information learnt about these cues across multiple timescales1-3. However, it remains unclear how the innate valence of a sensory stimulus affects the acquisition of learnt valence information and subsequent memory dynamics. Here we show that in the Drosophila brain, interconnected short- and long-term memory units of the mushroom body jointly regulate memory through dopamine signals that encode innate and learnt sensory valences. By performing time-lapse in vivo voltage-imaging studies of neural spiking in more than 500 flies undergoing olfactory associative conditioning, we found that protocerebral posterior lateral 1 dopamine neurons (PPL1-DANs)4 heterogeneously and bidirectionally encode innate and learnt valences of punishment, reward and odour cues. During learning, these valence signals regulate memory storage and extinction in mushroom body output neurons (MBONs)5. During initial conditioning bouts, PPL1-γ1pedc and PPL1-γ2α'1 neurons control short-term memory formation, which weakens inhibitory feedback from MBON-γ1pedc>α/β to PPL1-α'2α2 and PPL1-α3. During further conditioning, this diminished feedback allows these two PPL1-DANs to encode the net innate plus learnt valence of the conditioned odour cue, which gates long-term memory formation. A computational model constrained by the fly connectome6,7 and our spiking data explains how dopamine signals mediate the circuit interactions between short- and long-term memory traces, yielding predictions that our experiments confirmed. Overall, the mushroom body achieves flexible learning through the integration of innate and learnt valences in parallel learning units sharing feedback interconnections. This hybrid physiological-anatomical mechanism may be a general means by which dopamine regulates memory dynamics in other species and brain structures, including the vertebrate basal ganglia.

Analysis of fast calcium dynamics of honey bee olfactory coding

Odour processing exhibits multiple parallels between vertebrate and invertebrate olfactory systems. Insects, in particular, have emerged as relevant models for olfactory studies because of the tractability of their olfactory circuits. Here, we used fast calcium imaging to track the activity of projection neurons in the honey bee antennal lobe (AL) during olfactory stimulation at high temporal resolution. We observed a heterogeneity of response profiles and an abundance of inhibitory activities, resulting in various response latencies and stimulus-specific post-odour neural signatures. Recorded calcium signals were fed to a mushroom body (MB) model constructed implementing the fundamental features of connectivity between olfactory projection neurons, Kenyon cells (KC), and MB output neurons (MBON). The model accounts for the increase of odorant discrimination in the MB compared to the AL and reveals the recruitment of two distinct KC populations that represent odorants and their aftersmell as two separate but temporally coherent neural objects. Finally, we showed that the learning-induced modulation of KC-to-MBON synapses can explain both the variations in associative learning scores across different conditioning protocols used in bees and the bees' response latency. Thus, it provides a simple explanation of how the time contingency between the stimulus and the reward can be encoded without the need for time tracking. This study broadens our understanding of olfactory coding and learning in honey bees. It demonstrates that a model based on simple MB connectivity rules and fed with real physiological data can explain fundamental aspects of odour processing and associative learning.

Learning with sparse reward in a gap junction network inspired by the insect mushroom body

Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This 'distal' or 'sparse' reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.

Beyond prediction error: 25 years of modeling the associations formed in the insect mushroom body

The insect mushroom body has gained increasing attention as a system in which the computational basis of neural learning circuits can be unraveled. We now understand in detail the key locations in this circuit where synaptic associations are formed between sensory patterns and values leading to actions. However, the actual learning rule (or rules) implemented by neural activity and leading to synaptic change is still an open question. Here, I survey the diversity of answers that have been offered in computational models of this system over the past decades, including the recurring assumption-in line with top-down theories of associative learning-that the core function is to reduce prediction error. However, I will argue, a more bottom-up approach may ultimately reveal a richer algorithmic capacity in this still enigmatic brain neuropil.

Roles of feedback and feed-forward networks of dopamine subsystems: insights from Drosophila studies

Across animal species, dopamine-operated memory systems comprise anatomically segregated, functionally diverse subsystems. Although individual subsystems could operate independently to support distinct types of memory, the logical interplay between subsystems is expected to enable more complex memory processing by allowing existing memory to influence future learning. Recent comprehensive ultrastructural analysis of the Drosophila mushroom body revealed intricate networks interconnecting the dopamine subsystems-the mushroom body compartments. Here, we review the functions of some of these connections that are beginning to be understood. Memory consolidation is mediated by two different forms of network: A recurrent feedback loop within a compartment maintains sustained dopamine activity required for consolidation, whereas feed-forward connections across compartments allow short-term memory formation in one compartment to open the gate for long-term memory formation in another compartment. Extinction and reversal of aversive memory rely on a similar feed-forward circuit motif that signals omission of punishment as a reward, which triggers plasticity that counteracts the original aversive memory trace. Finally, indirect feed-forward connections from a long-term memory compartment to short-term memory compartments mediate higher-order conditioning. Collectively, these emerging studies indicate that feedback control and hierarchical connectivity allow the dopamine subsystems to work cooperatively to support diverse and complex forms of learning.

Prediction error drives associative learning and conditioned behavior in a spiking model of Drosophila larva

Predicting reinforcement from sensory cues is beneficial for goal-directed behavior. In insect brains, underlying associations between cues and reinforcement, encoded by dopaminergic neurons, are formed in the mushroom body. We propose a spiking model of the Drosophila larva mushroom body. It includes a feedback motif conveying learned reinforcement expectation to dopaminergic neurons, which can compute prediction error as the difference between expected and present reinforcement. We demonstrate that this can serve as a driving force in learning. When combined with synaptic homeostasis, our model accounts for theoretically derived features of acquisition and loss of associations that depend on the intensity of the reinforcement and its temporal proximity to the cue. From modeling olfactory learning over the time course of behavioral experiments and simulating the locomotion of individual larvae toward or away from odor sources in a virtual environment, we conclude that learning driven by prediction errors can explain larval behavior.

Minimal circuit motifs for second-order conditioning in the insect mushroom body

In well-established first-order conditioning experiments, the concurrence of a sensory cue with reinforcement forms an association, allowing the cue to predict future reinforcement. In the insect mushroom body, a brain region central to learning and memory, such associations are encoded in the synapses between its intrinsic and output neurons. This process is mediated by the activity of dopaminergic neurons that encode reinforcement signals. In second-order conditioning, a new sensory cue is paired with an already established one that presumably activates dopaminergic neurons due to its predictive power of the reinforcement. We explored minimal circuit motifs in the mushroom body for their ability to support second-order conditioning using mechanistic models. We found that dopaminergic neurons can either be activated directly by the mushroom body's intrinsic neurons or via feedback from the output neurons via several pathways. We demonstrated that the circuit motifs differ in their computational efficiency and robustness. Beyond previous research, we suggest an additional motif that relies on feedforward input of the mushroom body intrinsic neurons to dopaminergic neurons as a promising candidate for experimental evaluation. It differentiates well between trained and novel stimuli, demonstrating robust performance across a range of model parameters.

The mushroom body output encodes behavioral decision during sensory-motor transformation

Animals form a behavioral decision by evaluating sensory evidence on the background of past experiences and the momentary motivational state. In insects, we still lack understanding of how and at which stage of the recurrent sensory-motor pathway behavioral decisions are formed. The mushroom body (MB), a central brain structure in insects1 and crustaceans,2,3 integrates sensory input of different modalities4,5,6 with the internal state, the behavioral state, and external sensory context7,8,9,10 through a large number of recurrent, mostly neuromodulatory inputs,11,12 implicating a functional role for MBs in state-dependent sensory-motor transformation.13,14 A number of classical conditioning studies in honeybees15,16 and fruit flies17,18,19 have provided accumulated evidence that at its output, the MB encodes the valence of a sensory stimulus with respect to its behavioral relevance. Recent work has extended this notion of valence encoding to the context of innate behaviors.8,20,21,22 Here, we co-analyzed a defined feeding behavior and simultaneous extracellular single-unit recordings from MB output neurons (MBONs) in the cockroach in response to timed sensory stimulation with odors. We show that clear neuronal responses occurred almost exclusively during behaviorally responded trials. Early MBON responses to the sensory stimulus preceded the feeding behavior and predicted its occurrence or non-occurrence from the single-trial population activity. Our results therefore suggest that at its output, the MB does not merely encode sensory stimulus valence. We hypothesize instead that the MB output represents an integrated signal of internal state, momentary environmental conditions, and experience-dependent memory to encode a behavioral decision.

Reward expectations direct learning and drive operant matching in Drosophila

Foraging animals must use decision-making strategies that dynamically adapt to the changing availability of rewards in the environment. A wide diversity of animals do this by distributing their choices in proportion to the rewards received from each option, Herrnstein's operant matching law. Theoretical work suggests an elegant mechanistic explanation for this ubiquitous behavior, as operant matching follows automatically from simple synaptic plasticity rules acting within behaviorally relevant neural circuits. However, no past work has mapped operant matching onto plasticity mechanisms in the brain, leaving the biological relevance of the theory unclear. Here, we discovered operant matching in Drosophila and showed that it requires synaptic plasticity that acts in the mushroom body and incorporates the expectation of reward. We began by developing a dynamic foraging paradigm to measure choices from individual flies as they learn to associate odor cues with probabilistic rewards. We then built a model of the fly mushroom body to explain each fly's sequential choice behavior using a family of biologically realistic synaptic plasticity rules. As predicted by past theoretical work, we found that synaptic plasticity rules could explain fly matching behavior by incorporating stimulus expectations, reward expectations, or both. However, by optogenetically bypassing the representation of reward expectation, we abolished matching behavior and showed that the plasticity rule must specifically incorporate reward expectations. Altogether, these results reveal the first synapse-level mechanisms of operant matching and provide compelling evidence for the role of reward expectation signals in the fly brain.

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.

Optogenetically induced reward and 'frustration' memory in larval Drosophila

Humans and animals alike form oppositely valenced memories for stimuli that predict the occurrence versus the termination of a reward: appetitive 'reward' memory for stimuli associated with the occurrence of a reward and aversive 'frustration' memory for stimuli that are associated with its termination. We characterize these memories in larval Drosophila using a combination of Pavlovian conditioning, optogenetic activation of the dopaminergic central-brain DAN-i1864 neuron, and high-resolution video-tracking. This reveals their dependency on the number of training trials and the duration of DAN-i1864 activation, their temporal stability, and the parameters of locomotion that are modulated during memory expression. Together with previous results on 'punishment' versus 'relief' learning by DAN-f1 neuron activation, this reveals a 2x2 matrix of timing-dependent memory valence for the occurrence/ termination of reward/ punishment. These findings should aid the understanding and modelling of how brains decipher the predictive, causal structure of events around a target reinforcing occurrence.

Visualization of learning-induced synaptic plasticity in output neurons of the Drosophila mushroom body γ-lobe

By learning, through experience, which stimuli coincide with dangers, it is possible to predict outcomes and act pre-emptively to ensure survival. In insects, this process is localized to the mushroom body (MB), the circuitry of which facilitates the coincident detection of sensory stimuli and punishing or rewarding cues and, downstream, the execution of appropriate learned behaviors. Here, we focused our attention on the mushroom body output neurons (MBONs) of the γ-lobes that act as downstream synaptic partners of the MB γ-Kenyon cells (KCs) to ask how the output of the MB γ-lobe is shaped by olfactory associative conditioning, distinguishing this from non-associative stimulus exposure effects, and without the influence of downstream modulation. This was achieved by employing a subcellularly localized calcium sensor to specifically monitor activity at MBON postsynaptic sites. Therein, we identified a robust associative modulation within only one MBON postsynaptic compartment (MBON-γ1pedc > α/β), which displayed a suppressed postsynaptic response to an aversively paired odor. While this MBON did not undergo non-associative modulation, the reverse was true across the remainder of the γ-lobe, where general odor-evoked adaptation was observed, but no conditioned odor-specific modulation. In conclusion, associative synaptic plasticity underlying aversive olfactory learning is localized to one distinct synaptic γKC-to-γMBON connection.

An incentive circuit for memory dynamics in the mushroom body of Drosophila melanogaster

Insects adapt their response to stimuli, such as odours, according to their pairing with positive or negative reinforcements, such as sugar or shock. Recent electrophysiological and imaging findings in Drosophila melanogaster allow detailed examination of the neural mechanisms supporting the acquisition, forgetting, and assimilation of memories. We propose that this data can be explained by the combination of a dopaminergic plasticity rule that supports a variety of synaptic strength change phenomena, and a circuit structure (derived from neuroanatomy) between dopaminergic and output neurons that creates different roles for specific neurons. Computational modelling shows that this circuit allows for rapid memory acquisition, transfer from short term to long term, and exploration/exploitation trade-off. The model can reproduce the observed changes in the activity of each of the identified neurons in conditioning paradigms and can be used for flexible behavioural control.

Associative learning drives longitudinally graded presynaptic plasticity of neurotransmitter release along axonal compartments

Anatomical and physiological compartmentalization of neurons is a mechanism to increase the computational capacity of a circuit, and a major question is what role axonal compartmentalization plays. Axonal compartmentalization may enable localized, presynaptic plasticity to alter neuronal output in a flexible, experience-dependent manner. Here, we show that olfactory learning generates compartmentalized, bidirectional plasticity of acetylcholine release that varies across the longitudinal compartments of Drosophila mushroom body (MB) axons. The directionality of the learning-induced plasticity depends on the valence of the learning event (aversive vs. appetitive), varies linearly across proximal to distal compartments following appetitive conditioning, and correlates with learning-induced changes in downstream mushroom body output neurons (MBONs) that modulate behavioral action selection. Potentiation of acetylcholine release was dependent on the CaV2.1 calcium channel subunit cacophony. In addition, contrast between the positive conditioned stimulus and other odors required the inositol triphosphate receptor, which maintained responsivity to odors upon repeated presentations, preventing adaptation. Downstream from the MB, a set of MBONs that receive their input from the γ3 MB compartment were required for normal appetitive learning, suggesting that they represent a key node through which reward learning influences decision-making. These data demonstrate that learning drives valence-correlated, compartmentalized, bidirectional potentiation, and depression of synaptic neurotransmitter release, which rely on distinct mechanisms and are distributed across axonal compartments in a learning circuit.

From Biological Synapses to “Intelligent” Robots

This selective review explores biologically inspired learning as a model for intelligent robot control and sensing technology on the basis of specific examples. Hebbian synaptic learning is discussed as a functionally relevant model for machine learning and intelligence, as explained on the basis of examples from the highly plastic biological neural networks of invertebrates and vertebrates. Its potential for adaptive learning and control without supervision, the generation of functional complexity, and control architectures based on self-organization is brought forward. Learning without prior knowledge based on excitatory and inhibitory neural mechanisms accounts for the process through which survival-relevant or task-relevant representations are either reinforced or suppressed. The basic mechanisms of unsupervised biological learning drive synaptic plasticity and adaptation for behavioral success in living brains with different levels of complexity. The insights collected here point toward the Hebbian model as a choice solution for “intelligent” robotics and sensor systems.

Neural Circuits Underlying Behavioral Flexibility: Insights From Drosophila

Behavioral flexibility is critical to survival. Animals must adapt their behavioral responses based on changes in the environmental context, internal state, or experience. Studies in Drosophila melanogaster have provided insight into the neural circuit mechanisms underlying behavioral flexibility. Here we discuss how Drosophila behavior is modulated by internal and behavioral state, environmental context, and learning. We describe general principles of neural circuit organization and modulation that underlie behavioral flexibility, principles that are likely to extend to other species.

Neural Design Principles for Subjective Experience: Implications for Insects

How subjective experience is realized in nervous systems remains one of the great challenges in the natural sciences. An answer to this question should resolve debate about which animals are capable of subjective experience. We contend that subjective experience of sensory stimuli is dependent on the brain's awareness of its internal neural processing of these stimuli. This premise is supported by empirical evidence demonstrating that disruption to either processing streams or awareness states perturb subjective experience. Given that the brain must predict the nature of sensory stimuli, we reason that conscious awareness is itself dependent on predictions generated by hierarchically organized forward models of the organism's internal sensory processing. The operation of these forward models requires a specialized neural architecture and hence any nervous system lacking this architecture is unable to subjectively experience sensory stimuli. This approach removes difficulties associated with extrapolations from behavioral and brain homologies typically employed in addressing whether an animal can feel. Using nociception as a model sensation, we show here that the Drosophila brain lacks the required internal neural connectivity to implement the computations required of hierarchical forward models. Consequently, we conclude that Drosophila, and those insects with similar neuroanatomy, do not subjectively experience noxious stimuli and therefore cannot feel pain.