Invertebrate Learning: What Can't a Worm Learn?

Uncertainty maximization in partially observable domains: A cognitive perspective

Faced with an ever-increasing complexity of their domains of application, artificial learning agents are now able to scale up in their ability to process an overwhelming amount of data. However, this comes at the cost of encoding and processing an increasing amount of redundant information. This work exploits the possibility of learning systems, applied in partially observable domains, to selectively focus on the specific type of information that is more likely related to the causal interaction among transitioning states. A temporal difference displacement criterion is defined to implement adaptive masking of the observations. It can enable a significant improvement of convergence of temporal difference algorithms applied to partially observable Markov processes, as shown by experiments performed under a variety of machine learning problems, ranging from highly complex visuals as Atari games to simple textbook control problems such as CartPole. The proposed framework can be added to most RL algorithms since it only affects the observation process, selecting the parts more promising to explain the dynamics of the environment and reducing the dimension of the observation space.

Humans, fish, spiders and bees inherited working memory and attention from their last common ancestor

All brain processes that generate behaviour, apart from reflexes, operate with information that is in an "activated" state. This activated information, which is known as working memory (WM), is generated by the effect of attentional processes on incoming information or information previously stored in short-term or long-term memory (STM or LTM). Information in WM tends to remain the focus of attention; and WM, attention and STM together enable information to be available to mental processes and the behaviours that follow on from them. WM and attention underpin all flexible mental processes, such as solving problems, making choices, preparing for opportunities or threats that could be nearby, or simply finding the way home. Neither WM nor attention are necessarily conscious, and both may have evolved long before consciousness. WM and attention, with similar properties, are possessed by humans, archerfish, and other vertebrates; jumping spiders, honey bees, and other arthropods; and members of other clades, whose last common ancestor (LCA) is believed to have lived more than 600 million years ago. It has been reported that very similar genes control the development of vertebrate and arthropod brains, and were likely inherited from their LCA. Genes that control brain development are conserved because brains generate adaptive behaviour. However, the neural processes that generate behaviour operate with the activated information in WM, so WM and attention must have existed prior to the evolution of brains. It is proposed that WM and attention are widespread amongst animal species because they are phylogenetically conserved mechanisms that are essential to all mental processing, and were inherited from the LCA of vertebrates, arthropods, and some other animal clades.

Cephalopod Behavior: From Neural Plasticity to Consciousness

It is only in recent decades that subjective experience - or consciousness - has become a legitimate object of scientific inquiry. As such, it represents perhaps the greatest challenge facing neuroscience today. Subsumed within this challenge is the study of subjective experience in non-human animals: a particularly difficult endeavor that becomes even more so, as one crosses the great evolutionary divide between vertebrate and invertebrate phyla. Here, we explore the possibility of consciousness in one group of invertebrates: cephalopod molluscs. We believe such a review is timely, particularly considering cephalopods' impressive learning and memory abilities, rich behavioral repertoire, and the relative complexity of their nervous systems and sensory capabilities. Indeed, in some cephalopods, these abilities are so sophisticated that they are comparable to those of some higher vertebrates. Following the criteria and framework outlined for the identification of hallmarks of consciousness in non-mammalian species, here we propose that cephalopods - particularly the octopus - provide a unique test case among invertebrates for examining the properties and conditions that, at the very least, afford a basal faculty of consciousness. These include, among others: (i) discriminatory and anticipatory behaviors indicating a strong link between perception and memory recall; (ii) the presence of neural substrates representing functional analogs of thalamus and cortex; (iii) the neurophysiological dynamics resembling the functional signatures of conscious states in mammals. We highlight the current lack of evidence as well as potentially informative areas that warrant further investigation to support the view expressed here. Finally, we identify future research directions for the study of consciousness in these tantalizing animals.

Genetic influences on central and peripheral nervous system activity during fear conditioning

Fear conditioning is an evolutionarily conserved type of learning serving as a model for the acquisition of situationally induced anxiety. Brain function supporting fear conditioning may be genetically influenced, which in part could explain genetic susceptibility for anxiety following stress exposure. Using a classical twin design and functional magnetic resonance imaging, we evaluated genetic influences (h²) on brain activity and standard autonomic measures during fear conditioning. We found an additive genetic influence on mean brain activation (h² = 0.34) and autonomic responses (h² = 0.24) during fear learning. The experiment also allowed estimation of the genetic influence on brain activation during safety learning (h² = 0.55). The mean safety, but not fear, related brain activation was genetically correlated with autonomic responses. We conclude that fear and safety learning processes, both involved in anxiety development, are moderately genetically influenced as expressed both in the brain and the body.

Inversion of pheromone preference optimizes foraging in C. elegans

Foraging animals have to locate food sources that are usually patchily distributed and subject to competition. Deciding when to leave a food patch is challenging and requires the animal to integrate information about food availability with cues signaling the presence of other individuals (e.g., pheromones). To study how social information transmitted via pheromones can aid foraging decisions, we investigated the behavioral responses of the model animal Caenorhabditis elegans to food depletion and pheromone accumulation in food patches. We experimentally show that animals consuming a food patch leave it at different times and that the leaving time affects the animal preference for its pheromones. In particular, worms leaving early are attracted to their pheromones, while worms leaving later are repelled by them. We further demonstrate that the inversion from attraction to repulsion depends on associative learning and, by implementing a simple model, we highlight that it is an adaptive solution to optimize food intake during foraging.

Analysis of Mutants Suggests Kamin Blocking in C. elegans is Due to Interference with Memory Recall Rather than Storage

Higher-order conditioning phenomena, including context conditioning and blocking, occur when conditioning to one set of stimuli interacts with conditioning to a second set of stimuli to modulate the strength of the resultant memories. Here we analyze higher-order conditioning in the nematode worm Caenorhabditis elegans, demonstrating for the first time the presence of blocking in this animal, and dissociating it from context conditioning. We present an initial genetic dissection of these phenomena in a model benzaldehyde/NH₄Cl aversive learning system, and suggest that blocking may involve an alteration of memory retrieval rather than storage. These findings offer a fundamentally different explanation for blocking than traditional explanations, and position C. elegans as a powerful model organism for the study of higher order conditioning.

An Inconvenient Truth: Some Neglected Issues in Invertebrate Learning

The burgeoning field of invertebrate behavior is moving into what was the realm of human psychology concepts. This invites comparative studies not only between invertebrate and vertebrate species but also among the diverse taxa within the invertebrates, diverse even when considering only the insects. In order to make lasting progress two issues must be addressed. The first is inconsistent use of fundamental terms defining learning. The second is a focus on similarities, giving little attention to dissimilarities. In addition, much work is needed on whether behavioral similarities are grounded in the same neuronal architecture when considering disparate phyla. These concerns identify are "inconvenient truths" that weaken comparative behavioral analysis.

Sparse identification of nonlinear dynamics for rapid model recovery

Big data have become a critically enabling component of emerging mathematical methods aimed at the automated discovery of dynamical systems, where first principles modeling may be intractable. However, in many engineering systems, abrupt changes must be rapidly characterized based on limited, incomplete, and noisy data. Many leading automated learning techniques rely on unrealistically large data sets, and it is unclear how to leverage prior knowledge effectively to re-identify a model after an abrupt change. In this work, we propose a conceptual framework to recover parsimonious models of a system in response to abrupt changes in the low-data limit. First, the abrupt change is detected by comparing the estimated Lyapunov time of the data with the model prediction. Next, we apply the sparse identification of nonlinear dynamics (SINDy) regression to update a previously identified model with the fewest changes, either by addition, deletion, or modification of existing model terms. We demonstrate this sparse model recovery on several examples for abrupt system change detection in periodic and chaotic dynamical systems. Our examples show that sparse updates to a previously identified model perform better with less data, have lower runtime complexity, and are less sensitive to noise than identifying an entirely new model. The proposed abrupt-SINDy architecture provides a new paradigm for the rapid and efficient recovery of a system model after abrupt changes.

Ecologically relevant neurobehavioral assessment of the development of threat learning

As altricial infants gradually transition to adults, their proximate environment changes. In three short weeks, pups transition from a small world with the caregiver and siblings to a complex milieu rich in dangers as their environment expands. Such contrasting environments require different learning abilities and lead to distinct responses throughout development. Here, we will review some of the learned fear conditioned responses to threats in rats during their ontogeny, including behavioral and physiological measures that permit the assessment of learning and its supporting neurobiology from infancy through adulthood. In adulthood, odor-shock conditioning produces robust fear learning to the odor that depends upon the amygdala and related circuitry. Paradoxically, this conditioning in young pups fails to support fear learning and supports approach learning to the odor previously paired with shock. This approach learning is mediated by the infant attachment network that does not include the amygdala. During the age range when pups transition from the infant to the adult circuit (10-15 d old), pups have access to both networks: odor-shock conditioning in maternal presence uses the attachment circuit but the adult amygdala-dependent circuit when alone. However, throughout development (as young as 5 d old) the attachment associated learning can be overridden and amygdala-dependent fear learning supported, if the mother expresses fear in the presence of the pup. This social modulation of the fear permits the expression of defense reactions in life threatening situations informed by the caregiver but prevents the learning of the caregiver itself as a threat.

A genome-wide screen identifies PAPP-AA-mediated IGFR signaling as a novel regulator of habituation learning

Habituation represents a fundamental form of learning, yet the underlying molecular genetic mechanisms are not well defined. Here we report on a genome-wide genetic screen, coupled with whole-genome sequencing, that identified 14 zebrafish startle habituation mutants including mutants of the vertebrate-specific gene pregnancy-associated plasma protein-aa (pappaa). PAPP-AA encodes an extracellular metalloprotease known to increase IGF bioavailability, thereby enhancing IGF receptor signaling. We find that pappaa is expressed by startle circuit neurons, and expression of wild-type but not a metalloprotease-inactive version of pappaa restores habituation in pappaa mutants. Furthermore, acutely inhibiting IGF1R function in wild-type reduces habituation, while activation of IGF1R downstream effectors in pappaa mutants restores habituation, demonstrating that pappaa promotes learning by acutely and locally increasing IGF bioavailability. In sum, our results define the first functional gene set for habituation learning in a vertebrate and identify PAPPAA-regulated IGF signaling as a novel mechanism regulating habituation learning.

Escape and avoidance learning in the earthworm Eisenia hortensis

Interest in instrumental learning in earthworms dates back to 1912 when Yerkes concluded that they can learn a spatial discrimination in a T-maze. Rosenkoetter and Boice determined in the 1970s that the “learning” that Yerkes observed was probably chemotaxis and not learning at all. We examined a different form of instrumental learning: the ability to learn both to escape and to avoid an aversive stimulus. Freely moving “master” worms could turn off an aversive white light by increasing their movement; the behavior of yoked controls had no effect on the light. We demonstrate that in as few as 12 trials the behavior of the master worms comes under the control of this contingency.

Reinforcement learning and Tourette syndrome

In this chapter, we report the first experimental explorations of reinforcement learning in Tourette syndrome, realized by our team in the last few years. This report will be preceded by an introduction aimed to provide the reader with the state of the art of the knowledge concerning the neural bases of reinforcement learning at the moment of these studies and the scientific rationale beyond them. In short, reinforcement learning is learning by trial and error to maximize rewards and minimize punishments. This decision-making and learning process implicates the dopaminergic system projecting to the frontal cortex-basal ganglia circuits. A large body of evidence suggests that the dysfunction of the same neural systems is implicated in the pathophysiology of Tourette syndrome. Our results show that Tourette condition, as well as the most common pharmacological treatments (dopamine antagonists), affects reinforcement learning performance in these patients. Specifically, the results suggest a deficit in negative reinforcement learning, possibly underpinned by a functional hyperdopaminergia, which could explain the persistence of tics, despite their evident inadaptive (negative) value. This idea, together with the implications of these results in Tourette therapy and the future perspectives, is discussed in Section 4 of this chapter.

The quest for action potentials in C. elegans neurons hits a plateau

The small size and high resistance of C. elegans neurons makes them sensitive to the random opening of single ion channels, probably rendering codes that are based on classical, all-or-none action potentials unworkable. The recent discovery in C. elegans of a special class of regenerative events known as plateau potentials introduces the possibility of digital neural codes. Such codes would solve the problem of representing information in nervous systems in which action potentials are unreliable.

A synaptic DEG/ENaC ion channel mediates learning in C. elegans by facilitating dopamine signalling

An important component of learned behaviour is the ability to forecast positive or negative outcomes based on specific sensory cues. Predictive capacity is typically manifested by appropriate behavioural patterning. However, the molecular mechanisms underlying behavioural plasticity are poorly understood. Caenorhabditis elegans displays experience-dependent behavioural responses by associating distinct environmental signals. We find that ASIC-1, a member of the degenerin/epithelial sodium channel family, which localizes at presynaptic terminals of dopaminergic neurons, is required for associative learning in C. elegans. ASIC-1 functions in these neurons to amplify normal dopaminergic signalling, necessary for associative learning. Our results reveal a novel role of DEG/ENaC ion channels in neuronal communication by enhancing the activity of dopaminergic synapses. Similar mechanisms may facilitate synaptic plasticity in vertebrates.

Maze exploration and learning in C. elegans

The soil dwelling nematode, Caenorhabditis (C.) elegans, is a popular model system for studying behavioral plasticity. Noticeably absent from the C. elegans literature, however, are studies evaluating worm behavior in mazes. Here, we report the use of microfluidic mazes to investigate exploration and learning behaviors in wild-type C. elegans, as well as in the dopamine-poor mutant, cat-2. The key research findings include: (1)C. elegans worms are motivated to explore complex spatial environments with or without the presence of food/reward, (2) wild-type worms exhibit a greater tendency to explore relative to mutant worms, (3) both wild-type and mutant worms can learn to make unconditioned responses to food/reward, and (4) wild-type worms are significantly more likely to learn to make conditioned responses linking reward to location than mutant worms. These results introduce microfluidic mazes as a valuable new tool for biological behavioral analysis.

Caenorhabditis elegans senses bacterial autoinducers

Pseudomonas aeruginosa uses virulence factors controlled by quorum sensing (QS) to kill Caenorhabditis elegans. Here we show that C. elegans is attracted to the acylated homoserine lactones (AHSLs) that mediate QS in P. aeruginosa. Our data also indicate that C. elegans can distinguish AHSLs and may use them to mediate aversive or attractive learning.

Alpha7 mutants mimicking atypical motifs (YxxCC of loop-C, and E to H at -1' in TM2) in the C. elegans LEV-8 subunit affect nicotinic acetylcholine receptor function

The ACR-8-like group of C. elegans nicotinic acetylcholine receptor (nAChR) subunits contain unusual motifs in the ACh binding site and in the -1' position of transmembrane region two (TM2). Using site-directed mutagenesis (SDM) we have introduced these motifs into chicken alpha7 as it has not been possible to express C. elegans nAChR in vitro. Oocytes expressing alpha7 with the C. elegans binding motif show a reduced affinity and efficacy for both ACh and nicotine. The blocking action of the anthelmintic drug levamisole is reduced. The TM2 motif resulted in a non-functional receptor. We conclude that the TM2 motif profoundly restricts cation movement through the alpha7 channel but does not confer anion permeability. The altered form of the ACh binding motif is likely to result in a receptor with altered pharmacology, adding potential functional diversity at synapses in the nervous system and neuromuscular junctions of C. elegans.

A proposed hypothalamic-thalamic-striatal axis for the integration of energy balance, arousal, and food reward

We elaborate herein a novel theory of basal ganglia function that accounts for why palatable, energy-dense foods retain high incentive value even when immediate physiological energy requirements have been met. Basal ganglia function has been studied from the perspective of topographical segregation of processing within parallel circuits, with primary focus on motor control and cognition. Recent findings suggest, however, that the striatum can act as an integrated unit to modulate motivational state. We describe evidence that the striatal enkephalin system, which regulates the hedonic impact of preferred foods, undergoes coordinated gene expression changes that track current motivational state with regard to food intake. Striatal enkephalin gene expression is also downregulated by an intrastriatal infusion of a cholinergic muscarinic antagonist, a manipulation that greatly suppresses food intake. To account for these findings, we propose that signaling through a hypothalamic-midline thalamic-striatal axis impinges on the cholinergic interneurons of the striatum, which via their large, overlapping axonal fields act as a network to modulate enkephalin-containing striatal output neurons. A key relay in this circuit is the paraventricular thalamic nucleus, which receives convergent input from orexin-coded hypothalamic energy-sensing and behavioral state-regulating neurons, as well as from circadian oscillators, and projects to cholinergic interneurons throughout the striatal complex. We hypothesize that this system evolved to coordinate feeding and arousal, and to prolong the feeding central motivational state beyond the fulfillment of acute energy needs, thereby promoting "overeating" and the consequent development of an energy reserve for potential future food shortages.

Subcellular, cellular, and circuit mechanisms underlying classical conditioning in Hermissenda crassicornis

A breakthrough for studying the neuronal basis of learning emerged when invertebrates with simple nervous systems, such as the sea slug Hermissenda crassicornis, were shown to exhibit classical conditioning. Hermissenda learns to associate light with turbulence: prior to learning, naive animals move toward light (phototaxis) and contract their foot in response to turbulence; after learning, conditioned animals delay phototaxis in response to light. The photoreceptors of the eye, which receive monosynaptic inputs from statocyst hair cells, are both sensory neurons and the first site of sensory convergence. The memory of light associated with turbulence is stored as changes in intrinsic and synaptic currents in these photoreceptors. The subcellular mechanisms producing these changes include activation of protein kinase C and MAP kinase, which act as coincidence detectors because they are activated by convergent signaling pathways. Pathways of interneurons and motorneurons, where additional changes in excitability and synaptic connections are found, contribute to delayed phototaxis. Bursting activity recorded at several points suggest the existence of small networks that produce complex spatiotemporal firing patterns. Thus, the change in behavior may be produced by a nonlinear transformation of spatiotemporal firing patterns caused by plasticity of synaptic and intrinsic channels. The change in currents and the activation of PKC and MAPK produced by associative learning are similar to those observed in hippocampal and cerebellar neurons after rabbit classical conditioning, suggesting that these represent general mechanisms of memory storage. Thus, the knowledge gained from further study of Hermissenda will continue to illuminate mechanisms of mammalian learning.

A fresh look at an ancient receptor family: emerging roles for low density lipoprotein receptors in synaptic plasticity and memory formation

The well-known family of low-density lipoprotein receptors represents a collection of ancient membrane receptors that have been remarkably conserved throughout evolution. These multifunctional receptors, known to regulate cholesterol transport, are becoming increasingly interesting to the neuroscience community due to their ability to transduce a diversity of extracellular signals across the membrane in the adult CNS. Their roles in modulating synaptic plasticity and necessity in hippocampus-specific learning and memory have recently come to light. In addition, genetic, biochemical and behavioral studies have implicated these signaling systems in a number of human neurodegenerative and neuropsychiatric disorders involving loss of cognitive ability, such as Alzheimer's disease, schizophrenia and autism. This review describes the known functions of these receptors and discusses their potential role in processes of synaptic regulation and memory formation.