Reversal Learning in Humans and Gerbils: Dynamic Control Network Facilitates Learning

Linking meta-learning to meta-structure

We propose that a principled understanding of meta-learning, as aimed for by the authors, benefits from linking the focus on learning with an equally strong focus on structure, which means to address the question: What are the meta-structures that can guide meta-learning?

Modularity in Nervous Systems—a Key to Efficient Adaptivity for Deep Reinforcement Learning

Modularity as observed in biological systems has proven valuable for guiding classical motor theories towards good answers about action selection and execution. New challenges arise when we turn to learning: Trying to scale current computational models, such as deep reinforcement learning (DRL), to action spaces, input dimensions, and time horizons seen in biological systems still faces severe obstacles unless vast amounts of training data are available. This leads to the question: does biological modularity also hold an important key for better answers to obtain efficient adaptivity for deep reinforcement learning? We review biological experimental work on modularity in biological motor control and link this with current examples of (deep) RL approaches. Analyzing outcomes of simulation studies, we show that these approaches benefit from forms of modularization as found in biological systems. We identify three different strands of modularity exhibited in biological control systems. Two of them—modularity in state (i) and in action (ii) spaces—appear as a consequence of local interconnectivity (as in reflexes) and are often modulated by higher levels in a control hierarchy. A third strand arises from chunking of action elements along a (iii) temporal dimension. Usually interacting in an overarching spatio-temporal hierarchy of the overall system, the three strands offer major “factors” decomposing the entire modularity structure. We conclude that modularity with its above strands can provide an effective prior for DRL approaches to speed up learning considerably and making learned controllers more robust and adaptive.

Serotonergic Facilitation of Forelimb Functional Recovery in Rats with Cervical Spinal Cord Injury

Serotonergic agents can improve the recovery of motor ability after a spinal cord injury. Herein, we compare the effects of buspirone, a 5-HT1A receptor partial agonist, to fluoxetine, a selective serotonin reuptake inhibitor, on forelimb motor function recovery after a C4 bilateral dorsal funiculi crush in adult female rats. After injury, single pellet reaching performance and forelimb muscle activity decreased in all rats. From 1 to 6 weeks after injury, rats were tested on these tasks with and without buspirone (1-2 mg/kg) or fluoxetine (1-5 mg/kg). Reaching and grasping success rates of buspirone-treated rats improved rapidly within 2 weeks after injury and plateaued over the next 4 weeks of testing. Electromyography (EMG) from selected muscles in the dominant forelimb showed that buspirone-treated animals used new reaching strategies to achieve success after the injury. However, forelimb performance dramatically decreased within 2 weeks of buspirone withdrawal. In contrast, fluoxetine treatment resulted in a more progressive rate of improvement in forelimb performance over 8 weeks after injury. Neither buspirone nor fluoxetine significantly improved quadrupedal locomotion on the horizontal ladder test. The improved accuracy of reaching and grasping, patterns of muscle activity, and increased excitability of spinal motor-evoked potentials after buspirone administration reflect extensive reorganization of connectivity within and between supraspinal and spinal sensory-motor netxcopy works. Thus, both serotonergic drugs, buspirone and fluoxetine, neuromodulated these networks to physiological states that enabled markedly improved forelimb function after cervical spinal cord injury.

Machine learning identifies the dynamics and influencing factors in an auditory category learning experiment

Human learning is one of the main topics in psychology and cognitive neuroscience. The analysis of experimental data, e.g. from category learning experiments, is a major challenge due to confounding factors related to perceptual processing, feedback value, response selection, as well as inter-individual differences in learning progress due to differing strategies or skills. We use machine learning to investigate (Q1) how participants of an auditory category-learning experiment evolve towards learning, (Q2) how participant performance saturates and (Q3) how early we can differentiate whether a participant has learned the categories or not. We found that a Gaussian Mixture Model describes well the evolution of participant performance and serves as basis for identifying influencing factors of task configuration (Q1). We found early saturation trends (Q2) and that CatBoost, an advanced classification algorithm, can separate between participants who learned the categories and those who did not, well before the end of the learning session, without much degradation of separation quality (Q3). Our results show that machine learning can model participant dynamics, identify influencing factors of task design and performance trends. This will help to improve computational models of auditory category learning and define suitable time points for interventions into learning, e.g. by tutorial systems.

β-adrenergic modulation of discrimination learning and memory in the auditory cortex

Despite vast literature on catecholaminergic neuromodulation of auditory cortex functioning in general, knowledge about its role for long‐term memory formation is scarce. Our previous pharmacological studies on cortex‐dependent frequency‐modulated tone‐sweep discrimination learning of Mongolian gerbils showed that auditory‐cortical D_1/5‐dopamine receptor activity facilitates memory consolidation and anterograde memory formation. Considering overlapping functions of D_1/5‐dopamine receptors and β‐adrenoceptors, we hypothesised a role of β‐adrenergic signalling in the auditory cortex for sweep discrimination learning and memory. Supporting this hypothesis, the β_1/2‐adrenoceptor antagonist propranolol bilaterally applied to the gerbil auditory cortex after task acquisition prevented the discrimination increment that was normally monitored 1 day later. The increment in the total number of hurdle crossings performed in response to the sweeps per se was normal. Propranolol infusion after the seventh training session suppressed the previously established sweep discrimination. The suppressive effect required antagonist injection in a narrow post‐session time window. When applied to the auditory cortex 1 day before initial conditioning, β₁‐adrenoceptor‐antagonising and β₁‐adrenoceptor‐stimulating agents retarded and facilitated, respectively, sweep discrimination learning, whereas β₂‐selective drugs were ineffective. In contrast, single‐sweep detection learning was normal after propranolol infusion. By immunohistochemistry, β₁‐ and β₂‐adrenoceptors were identified on the neuropil and somata of pyramidal and non‐pyramidal neurons of the gerbil auditory cortex. The present findings suggest that β‐adrenergic signalling in the auditory cortex has task‐related importance for discrimination learning of complex sounds: as previously shown for D_1/5‐dopamine receptor signalling, β‐adrenoceptor activity supports long‐term memory consolidation and reconsolidation; additionally, tonic input through β₁‐adrenoceptors may control mechanisms permissive for memory acquisition.

A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making

Decision-making is a high-level cognitive process based on cognitive processes like perception, attention, and memory. Real-life situations require series of decisions to be made, with each decision depending on previous feedback from a potentially changing environment. To gain a better understanding of the underlying processes of dynamic decision-making, we applied the method of cognitive modeling on a complex rule-based category learning task. Here, participants first needed to identify the conjunction of two rules that defined a target category and later adapt to a reversal of feedback contingencies. We developed an ACT-R model for the core aspects of this dynamic decision-making task. An important aim of our model was that it provides a general account of how such tasks are solved and, with minor changes, is applicable to other stimulus materials. The model was implemented as a mixture of an exemplar-based and a rule-based approach which incorporates perceptual-motor and metacognitive aspects as well. The model solves the categorization task by first trying out one-feature strategies and then, as a result of repeated negative feedback, switching to two-feature strategies. Overall, this model solves the task in a similar way as participants do, including generally successful initial learning as well as reversal learning after the change of feedback contingencies. Moreover, the fact that not all participants were successful in the two learning phases is also reflected in the modeling data. However, we found a larger variance and a lower overall performance of the modeling data as compared to the human data which may relate to perceptual preferences or additional knowledge and rules applied by the participants. In a next step, these aspects could be implemented in the model for a better overall fit. In view of the large interindividual differences in decision performance between participants, additional information about the underlying cognitive processes from behavioral, psychobiological and neurophysiological data may help to optimize future applications of this model such that it can be transferred to other domains of comparable dynamic decision tasks.