A model of operant learning based on chaotically varying synaptic strength

Bio-inspired neural networks with central pattern generators for learning multi-skill locomotion

Biological neural circuits, central pattern generators (CPGs), located at the spinal cord are the underlying mechanisms that play a crucial role in generating rhythmic locomotion patterns. In this paper, we propose a novel approach that leverages the inherent rhythmicity of CPGs to enhance the locomotion capabilities of quadruped robots. Our proposed network architecture incorporates CPGs for rhythmic pattern generation and a multi-layer perceptron (MLP) network for fusing multi-dimensional sensory feedback. In particular, we also proposed a method to reformulate CPGs into a fully-differentiable, stateless network, allowing CPGs and MLP to be jointly trained using gradient-based learning. The effectiveness and performance of our approach are demonstrated through extensive experiments. Our learned locomotion policies exhibit agile and dynamic locomotion behaviors which are capable of traversing over uneven terrain blindly and resisting external perturbations. Furthermore, results demonstrated the remarkable multi-skill capability within a single unified policy network, including fall recovery and various quadrupedal gaits. Our study highlights the advantages of integrating bio-inspired neural networks which are capable of achieving intrinsic rhythmicity and fusing sensory feedback for generating smooth, versatile, and robust locomotion behaviors, including both rhythmic and non-rhythmic locomotion skills.

Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks

Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work.