Incremental learning of skill collections based on intrinsic motivation

An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey

The reinforcement learning (RL) research area is very active, with an important number of new contributions, especially considering the emergent field of deep RL (DRL). However, a number of scientific and technical challenges still need to be resolved, among which we acknowledge the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty, and skill-learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills which abstracts dynamics and makes the exploration process more robust.

Toward Computational Motivation for Multi-Agent Systems and Swarms

Motivation is a crucial part of animal and human mental development, fostering competence, autonomy, and open-ended development. Motivational constructs have proved to be an integral part of explaining human and animal behavior. Computer scientists have proposed various computational models of motivation for artificial agents, with the aim of building artificial agents capable of autonomous goal generation. Multi-agent systems and swarm intelligence are natural extensions to the individual agent setting. However, there are only a few works that focus on motivation theories in multi-agent or swarm settings. In this study, we review current computational models of motivation settings, mechanisms, functions and evaluation methods and discuss how we can produce systems with new kinds of functions not possible using individual agents. We describe in detail this open area of research and the major research challenges it holds.

Intrinsic motivation and mental replay enable efficient online adaptation in stochastic recurrent networks

Autonomous robots need to interact with unknown, unstructured and changing environments, constantly facing novel challenges. Therefore, continuous online adaptation for lifelong-learning and the need of sample-efficient mechanisms to adapt to changes in the environment, the constraints, the tasks, or the robot itself are crucial. In this work, we propose a novel framework for probabilistic online motion planning with online adaptation based on a bio-inspired stochastic recurrent neural network. By using learning signals which mimic the intrinsic motivation signal cognitive dissonance in addition with a mental replay strategy to intensify experiences, the stochastic recurrent network can learn from few physical interactions and adapts to novel environments in seconds. We evaluate our online planning and adaptation framework on an anthropomorphic KUKA LWR arm. The rapid online adaptation is shown by learning unknown workspace constraints sample-efficiently from few physical interactions while following given way points.

Mindful movement and skilled attention

Bodily movement has long been employed as a foundation for cultivating mental skills such as attention, self-control or mindfulness, with recent studies documenting the positive impacts of mindful movement training, such as yoga and tai chi. A parallel “mind-body connection” has also been observed in many developmental disorders. We elaborate a spectrum of mindfulness by considering ADHD, in which deficient motor control correlates with impaired (disinhibited) behavioral control contributing to defining features of excessive distractibility and impulsivity. These data provide evidence for an important axis of variation for wellbeing, in which skillful cognitive control covaries with a capacity for skillful movement. We review empirical and theoretical literature on attention, cognitive control, mind wandering, mindfulness and skill learning, endorsing a model of skilled attention in which motor plans, attention, and executive goals are seen as mutually co-defining aspects of skilled behavior that are linked by reciprocal inhibitory and excitatory connections. Thus, any movement training should engage “higher-order” inhibition and selection and develop a repertoire of rehearsed procedures that coordinate goals, attention and motor plans. However, we propose that mindful movement practice may improve the functional quality of rehearsed procedures, cultivating a transferrable skill of attention. We adopt Langer’s spectrum of mindful learning that spans from “mindlessness” to engagement with the details of the present task and contrast this with the mental attitudes cultivated in standard mindfulness meditation. We particularly follow Feldenkrais’ suggestion that mindful learning of skills for organizing the body in movement might transfer to other forms of mental activity. The results of mindful movement training should be observed in multiple complementary measures, and may have tremendous potential benefit for individuals with ADHD and other populations.