Qualitative adaptive reward learning with success failure maps: applied to humanoid robot walking

Parameterized gait pattern generator based on linear inverted pendulum model with natural ZMP references

This paper presents a parameterized gait generator based on linear inverted pendulum model (LIPM) theory, which allows users to generate a natural gait pattern with desired step sizes. Five types of zero moment point (ZMP) components are proposed for formulating a natural ZMP reference, where ZMP moves continuously during single support phases instead of staying at a fixed point in the sagittal and lateral plane. The corresponding center of mass (CoM) trajectories for these components are derived by LIPM theory. To generate a parameterized gait pattern with user-defined parameters, a gait planning algorithm is proposed, which determines related coefficients and boundary conditions of the CoM trajectory for each step. The proposed parameterized gait generator also provides a concept for users to generate gait patterns with self-defined ZMP references by using different components. Finally, the feasibility of the proposed method is validated by the experimental results with a teen-sized humanoid robot, David, which won first place in the sprint event at the 20th Federation of International Robot-soccer Association (FIRA) RoboWorld Cup.

Neural-Dynamic-Method-Based Dual-Arm CMG Scheme With Time-Varying Constraints Applied to Humanoid Robots

We propose a dual-arm cyclic-motion-generation (DACMG) scheme by a neural-dynamic method, which can remedy the joint-angle-drift phenomenon of a humanoid robot. In particular, according to a neural-dynamic design method, first, a cyclic-motion performance index is exploited and applied. This cyclic-motion performance index is then integrated into a quadratic programming (QP)-type scheme with time-varying constraints, called the time-varying-constrained DACMG (TVC-DACMG) scheme. The scheme includes the kinematic motion equations of two arms and the time-varying joint limits. The scheme can not only generate the cyclic motion of two arms for a humanoid robot but also control the arms to move to the desired position. In addition, the scheme considers the physical limit avoidance. To solve the QP problem, a recurrent neural network is presented and used to obtain the optimal solutions. Computer simulations and physical experiments demonstrate the effectiveness and the accuracy of such a TVC-DACMG scheme and the neural network solver.

A novel approach to locomotion learning: Actor-Critic architecture using central pattern generators and dynamic motor primitives

In this article, we propose an architecture of a bio-inspired controller that addresses the problem of learning different locomotion gaits for different robot morphologies. The modeling objective is split into two: baseline motion modeling and dynamics adaptation. Baseline motion modeling aims to achieve fundamental functions of a certain type of locomotion and dynamics adaptation provides a "reshaping" function for adapting the baseline motion to desired motion. Based on this assumption, a three-layer architecture is developed using central pattern generators (CPGs, a bio-inspired locomotor center for the baseline motion) and dynamic motor primitives (DMPs, a model with universal "reshaping" functions). In this article, we use this architecture with the actor-critic algorithms for finding a good "reshaping" function. In order to demonstrate the learning power of the actor-critic based architecture, we tested it on two experiments: (1) learning to crawl on a humanoid and, (2) learning to gallop on a puppy robot. Two types of actor-critic algorithms (policy search and policy gradient) are compared in order to evaluate the advantages and disadvantages of different actor-critic based learning algorithms for different morphologies. Finally, based on the analysis of the experimental results, a generic view/architecture for locomotion learning is discussed in the conclusion.

Multi-layered multi-pattern CPG for adaptive locomotion of humanoid robots

In this paper, we present an extended mathematical model of the central pattern generator (CPG) in the spinal cord. The proposed CPG model is used as the underlying low-level controller of a humanoid robot to generate various walking patterns. Such biological mechanisms have been demonstrated to be robust in locomotion of animal. Our model is supported by two neurophysiological studies. The first study identified a neural circuitry consisting of a two-layered CPG, in which pattern formation and rhythm generation are produced at different levels. The second study focused on a specific neural model that can generate different patterns, including oscillation. This neural model was employed in the pattern generation layer of our CPG, which enables it to produce different motion patterns-rhythmic as well as non-rhythmic motions. Due to the pattern-formation layer, the CPG is able to produce behaviors related to the dominating rhythm (extension/flexion) and rhythm deletion without rhythm resetting. The proposed multi-layered multi-pattern CPG model (MLMP-CPG) has been deployed in a 3D humanoid robot (NAO) while it performs locomotion tasks. The effectiveness of our model is demonstrated in simulations and through experimental results.

Neural network architecture for cognitive navigation in dynamic environments

Navigation in time-evolving environments with moving targets and obstacles requires cognitive abilities widely demonstrated by even simplest animals. However, it is a long-standing challenging problem for artificial agents. Cognitive autonomous robots coping with this problem must solve two essential tasks: 1) understand the environment in terms of what may happen and how I can deal with this and 2) learn successful experiences for their further use in an automatic subconscious way. The recently introduced concept of compact internal representation (CIR) provides the ground for both the tasks. CIR is a specific cognitive map that compacts time-evolving situations into static structures containing information necessary for navigation. It belongs to the class of global approaches, i.e., it finds trajectories to a target when they exist but also detects situations when no solution can be found. Here we extend the concept of situations with mobile targets. Then using CIR as a core, we propose a closed-loop neural network architecture consisting of conscious and subconscious pathways for efficient decision-making. The conscious pathway provides solutions to novel situations if the default subconscious pathway fails to guide the agent to a target. Employing experiments with roving robots and numerical simulations, we show that the proposed architecture provides the robot with cognitive abilities and enables reliable and flexible navigation in realistic time-evolving environments. We prove that the subconscious pathway is robust against uncertainty in the sensory information. Thus if a novel situation is similar but not identical to the previous experience (because of, e.g., noisy perception) then the subconscious pathway is able to provide an effective solution.

Artificial endocrine controller for power management in robotic systems

The robots that operate autonomously for extended periods in remote environments are often limited to gather only small amounts of power through photovoltaic solar panels. Such limited power budgets make power management critical to the success of the robot's mission. Artificial endocrine controllers, inspired by the mammalian endocrine system, have shown potential as a method for managing competing demands, gradually switching between behaviors, synchronizing behavior with external events, and maintaining a stable internal state of the robot. This paper reports the results obtained using these methods to manage power in an autonomous sailing robot. Artificial neural networks are used for sail and rudder control, while an artificial endocrine controller modulates the magnitude of actuator movements in response to battery or sunlight levels. Experiments are performed both in simulation and using a real robot. In simulation a 13-fold reduction in median power consumption is achieved; in the robot this is reduced to a twofold reduction because of the limitations of the simulation model. Additional simulations of a long term mission demonstrate the controller's ability to make gradual behavioral transitions and to synchronize behaviors with diurnal and seasonal changes in sunlight levels.

Cube Kohonen self-organizing map (CKSOM) model with new equations in organizing unstructured data

Surface reconstruction by using 3-D data is used to represent the surface of an object and perform important tasks. The type of data used is important and can be described as either structured or unstructured. For unstructured data, there is no connectivity information between data points. As a result, incorrect shapes will be obtained during the imaging process. Therefore, the data should be reorganized by finding the correct topology so that the correct shape can be obtained. Previous studies have shown that the Kohonen self-organizing map (KSOM) could be used to solve data organizing problems. However, 2-D Kohonen maps are limited because they are unable to cover the whole surface of closed 3-D surface data. Furthermore, the neurons inside the 3-D KSOM structure should be removed in order to create a correct wireframe model. This is because only the outside neurons are used to represent the surface of an object. The aim of this paper is to use KSOM to organize unstructured data for closed surfaces. KSOM isused in this paper by testing its ability to organize medical image data because KSOM is mostly used in constructing engineering field data. Enhancements are added to the model by introducing class number and the index vector, and new equations are created. Various grid sizes and maximum iterations are tested in the experiments. Based on the results, the number of redundancies is found to be directly proportional to the grid size. When we increase the maximum iterations, the surface of the image becomes smoother. An area formula is used and manual calculations are performed to validate the results. This model is implemented and images are created using Dev C++ and GNUPlot.

Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture

The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value.