Online kernel-based learning for task-space tracking robot control

Adaptive robust synchronized control for cooperative robotic manipulators with uncertain base coordinate system

In this paper, cooperative robotic manipulators under uncertain base coordinate are investigated. The coordinate uncertainties result in biases of cooperative robotic dynamics, which involve horizontal and vertical translational errors in the task space and rotational errors in the joint space. To the best of our knowledge, uncertainties in the base coordinate system of cooperative robotic manipulators have drawn little attention in existing literature. To solve this problem, this paper presents an adaptive robust controller for the synchronized control of two cooperative robotic manipulators. An adaptive neural network associated with base coordinate parameter adaption law is proposed to estimate the cooperative system parameters given unknown system dynamics and base coordinate uncertainties. A synchronization-factor-based robust slide mode controller is then derived to stabilize the target position and internal force between the cooperative manipulators. Mathematical proof and numerical experiments under various conditions are conducted. The results demonstrate the satisfactory and effective convergences of both the cooperative robotic trajectory and internal force despite of uncertainties in the base coordinate system.

Autonomous Control of a Tendon-driven Robotic Limb with Elastic Elements Reveals that Added Elasticity can Enhance Learning

Passive elastic elements can contribute to stability, energetic efficiency, and impact absorption in both biological and robotic systems. They also add dynamical complexity which makes them more challenging to model and control. The impact of this added complexity to autonomous learning has not been thoroughly explored. This is especially relevant to tendon-driven limbs whose cables and tendons are inevitably elastic. Here, we explored the efficacy of autonomous learning and control on a simulated bio-plausible tendon-driven leg across different tendon stiffness values. We demonstrate that increasing stiffness of the simulated muscles can require more iterations for the inverse map to converge but can then perform more accurately, especially in discrete tasks. Moreover, the system is robust to subsequent changes in muscle stiffnesses and can adapt on-the-go within 5 attempts. Lastly, we test the system for the functional task of locomotion and found similar effects of muscle stiffness to learning and performance. Given that a range of stiffness values led to improved learning and maximized performance, we conclude the robot bodies and autonomous controllers-at least for tendon-driven systems-can be co-developed to take advantage of elastic elements. Importantly, this opens also the door to development efforts that recapitulate the beneficial aspects of the co-evolution of brains and bodies in vertebrates.

Efficient Reinforcement Learning via Probabilistic Trajectory Optimization

We present a trajectory optimization approach to reinforcement learning in continuous state and action spaces, called probabilistic differential dynamic programming (PDDP). Our method represents systems dynamics using Gaussian processes (GPs), and performs local dynamic programming iteratively around a nominal trajectory in Gaussian belief spaces. Different from model-based policy search methods, PDDP does not require a policy parameterization and learns a time-varying control policy via successive forward-backward sweeps. A convergence analysis of the iterative scheme is given, showing that our algorithm converges to a stationary point globally under certain conditions. We show that prior model knowledge can be incorporated into the proposed framework to speed up learning, and a generalized optimization criterion based on the predicted cost distribution can be employed to enable risk-sensitive learning. We demonstrate the effectiveness and efficiency of the proposed algorithm using nontrivial tasks. Compared with a state-of-the-art GP-based policy search method, PDDP offers a superior combination of learning speed, data efficiency, and applicability.

Extracting motor synergies from random movements for low-dimensional task-space control of musculoskeletal robots

In the field of human motor control, the motor synergy hypothesis explains how humans simplify body control dimensionality by coordinating groups of muscles, called motor synergies, instead of controlling muscles independently. In most applications of motor synergies to low-dimensional control in robotics, motor synergies are extracted from given optimal control signals. In this paper, we address the problems of how to extract motor synergies without optimal data given, and how to apply motor synergies to achieve low-dimensional task-space tracking control of a human-like robotic arm actuated by redundant muscles, without prior knowledge of the robot. We propose to extract motor synergies from a subset of randomly generated reaching-like movement data. The essence is to first approximate the corresponding optimal control signals, using estimations of the robot's forward dynamics, and to extract the motor synergies subsequently. In order to avoid modeling difficulties, a learning-based control approach is adopted such that control is accomplished via estimations of the robot's inverse dynamics. We present a kernel-based regression formulation to estimate the forward and the inverse dynamics, and a sliding controller in order to cope with estimation error. Numerical evaluations show that the proposed method enables extraction of motor synergies for low-dimensional task-space control.

Learning to track multiple targets

Monocular multiple-object tracking is a fundamental yet under-addressed computer vision problem. In this paper, we propose a novel learning framework for tracking multiple objects by detection. First, instead of heuristically defining a tracking algorithm, we learn that a discriminative structure prediction model from labeled video data captures the interdependence of multiple influence factors. Given the joint targets state from the last time step and the observation at the current frame, the joint targets state at the current time step can then be inferred by maximizing the joint probability score. Second, our detection results benefit from tracking cues. The traditional detection algorithms need a nonmaximal suppression postprocessing to select a subset from the total detection responses as the final output and a large number of selection mistakes are induced, especially under a congested circumstance. Our method integrates both detection and tracking cues. This integration helps to decrease the postprocessing mistake risk and to improve performance in tracking. Finally, we formulate the entire model training into a convex optimization problem and estimate its parameters using the cutting plane optimization. Experiments show that our method performs effectively in a large variety of scenarios, including pedestrian tracking in crowd scenes and vehicle tracking in congested traffic.

Linguistic decision making for robot route learning

Machine learning enables the creation of a nonlinear mapping that describes robot-environment interaction, whereas computing linguistics make the interaction transparent. In this paper, we develop a novel application of a linguistic decision tree for a robot route learning problem by dynamically deciding the robot's behavior, which is decomposed into atomic actions in the context of a specified task. We examine the real-time performance of training and control of a linguistic decision tree, and explore the possibility of training a machine learning model in an adaptive system without dual CPUs for parallelization of training and control. A quantified evaluation approach is proposed, and a score is defined for the evaluation of a model's robustness regarding the quality of training data. Compared with the nonlinear system identification nonlinear auto-regressive moving average with eXogeneous inputs model structure with offline parameter estimation, the linguistic decision tree model with online linguistic ID3 learning achieves much better performance, robustness, and reliability.

Online selective kernel-based temporal difference learning

In this paper, an online selective kernel-based temporal difference (OSKTD) learning algorithm is proposed to deal with large scale and/or continuous reinforcement learning problems. OSKTD includes two online procedures: online sparsification and parameter updating for the selective kernel-based value function. A new sparsification method (i.e., a kernel distance-based online sparsification method) is proposed based on selective ensemble learning, which is computationally less complex compared with other sparsification methods. With the proposed sparsification method, the sparsified dictionary of samples is constructed online by checking if a sample needs to be added to the sparsified dictionary. In addition, based on local validity, a selective kernel-based value function is proposed to select the best samples from the sample dictionary for the selective kernel-based value function approximator. The parameters of the selective kernel-based value function are iteratively updated by using the temporal difference (TD) learning algorithm combined with the gradient descent technique. The complexity of the online sparsification procedure in the OSKTD algorithm is O(n). In addition, two typical experiments (Maze and Mountain Car) are used to compare with both traditional and up-to-date O(n) algorithms (GTD, GTD2, and TDC using the kernel-based value function), and the results demonstrate the effectiveness of our proposed algorithm. In the Maze problem, OSKTD converges to an optimal policy and converges faster than both traditional and up-to-date algorithms. In the Mountain Car problem, OSKTD converges, requires less computation time compared with other sparsification methods, gets a better local optima than the traditional algorithms, and converges much faster than the up-to-date algorithms. In addition, OSKTD can reach a competitive ultimate optima compared with the up-to-date algorithms.

Artificial endocrine controller for power management in robotic systems

The robots that operate autonomously for extended periods in remote environments are often limited to gather only small amounts of power through photovoltaic solar panels. Such limited power budgets make power management critical to the success of the robot's mission. Artificial endocrine controllers, inspired by the mammalian endocrine system, have shown potential as a method for managing competing demands, gradually switching between behaviors, synchronizing behavior with external events, and maintaining a stable internal state of the robot. This paper reports the results obtained using these methods to manage power in an autonomous sailing robot. Artificial neural networks are used for sail and rudder control, while an artificial endocrine controller modulates the magnitude of actuator movements in response to battery or sunlight levels. Experiments are performed both in simulation and using a real robot. In simulation a 13-fold reduction in median power consumption is achieved; in the robot this is reduced to a twofold reduction because of the limitations of the simulation model. Additional simulations of a long term mission demonstrate the controller's ability to make gradual behavioral transitions and to synchronize behaviors with diurnal and seasonal changes in sunlight levels.