Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning

Soft-body dynamics induces energy efficiency in undulatory swimming: A deep learning study

Recently, soft robotics has gained considerable attention as it promises numerous applications thanks to unique features originating from the physical compliance of the robots. Biomimetic underwater robots are a promising application in soft robotics and are expected to achieve efficient swimming comparable to the real aquatic life in nature. However, the energy efficiency of soft robots of this type has not gained much attention and has been fully investigated previously. This paper presents a comparative study to verify the effect of soft-body dynamics on energy efficiency in underwater locomotion by comparing the swimming of soft and rigid snake robots. These robots have the same motor capacity, mass, and body dimensions while maintaining the same actuation degrees of freedom. Different gait patterns are explored using a controller based on grid search and the deep reinforcement learning controller to cover the large solution space for the actuation space. The quantitative analysis of the energy consumption of these gaits indicates that the soft snake robot consumed less energy to reach the same velocity as the rigid snake robot. When the robots swim at the same average velocity of 0.024 m/s, the required power for the soft-body robot is reduced by 80.4% compared to the rigid counterpart. The present study is expected to contribute to promoting a new research direction to emphasize the energy efficiency advantage of soft-body dynamics in robot design.

Joint elasticity produces energy efficiency in underwater locomotion: Verification with deep reinforcement learning

Underwater snake robots have received attention because of their unique mechanics and locomotion patterns. Given their highly redundant degrees of freedom, designing an energy-efficient gait has been a main challenge for the long-term autonomy of underwater snake robots. We propose a gait design method for an underwater snake robot based on deep reinforcement learning and curriculum learning. For comparison, we consider the gait generated by a conventional parametric gait equation controller as the baseline. Furthermore, inspired by the joints of living organisms, we consider elasticity (stiffness) in the joints of the snake robot to verify whether it contributes to the generation of energy efficiency in the underwater gait. We first demonstrate that the deep reinforcement learning controller can produce a more energy-efficient gait than the gait equation controller in underwater locomotion, by finding the control patterns which maximize the effect of energy efficiency through the exploitation of joint elasticity. In addition, appropriate joint elasticity can increase the maximum velocity achievable by a snake robot. Finally, simulation results in different liquid environments confirm that the deep reinforcement learning controller is superior to the gait equation controller, and it can find adaptive energy-efficient motion even when the liquid environment is changed. The video can be viewed at https://youtu.be/wpwQihhntEY.

PSTO: Learning Energy-Efficient Locomotion for Quadruped Robots

Energy efficiency is critical for the locomotion of quadruped robots. However, energy efficiency values found in simulations do not transfer adequately to the real world. To address this issue, we present a novel method, named Policy Search Transfer Optimization (PSTO), which combines deep reinforcement learning and optimization to create energy-efficient locomotion for quadruped robots in the real world. The deep reinforcement learning and policy search process are performed by the TD3 algorithm and the policy is transferred to the open-loop control trajectory further optimized by numerical methods, and conducted on the robot in the real world. In order to ensure the high uniformity of the simulation results and the behavior of the hardware platform, we introduce and validate the accurate model in simulation including consistent size and fine-tuning parameters. We then validate those results with real-world experiments on the quadruped robot Ant by executing dynamic walking gaits with different leg lengths and numbers of amplifications. We analyze the results and show that our methods can outperform the control method provided by the state-of-the-art policy search algorithm TD3 and sinusoid function on both energy efficiency and speed.

Creating Better Collision-Free Trajectory for Robot Motion Planning by Linearly Constrained Quadratic Programming

Many algorithms in probabilistic sampling-based motion planning have been proposed to create a path for a robot in an environment with obstacles. Due to the randomness of sampling, they can efficiently compute the collision-free paths made of segments lying in the configuration space with probabilistic completeness. However, this property also makes the trajectories have some unnecessary redundant or jerky motions, which need to be optimized. For most robotics applications, the trajectories should be short, smooth and keep away from obstacles. This paper proposes a new trajectory optimization technique which transforms a polygon collision-free path into a smooth path, and can deal with trajectories which contain various task constraints. The technique removes redundant motions by quadratic programming in the parameter space of trajectory, and converts collision avoidance conditions to linear constraints to ensure absolute safety of trajectories. Furthermore, the technique uses a projection operator to realize the optimization of trajectories which are subject to some hard kinematic constraints, like keeping a glass of water upright or coordinating operation with dual robots. The experimental results proved the feasibility and effectiveness of the proposed method, when it is compared with other trajectory optimization methods.

Domain Adaptation for Imitation Learning Using Generative Adversarial Network

Imitation learning is an effective approach for an autonomous agent to learn control policies when an explicit reward function is unavailable, using demonstrations provided from an expert. However, standard imitation learning methods assume that the agents and the demonstrations provided by the expert are in the same domain configuration. Such an assumption has made the learned policies difficult to apply in another distinct domain. The problem is formalized as domain adaptive imitation learning, which is the process of learning how to perform a task optimally in a learner domain, given demonstrations of the task in a distinct expert domain. We address the problem by proposing a model based on Generative Adversarial Network. The model aims to learn both domain-shared and domain-specific features and utilizes it to find an optimal policy across domains. The experimental results show the effectiveness of our model in a number of tasks ranging from low to complex high-dimensional.

Reinforcement learning for robot research: A comprehensive review and open issues

Applying the learning mechanism of natural living beings to endow intelligent robots with humanoid perception and decision-making wisdom becomes an important force to promote the revolution of science and technology in robot domains. Advances in reinforcement learning (RL) over the past decades have led robotics to be highly automated and intelligent, which ensures safety operation instead of manual work and implementation of more intelligence for many challenging tasks. As an important branch of machine learning, RL can realize sequential decision-making under uncertainties through end-to-end learning and has made a series of significant breakthroughs in robot applications. In this review article, we cover RL algorithms from theoretical background to advanced learning policies in different domains, which accelerate to solving practical problems in robotics. The challenges, open issues, and our thoughts on future research directions of RL are also presented to discover new research areas with the objective to motivate new interest.

Hexapod Robot Gait Switching for Energy Consumption and Cost of Transport Management Using Heuristic Algorithms

Due to the prospect of using walking robots in an impassable environment for tracked or wheeled vehicles, walking locomotion is one of the most remarkable accomplishments in robotic history. Walking robots, however, are still being deeply researched and created. Locomotion over irregular terrain and energy consumption are among the major problems. Walking robots require many actuators to cross different terrains, leading to substantial consumption of energy. A robot must be carefully designed to solve this problem, and movement parameters must be correctly chosen. We present a minimization of the hexapod robot’s energy consumption in this paper. Secondly, we investigate the reliance on power consumption in robot movement speed and gaits along with the Cost of Transport (CoT). To perform optimization of the hexapod robot energy consumption, we propose two algorithms. The heuristic algorithm performs gait switching based on the current speed of the robot to ensure minimum energy consumption. The Red Fox Optimization (RFO) algorithm performs a nature-inspired search of robot gait variable space to minimize CoT as a target function. The algorithms are tested to assess the efficiency of the hexapod robot walking through real-life experiments. We show that it is possible to save approximately 7.7–21% by choosing proper gaits at certain speeds. Finally, we demonstrate that our hexapod robot is one of the most energy-efficient hexapods by comparing the CoT values of various walking robots.

Spatial Topological Relation Analysis for Cluttered Scenes

The spatial topological relations are the foundation of robot operation planning under unstructured and cluttered scenes. Defining complex relations and dealing with incomplete point clouds from the surface of objects are the most difficult challenge in the spatial topological relation analysis. In this paper, we presented the classification of spatial topological relations by dividing the intersection space into six parts. In order to improve accuracy and reduce computing time, convex hulls are utilized to represent the boundary of objects and the spatial topological relations can be determined by the category of points in point clouds. We verified our method on the datasets. The result demonstrated that we have great improvement comparing with the previous method.

Attitude Trajectory Optimization to Ensure Balance Hexapod Locomotion

This paper proposes a simple attitude trajectory optimization method to enhance the walking balance of a large-size hexapod robot. To achieve balance motion control of a large-size hexapod robot on different outdoor terrains, we planned the balance attitude trajectories of the robot during walking and introduced how leg trajectories are generated based on the planned attitude trajectories. While planning the attitude trajectories, high order polynomial interpolation was employed with attitude fluctuation counteraction considered. Constraints that the planned attitude trajectories must satisfy during walking were well-considered. The trajectory of the swing leg was well designed with the terrain attitude considered to improve the environmental adaptability of the robot during the attitude adjustment process, and the trajectory of the support leg was automatically generated to satisfy the demand of the balance attitude trajectories planned. Comparative experiments of the real large-size hexapod robot walking on different terrains were carried out to validate the effectiveness and applicability of the attitude trajectory optimization method proposed, which demonstrated that, compared with the currently developed balance motion controllers, the attitude trajectory optimization method proposed can simplify the control system design and improve the walking balance of a hexapod robot.