Comparing Robot Controller Optimization Methods on Evolvable Morphologies

In this paper we compare Bayesian Optimization, Differential Evolution, and an Evolution Strategy, employed as a gait learning algorithm in modular robots. The motivational scenario is the joint evolution of morphologies and controllers, where 'newborn' robots also undergo a learning process to optimize their inherited controllers (without changing their bodies). This context raises the question: How do gait learning algorithms compare when applied to various morphologies that are not known in advance (thus need to be treated without priors)? To answer this question, we use a test suite of twenty different robot morphologies to evaluate our gait learners and compare their efficiency, efficacy, and sensitivity to morphological differences. The results indicate that Bayesian Optimization and Differential Evolution deliver the same solution quality (walking speed for the robot) with fewer evaluations than the Evolution Strategy. Furthermore, the Evolution Strategy is more sensitive for morphological differences (its efficacy varies more between different morphologies) and is more subject to luck (repeated runs on the same morphology show greater variance in the outcomes).

Environmental Adaptation of Robot Morphology and Control Through Real-World Evolution

Robots operating in the real world will experience a range of different environments and tasks. It is essential for the robot to have the ability to adapt to its surroundings to work efficiently in changing conditions. Evolutionary robotics aims to solve this by optimizing both the control and body (morphology) of a robot, allowing adaptation to internal, as well as external factors. Most work in this field has been done in physics simulators, which are relatively simple and not able to replicate the richness of interactions found in the real world. Solutions that rely on the complex interplay among control, body, and environment are therefore rarely found. In this article, we rely solely on real-world evaluations and apply evolutionary search to yield combinations of morphology and control for our mechanically self-reconfiguring quadruped robot. We evolve solutions on two distinct physical surfaces and analyze the results in terms of both control and morphology. We then transition to two previously unseen surfaces to demonstrate the generality of our method. We find that the evolutionary search finds high-performing and diverse morphology-controller configurations by adapting both control and body to the different properties of the physical environments. We additionally find that morphology and control vary with statistical significance between the environments. Moreover, we observe that our method allows for morphology and control parameters to transfer to previously unseen terrains, demonstrating the generality of our approach.

Evolutionary Robotics Controllers with Proprioception Facilitated by Neural-Network-Based Simulators

Taking inspiration from the navigation ability of humans, this study investigated a method of providing robotic controllers with a basic sense of position. It incorporated robotic simulators into robotic controllers to provide them with a mechanism to approximate the effects their actions had on the robot. Controllers with and without internal simulators were tested and compared. The proposed controller architecture was shown to outperform the regular controller architecture. However, the longer an internal simulator was executed, the more inaccurate it became. Thus, the performance of controllers with internal simulators reduced over time unless their internal simulator was periodically corrected.

Iterative improvement in the automatic modular design of robot swarms

Iterative improvement is an optimization technique that finds frequent application in heuristic optimization, but, to the best of our knowledge, has not yet been adopted in the automatic design of control software for robots. In this work, we investigate iterative improvement in the context of the automatic modular design of control software for robot swarms. In particular, we investigate the optimization of two control architectures: finite-state machines and behavior trees. Finite state machines are a common choice for the control architecture in swarm robotics whereas behavior trees have received less attention so far. We compare three different optimization techniques: iterative improvement, Iterated F-race, and a hybridization of Iterated F-race and iterative improvement. For reference, we include in our study also (i) a design method in which behavior trees are optimized via genetic programming and (ii) EvoStick, a yardstick implementation of the neuro-evolutionary swarm robotics approach. The results indicate that iterative improvement is a viable optimization algorithm in the automatic modular design of control software for robot swarms.

Automatic modular design of robot swarms using behavior trees as a control architecture

We investigate the possibilities, challenges, and limitations that arise from the use of behavior trees in the context of the automatic modular design of collective behaviors in swarm robotics. To do so, we introduce Maple, an automatic design method that combines predefined modules-low-level behaviors and conditions-into a behavior tree that encodes the individual behavior of each robot of the swarm. We present three empirical studies based on two missions: aggregation and Foraging. To explore the strengths and weaknesses of adopting behavior trees as a control architecture, we compare Maple with Chocolate, a previously proposed automatic design method that uses probabilistic finite state machines instead. In the first study, we assess Maple's ability to produce control software that crosses the reality gap satisfactorily. In the second study, we investigate Maple's performance as a function of the design budget, that is, the maximum number of simulation runs that the design process is allowed to perform. In the third study, we explore a number of possible variants of Maple that differ in the constraints imposed on the structure of the behavior trees generated. The results of the three studies indicate that, in the context of swarm robotics, behavior trees might be appealing but in many settings do not produce better solutions than finite state machines.

Can Signal Delay be Functional? Including Delay in Evolved Robot Controllers

Engineers, control theorists, and neuroscientists often view the delay imposed by finite signal propagation velocities as a problem that needs to be compensated for or avoided. In this article, we consider the alternative possibility that in some cases, signal delay can be used functionally, that is, as an essential component of a cognitive system. To investigate this idea, we evolve a minimal robot controller to solve a basic stimulus-distinction task. The controller is constrained so that the solution must utilize a delayed recurrent signal. Different from previous evolutionary robotics studies, our controller is modeled using delay differential equations, which (unlike the ordinary differential equations of conventional continuous-time recurrent neural networks) can accurately capture delays in signal propagation. We analyze the evolved controller and its interaction with its environment using classical dynamical systems techniques. The analysis shows what kinds of invariant sets underlie the various successful and unsuccessful performances of the robot, and what kinds of bifurcations produce these invariant sets. In the second phase of our analysis, we turn our attention to the parameter θ, which describes the amount of signal delay included in the model. We show how the delay destabilizes certain attractors that would exist if there were no delay and creates other stable attractors, resulting in an agent that performs well at the target task.

Exploring Bipedal Hopping through Computational Evolution

Bipedal hopping is an efficient form of locomotion, yet it remains relatively rare in the natural world. Previous research has suggested that the tail balances the angular momentum of the legs to produce steady state bipedal hopping. In this study, we employ a 3D physics simulation engine to optimize gaits for an animat whose control and morphological characteristics are subject to computational evolution, which emulates properties of natural evolution. Results indicate that the order of gene fixation during the evolutionary process influences whether a bipedal hopping or quadrupedal bounding gait emerges. Furthermore, we found that in the most effective bipedal hoppers the tail balances the angular momentum of the torso, rather than the legs as previously thought. Finally, there appears to be a specific range of tail masses, as a proportion of total body mass, wherein the most effective bipedal hoppers evolve.

Information-theoretic decomposition of embodied and situated systems

The embodied and situated view of cognition stresses the importance of real-time and nonlinear bodily interaction with the environment for developing concepts and structuring knowledge. In this article, populations of robots controlled by an artificial neural network learn a wall-following task through artificial evolution. At the end of the evolutionary process, time series are recorded from perceptual and motor neurons of selected robots. Information-theoretic measures are estimated on pairings of variables to unveil nonlinear interactions that structure the agent-environment system. Specifically, the mutual information is utilized to quantify the degree of dependence and the transfer entropy to detect the direction of the information flow. Furthermore, the system is analyzed with the local form of such measures, thus capturing the underlying dynamics of information. Results show that different measures are interdependent and complementary in uncovering aspects of the robots' interaction with the environment, as well as characteristics of the functional neural structure. Therefore, the set of information-theoretic measures provides a decomposition of the system, capturing the intricacy of nonlinear relationships that characterize robots' behavior and neural dynamics.

Real-World Evolution of Robot Morphologies: A Proof of Concept

Evolutionary robotics using real hardware has been almost exclusively restricted to evolving robot controllers, but the technology for evolvable morphologies is advancing quickly. We discuss a proof-of-concept study to demonstrate real robots that can reproduce. Following a general system plan, we implement a robotic habitat that contains all system components in the simplest possible form. We create an initial population of two robots and run a complete life cycle, resulting in a new robot, parented by the first two. Even though the individual steps are simplified to the maximum, the whole system validates the underlying concepts and provides a generic workflow for the creation of more complex incarnations. This hands-on experience provides insights and helps us elaborate on interesting research directions for future development.

Evolutionary Developmental Robotics: Improving Morphology and Control of Physical Robots

Evolutionary algorithms have previously been applied to the design of morphology and control of robots. The design space for such tasks can be very complex, which can prevent evolution from efficiently discovering fit solutions. In this article we introduce an evolutionary-developmental (evo-devo) experiment with real-world robots. It allows robots to grow their leg size to simulate ontogenetic morphological changes, and this is the first time that such an experiment has been performed in the physical world. To test diverse robot morphologies, robot legs of variable shapes were generated during the evolutionary process and autonomously built using additive fabrication. We present two cases with evo-devo experiments and one with evolution, and we hypothesize that the addition of a developmental stage can be used within robotics to improve performance. Moreover, our results show that a nonlinear system-environment interaction exists, which explains the nontrivial locomotion patterns observed. In the future, robots will be present in our daily lives, and this work introduces for the first time physical robots that evolve and grow while interacting with the environment.

Evolution of Joint-Level Control for Quadrupedal Locomotion

We investigate a hierarchical approach to robot control inspired by joint-level control in animals. The method combines a high-level controller, consisting of an artificial neural network (ANN), with joint-level controllers based on digital muscles. In the digital muscle model (DMM), morphological and control aspects of joints evolve concurrently, emulating the musculoskeletal system of natural organisms. We introduce and compare different approaches for connecting outputs of the ANN to DMM-based joints. We also compare the performance of evolved animats with ANN-DMM controllers with those governed by only high-level (ANN-only) and low-level (DMM-only) controllers. These results show that DMM-based systems outperform their ANN-only counterparts while also exhibiting less complex ANNs in terms of the number of connections and neurons. The main contribution of this work is to explore the evolution of artificial systems where, as in natural organisms, some aspects of control are realized at the joint level.

Online Gait Learning for Modular Robots with Arbitrary Shapes and Sizes

Evolutionary robotics using real hardware is currently restricted to evolving robot controllers, but the technology for evolvable morphologies is advancing quickly. Rapid prototyping (3D printing) and automated assembly are the main enablers of robotic systems where robot offspring can be produced based on a blueprint that specifies the morphologies and the controllers of the parents. This article addresses the problem of gait learning in newborn robots whose morphology is unknown in advance. We investigate a reinforcement learning method and conduct simulation experiments using robot morphologies with different size and complexity. We establish that reinforcement learning does the job well and that it outperforms two alternative algorithms. The experiments also give insights into the online dynamics of gait learning and into the influence of the size, shape, and morphological complexity of the modular robots. These insights can potentially be used to predict the viability of modular robotic organisms before they are constructed.

Behavioural plasticity in evolving robots

In this paper, we show how the development of plastic behaviours, i.e., behaviour displaying a modular organisation characterised by behavioural subunits that are alternated in a context-dependent manner, can enable evolving robots to solve their adaptive task more efficiently also when it does not require the accomplishment of multiple conflicting functions. The comparison of the results obtained in different experimental conditions indicates that the most important prerequisites for the evolution of behavioural plasticity are: the possibility to generate and perceive affordances (i.e., opportunities for behaviour execution), the possibility to rely on flexible regulatory processes that exploit both external and internal cues, and the possibility to realise smooth and effective transitions between behaviours.

Open Issues in Evolutionary Robotics

One of the long-term goals in evolutionary robotics is to be able to automatically synthesize controllers for real autonomous robots based only on a task specification. While a number of studies have shown the applicability of evolutionary robotics techniques for the synthesis of behavioral control, researchers have consistently been faced with a number of issues preventing the widespread adoption of evolutionary robotics for engineering purposes. In this article, we review and discuss the open issues in evolutionary robotics. First, we analyze the benefits and challenges of simulation-based evolution and subsequent deployment of controllers versus evolution on real robotic hardware. Second, we discuss specific evolutionary computation issues that have plagued evolutionary robotics: (1) the bootstrap problem, (2) deception, and (3) the role of genomic encoding and genotype-phenotype mapping in the evolution of controllers for complex tasks. Finally, we address the absence of standard research practices in the field. We also discuss promising avenues of research. Our underlying motivation is the reduction of the current gap between evolutionary robotics and mainstream robotics, and the establishment of evolutionary robotics as a canonical approach for the engineering of autonomous robots.

Lessons from Speciation Dynamics: How to Generate Selective Pressure Towards Diversity

Recent approaches in evolutionary robotics (ER) propose to generate behavioral diversity in order to evolve desired behaviors more easily. These approaches require the definition of a behavioral distance, which often includes task-specific features and hence a priori knowledge. Alternative methods, which do not explicitly force selective pressure towards diversity (SPTD) but still generate it, are known from the field of artificial life, such as in artificial ecologies (AEs). In this study, we investigate how SPTD is generated without task-specific behavioral features or other forms of a priori knowledge and detect how methods of generating SPTD can be transferred from the domain of AE to ER. A promising finding is that in both types of systems, in systems from ER that generate behavioral diversity and also in the investigated speciation model, selective pressure is generated towards unpopulated regions of search space. In a simple case study we investigate the practical implications of these findings and point to options for transferring the idea of self-organizing SPTD in AEs to the domain of ER.

Investigating intertemporal choice through experimental evolutionary robotics

In intertemporal choices, subjects face a trade-off between value and delay: achieving the most valuable outcome requires a longer time, whereas the immediately available option is objectively poorer. Intertemporal choices are ubiquitous, and comparative studies reveal commonalities and differences across species: all species devalue future rewards as a function of delay (delay aversion), yet there is a lot of inter-specific variance in how rapidly such devaluation occurs. These differences are often interpreted in terms of ecological rationality, as depending on environmental factors (e.g., feeding ecology) and the physiological and morphological constraints of different species (e.g., metabolic rate). Evolutionary hypotheses, however, are hard to verify in vivo, since it is difficult to observe precisely enough real environments, not to mention ancestral ones. In this paper, we discuss the viability of an approach based on evolutionary robotics: in Study 1, we evolve robots without a metabolism in five different ecologies; in Study 2, we evolve metabolic robots (i.e., robots that consume energy over time) in three different ecologies. The intertemporal choices of the robots are analyzed both in their ecology and under laboratory conditions. Results confirm the generality of delay aversion and the usefulness of studying intertemporal choice through experimental evolutionary robotics.

How simple autonomous decisions evolve into robust behaviours? A review from neurorobotics, cognitive, self-organized and artificial immune systems fields

Researchers in diverse fields, such as in neuroscience, systems biology and autonomous robotics, have been intrigued by the origin and mechanisms for biological robustness. Darwinian evolution, in general, has suggested that adaptive mechanisms as a way of reaching robustness, could evolve by natural selection acting successively on numerous heritable variations. However, is this understanding enough for realizing how biological systems remain robust during their interactions with the surroundings? Here, we describe selected studies of bio-inspired systems that show behavioral robustness. From neurorobotics, cognitive, self-organizing and artificial immune system perspectives, our discussions focus mainly on how robust behaviors evolve or emerge in these systems, having the capacity of interacting with their surroundings. These descriptions are twofold. Initially, we introduce examples from autonomous robotics to illustrate how the process of designing robust control can be idealized in complex environments for autonomous navigation in terrain and underwater vehicles. We also include descriptions of bio-inspired self-organizing systems. Then, we introduce other studies that contextualize experimental evolution with simulated organisms and physical robots to exemplify how the process of natural selection can lead to the evolution of robustness by means of adaptive behaviors.