Guiding Neuroevolution with Structural Objectives

Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning

Decentralization is a central characteristic of biological motor control that allows for fast responses relying on local sensory information. In contrast, the current trend of Deep Reinforcement Learning (DRL) based approaches to motor control follows a centralized paradigm using a single, holistic controller that has to untangle the whole input information space. This motivates to ask whether decentralization as seen in biological control architectures might also be beneficial for embodied sensori-motor control systems when using DRL. To answer this question, we provide an analysis and comparison of eight control architectures for adaptive locomotion that were derived for a four-legged agent, but with their degree of decentralization varying systematically between the extremes of fully centralized and fully decentralized. Our comparison shows that learning speed is significantly enhanced in distributed architectures-while still reaching the same high performance level of centralized architectures-due to smaller search spaces and local costs providing more focused information for learning. Second, we find an increased robustness of the learning process in the decentralized cases-it is less demanding to hyperparameter selection and less prone to becoming trapped in poor local minima. Finally, when examining generalization to uneven terrains-not used during training-we find best performance for an intermediate architecture that is decentralized, but integrates only local information from both neighboring legs. Together, these findings demonstrate beneficial effects of distributing control into decentralized units and relying on local information. This appears as a promising approach towards more robust DRL and better generalization towards adaptive behavior.

An Empirical Investigation on Evolutionary Algorithm Evolving Developmental Timings

Recently, evolutionary algorithms that can efficiently solve decomposable binary optimization problems have been developed. They are so-called model-based evolutionary algorithms, which build a model for generating solution candidates by applying a machine learning technique to a population. Their central procedure is linkage detection that reveals a problem structure, that is, how the entire problem consists of sub-problems. However, the model-based evolutionary algorithms have been shown to be ineffective for problems that do not have relevant structures or those whose structures are hard to identify. Therefore, evolutionary algorithms that can solve both types of problems quickly, reliably, and accurately are required. The objective of the paper is to investigate whether the evolutionary algorithm evolving developmental timings (EDT) that we previously proposed can be the desired one. The EDT makes some variables values more quickly converge than the remains for any problems, and then, decides values of the remains to obtain a higher fitness value under the fixation of the variables values. In addition, factors to decide which variable values converge more quickly, that is, developmental timings are evolution targets. Simulation results reveal that the EDT has worse performance than the linkage tree genetic algorithm (LTGA), which is one of the state-of-the-art model-based evolutionary algorithms, for decomposable problems and also that the difference in the performance between them becomes smaller for problems with overlaps among linkages and also that the EDT has better performance than the LTGA for problems whose structures are hard to identify. Those results suggest that an appropriate search strategy is different between decomposable problems and those hard to decompose.