Policy search with rare significant events: Choosing the right partner to cooperate with

This paper focuses on a class of reinforcement learning problems where significant events are rare and limited to a single positive reward per episode. A typical example is that of an agent who has to choose a partner to cooperate with, while a large number of partners are simply not interested in cooperating, regardless of what the agent has to offer. We address this problem in a continuous state and action space with two different kinds of search methods: a gradient policy search method and a direct policy search method using an evolution strategy. We show that when significant events are rare, gradient information is also scarce, making it difficult for policy gradient search methods to find an optimal policy, with or without a deep neural architecture. On the other hand, we show that direct policy search methods are invariant to the rarity of significant events, which is yet another confirmation of the unique role evolutionary algorithms has to play as a reinforcement learning method.

Nothing better to do? Environment quality and the evolution of cooperation by partner choice

The effects of partner choice have been documented in a large number of biological systems such as sexual markets, interspecific mutualisms, or human cooperation. There are, however, a number of situations in which one would expect this mechanism to play a role, but where no such effect has ever been demonstrated. This is the case in particular in many intraspecific interactions, such as collective hunts, in non-human animals. Here we use individual-based simulations to solve this apparent paradox. We show that the conditions for partner choice to operate are in fact restrictive. They entail that individuals can compare social opportunities and choose the best. The challenge is that social opportunities are often rare because they necessitate the co-occurrence of (i) at least one available partner, and (ii) a resource to exploit together with this partner. This has three consequences. First, partner choice cannot lead to the evolution of cooperation when resources are scarce, which explains that this mechanism could never be observed in many cases of intraspecific cooperation in animals. Second, partner choice can operate when partners constitute in themselves a resource, which is the case in sexual interactions and interspecific mutualisms. Third, partner choice can lead to the evolution of cooperation when individuals live in a rich environment, and/or when they are highly efficient at extracting resources from their environment.