Caring for the future can turn tragedy into comedy for long-term collective action under risk of collapse

Efficiency and resilience of cooperation in asymmetric social dilemmas

Direct reciprocity is a powerful mechanism for cooperation in social dilemmas. The very logic of reciprocity, however, seems to require that individuals are symmetric, and that everyone has the same means to influence each others' payoffs. Yet in many applications, individuals are asymmetric. Herein, we study the effect of asymmetry in linear public good games. Individuals may differ in their endowments (their ability to contribute to a public good) and in their productivities (how effective their contributions are). Given the individuals' productivities, we ask which allocation of endowments is optimal for cooperation. To this end, we consider two notions of optimality. The first notion focuses on the resilience of cooperation. The respective endowment distribution ensures that full cooperation is feasible even under the most adverse conditions. The second notion focuses on efficiency. The corresponding endowment distribution maximizes group welfare. Using analytical methods, we fully characterize these two endowment distributions. This analysis reveals that both optimality notions favor some endowment inequality: More productive players ought to get higher endowments. Yet the two notions disagree on how unequal endowments are supposed to be. A focus on resilience results in less inequality. With additional simulations, we show that the optimal endowment allocation needs to account for both the resilience and the efficiency of cooperation.

Facilitating cooperation in human-agent hybrid populations through autonomous agents

Cooperative AI has shown its effectiveness in solving the conundrum of cooperation. Understanding how cooperation emerges in human-agent hybrid populations is a topic of significant interest, particularly in the realm of evolutionary game theory. In this article, we scrutinize how cooperative and defective Autonomous Agents (AAs) influence human cooperation in social dilemma games with a one-shot setting. Focusing on well-mixed populations, we find that cooperative AAs have a limited impact in the prisoner's dilemma games but facilitate cooperation in the stag hunt games. Surprisingly, defective AAs can promote complete dominance of cooperation in the snowdrift games. As the proportion of AAs increases, both cooperative and defective AAs have the potential to cause human cooperation to disappear. We then extend our investigation to consider the pairwise comparison rule and complex networks, elucidating that imitation strength and population structure are critical for the emergence of human cooperation in human-agent hybrid populations.

Human-environment feedback and the consistency of proenvironmental behavior

Addressing global environmental crises such as anthropogenic climate change requires the consistent adoption of proenvironmental behavior by a large part of a population. Here, we develop a mathematical model of a simple behavior-environment feedback loop to ask how the individual assessment of the environmental state combines with social interactions to influence the consistent adoption of proenvironmental behavior, and how this feeds back to the perceived environmental state. In this stochastic individual-based model, individuals can switch between two behaviors, 'active' (or actively proenvironmental) and 'baseline', differing in their perceived cost (higher for the active behavior) and environmental impact (lower for the active behavior). We show that the deterministic dynamics and the stochastic fluctuations of the system can be approximated by ordinary differential equations and a Ornstein-Uhlenbeck type process. By definition, the proenvironmental behavior is adopted consistently when, at population stationary state, its frequency is high and random fluctuations in frequency are small. We find that the combination of social and environmental feedbacks can promote the spread of costly proenvironmental behavior when neither, operating in isolation, would. To be adopted consistently, strong social pressure for proenvironmental action is necessary but not sufficient-social interactions must occur on a faster timescale compared to individual assessment, and the difference in environmental impact must be small. This simple model suggests a scenario to achieve large reductions in environmental impact, which involves incrementally more active and potentially more costly behavior being consistently adopted under increasing social pressure for proenvironmentalism.

The effect of environmental information on evolution of cooperation in stochastic games

Many human interactions feature the characteristics of social dilemmas where individual actions have consequences for the group and the environment. The feedback between behavior and environment can be studied with the framework of stochastic games. In stochastic games, the state of the environment can change, depending on the choices made by group members. Past work suggests that such feedback can reinforce cooperative behaviors. In particular, cooperation can evolve in stochastic games even if it is infeasible in each separate repeated game. In stochastic games, participants have an interest in conditioning their strategies on the state of the environment. Yet in many applications, precise information about the state could be scarce. Here, we study how the availability of information (or lack thereof) shapes evolution of cooperation. Already for simple examples of two state games we find surprising effects. In some cases, cooperation is only possible if there is precise information about the state of the environment. In other cases, cooperation is most abundant when there is no information about the state of the environment. We systematically analyze all stochastic games of a given complexity class, to determine when receiving information about the environment is better, neutral, or worse for evolution of cooperation.

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches.

Outlearning extortioners: unbending strategies can foster reciprocal fairness and cooperation

Recent theory shows that extortioners taking advantage of the zero-determinant (ZD) strategy can unilaterally claim an unfair share of the payoffs in the Iterated Prisoner's Dilemma. It is thus suggested that against a fixed extortioner, any adapting coplayer should be subdued with full cooperation as their best response. In contrast, recent experiments demonstrate that human players often choose not to accede to extortion out of concern for fairness, actually causing extortioners to suffer more loss than themselves. In light of this, here we reveal fair-minded strategies that are unbending to extortion such that any payoff-maximizing extortioner ultimately will concede in their own interest by offering a fair split in head-to-head matches. We find and characterize multiple general classes of such unbending strategies, including generous ZD strategies and Win-Stay, Lose-Shift (WSLS) as particular examples. When against fixed unbending players, extortioners are forced with consequentially increasing losses whenever intending to demand a more unfair share. Our analysis also pivots to the importance of payoff structure in determining the superiority of ZD strategies and in particular their extortion ability. We show that an extortionate ZD player can be even outperformed by, for example, WSLS, if the total payoff of unilateral cooperation is smaller than that of mutual defection. Unbending strategies can be used to outlearn evolutionary extortioners and catalyze the evolution of Tit-for-Tat-like strategies out of ZD players. Our work has implications for promoting fairness and resisting extortion so as to uphold a just and cooperative society.

Coevolutionary dynamics via adaptive feedback in collective-risk social dilemma game

Human society and natural environment form a complex giant ecosystem, where human activities not only lead to the change in environmental states, but also react to them. By using collective-risk social dilemma game, some studies have already revealed that individual contributions and the risk of future losses are inextricably linked. These works, however, often use an idealistic assumption that the risk is constant and not affected by individual behaviors. Here, we develop a coevolutionary game approach that captures the coupled dynamics of cooperation and risk. In particular, the level of contributions in a population affects the state of risk, while the risk in turn influences individuals' behavioral decision-making. Importantly, we explore two representative feedback forms describing the possible effect of strategy on risk, namely, linear and exponential feedbacks. We find that cooperation can be maintained in the population by keeping at a certain fraction or forming an evolutionary oscillation with risk, independently of the feedback type. However, such evolutionary outcome depends on the initial state. Taken together, a two-way coupling between collective actions and risk is essential to avoid the tragedy of the commons. More importantly, a critical starting portion of cooperators and risk level is what we really need for guiding the evolution toward a desired direction.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Intrinsic fluctuations of reinforcement learning promote cooperation

In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with [Formula: see text]-greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner's dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents' action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.

A cultural evolutionary theory that explains both gradual and punctuated change

Cumulative cultural evolution (CCE) occurs among humans who may be presented with many similar options from which to choose, as well as many social influences and diverse environments. It is unknown what general principles underlie the wide range of CCE dynamics and whether they can all be explained by the same unified paradigm. Here, we present a scalable evolutionary model of discrete choice with social learning, based on a few behavioural science assumptions. This paradigm connects the degree of transparency in social learning to the human tendency to imitate others. Computer simulations and quantitative analysis show the interaction of three primary factors—information transparency, popularity bias and population size—drives the pace of CCE. The model predicts a stable rate of evolutionary change for modest degrees of popularity bias. As popularity bias grows, the transition from gradual to punctuated change occurs, with maladaptive subpopulations arising on their own. When the popularity bias gets too severe, CCE stops. This provides a consistent framework for explaining the rich and complex adaptive dynamics taking place in the real world, such as modern digital media.

Modelling coupled human-environment complexity for the future of the biosphere: strengths, gaps and promising directions

Humans and the environment form a single complex system where humans not only influence ecosystems but also react to them. Despite this, there are far fewer coupled human–environment system (CHES) mathematical models than models of uncoupled ecosystems. We argue that these coupled models are essential to understand the impacts of social interventions and their potential to avoid catastrophic environmental events and support sustainable trajectories on multi-decadal timescales. A brief history of CHES modelling is presented, followed by a review spanning recent CHES models of systems including forests and land use, coral reefs and fishing and climate change mitigation. The ability of CHES modelling to capture dynamic two-way feedback confers advantages, such as the ability to represent ecosystem dynamics more realistically at longer timescales, and allowing insights that cannot be generated using ecological models. We discuss examples of such key insights from recent research. However, this strength brings with it challenges of model complexity and tractability, and the need for appropriate data to parameterize and validate CHES models. Finally, we suggest opportunities for CHES models to improve human–environment sustainability in future research spanning topics such as natural disturbances, social structure, social media data, model discovery and early warning signals.

This article is part of the theme issue ‘Ecological complexity and the biosphere: the next 30 years’.

Evolutionary dynamics of zero-determinant strategies in repeated multiplayer games

Several studies have confirmed the existence of zero-determinant (ZD) strategies in repeated social dilemmas since Press and Dyson's ingenious discovery of ZD strategies in iterated prisoner's dilemmas. However, less research studies evolutionary performance of multiplayer ZD strategies, especially from a theoretical perspective. Here, we use a state-clustering method to theoretically analyze evolutionary dynamics of two representative ZD strategies: generous ZD strategies and extortionate ZD strategies. We consider two new settings for multiplayer ZD strategies: competitions with all ZD strategies and competitions with all memory-one strategies, apart from the competitions between these strategies and some classical ones. Moreover, we investigate the influence of the level of generosity and extortion on evolutionary dynamics of generous and extortionate ZD strategies, which was commonly ignored in previous studies. Theoretical results show that players with limited generosity are at an advantageous place and extortioners extorting more severely hold their ground more readily. Our results may provide new insights into better understanding evolutionary dynamics of ZD strategies in repeated multiplayer games.

Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability

Assessing the systemic effects of uncertainty that arises from agents' partial observation of the true states of the world is critical for understanding a wide range of scenarios, from navigation and foraging behavior to the provision of renewable resources and public infrastructures. Yet previous modeling work on agent learning and decision-making either lacks a systematic way to describe this source of uncertainty or puts the focus on obtaining optimal policies using complex models of the world that would impose an unrealistically high cognitive demand on real agents. In this work we aim to efficiently describe the emergent behavior of biologically plausible and parsimonious learning agents faced with partially observable worlds. Therefore we derive and present deterministic reinforcement learning dynamics where the agents observe the true state of the environment only partially. We showcase the broad applicability of our dynamics across different classes of partially observable agent-environment systems. We find that partial observability creates unintuitive benefits in several specific contexts, pointing the way to further research on a general understanding of such effects. For instance, partially observant agents can learn better outcomes faster, in a more stable way, and even overcome social dilemmas. Furthermore, our method allows the application of dynamical systems theory to partially observable multiagent leaning. In this regard we find the emergence of catastrophic limit cycles, a critical slowing down of the learning processes between reward regimes, and the separation of the learning dynamics into fast and slow directions, all caused by partial observability. Therefore, the presented dynamics have the potential to become a formal, yet practical, lightweight and robust tool for researchers in biology, social science, and machine learning to systematically investigate the effects of interacting partially observant agents.

Dynamical systems as a level of cognitive analysis of multi-agent learning: Algorithmic foundations of temporal-difference learning dynamics

A dynamical systems perspective on multi-agent learning, based on the link between evolutionary game theory and reinforcement learning, provides an improved, qualitative understanding of the emerging collective learning dynamics. However, confusion exists with respect to how this dynamical systems account of multi-agent learning should be interpreted. In this article, I propose to embed the dynamical systems description of multi-agent learning into different abstraction levels of cognitive analysis. The purpose of this work is to make the connections between these levels explicit in order to gain improved insight into multi-agent learning. I demonstrate the usefulness of this framework with the general and widespread class of temporal-difference reinforcement learning. I find that its deterministic dynamical systems description follows a minimum free-energy principle and unifies a boundedly rational account of game theory with decision-making under uncertainty. I then propose an on-line sample-batch temporal-difference algorithm which is characterized by the combination of applying a memory-batch and separated state-action value estimation. I find that this algorithm serves as a micro-foundation of the deterministic learning equations by showing that its learning trajectories approach the ones of the deterministic learning equations under large batch sizes. Ultimately, this framework of embedding a dynamical systems description into different abstraction levels gives guidance on how to unleash the full potential of the dynamical systems approach to multi-agent learning.

Coordination and equilibrium selection in games: the role of local effects

We study the role of local effects and finite size effects in reaching coordination and in equilibrium selection in two-player coordination games. We investigate three update rules - the replicator dynamics (RD), the best response (BR), and the unconditional imitation (UI). For the pure coordination game with two equivalent strategies we find a transition from a disordered state to coordination for a critical value of connectivity. The transition is system-size-independent for the BR and RD update rules. For the IU it is system-size-dependent, but coordination can always be reached below the connectivity of a complete graph. We also consider the general coordination game which covers a range of games, such as the stag hunt. For these games there is a payoff-dominant strategy and a risk-dominant strategy with associated states of equilibrium coordination. We analyse equilibrium selection analytically and numerically. For the RD and BR update rules mean-field predictions agree with simulations and the risk-dominant strategy is evolutionary favoured independently of local effects. When players use the unconditional imitation, however, we observe coordination in the payoff-dominant strategy. Surprisingly, the selection of pay-off dominant equilibrium only occurs below a critical value of the network connectivity and disappears in complete graphs. As we show, it is a combination of local effects and update rule that allows for coordination on the payoff-dominant strategy.

Cooperation dynamics under pandemic risks and heterogeneous economic interdependence

The spread of COVID-19 and ensuing containment measures have accentuated the profound interdependence among nations or regions. This has been particularly evident in tourism, one of the sectors most affected by uncoordinated mobility restrictions. The impact of this interdependence on the tendency to adopt less or more restrictive measures is hard to evaluate, more so if diversity in economic exposures to citizens' mobility are considered. Here, we address this problem by developing an analytical and computational game-theoretical model encompassing the conflicts arising from the need to control the economic effects of global risks, such as in the COVID-19 pandemic. The model includes the individual costs derived from severe restrictions imposed by governments, including the resulting economic interdependence among all the parties involved in the game. By using tourism-based data, the model is enriched with actual heterogeneous income losses, such that every player has a different economic cost when applying restrictions. We show that economic interdependence enhances cooperation because of the decline in the expected payoffs by free-riding parties (i.e., those neglecting the application of mobility restrictions). Furthermore, we show (analytically and through numerical simulations) that these cross-exposures can transform the nature of the cooperation dilemma each region or country faces, modifying the position of the fixed points and the size of the basins of attraction that characterize this class of games. Finally, our results suggest that heterogeneity among regions may be used to leverage the impact of intervention policies by ensuring an agreement among the most relevant initial set of cooperators.

Evolution of cooperation in costly institutions exhibits Red Queen and Black Queen dynamics in heterogeneous public goods

Public goods are often subject to heterogeneous costs, such as the necessary costs to maintain the public goods infrastructure. However, the extent to which heterogeneity in participation cost can affect groups' ability to provide public goods is unclear. Here, by introducing a mathematical model, I show that when individuals face a costly institution and a free institution to perform a collective action task, the existence of a participation cost promotes cooperation in the costly institution. Despite paying for a participation cost, costly cooperators, who join the costly institution and cooperate, can outperform defectors who predominantly join a free institution. This promotes cooperation in the costly institution and can facilitate the evolution of cooperation in the free institution. For small profitability of the collective action, cooperation in a costly institution but not the free institution evolves. However, individuals are doomed to a winnerless red queen dynamics in which cooperators are unable to suppress defection. For large profitabilities, cooperation in both the costly and the free institution evolves. In this regime, cooperators with different game preferences complement each other to efficiently suppress defection in a black queen dynamic.

Stewardship of global collective behavior

Collective behavior provides a framework for understanding how the actions and properties of groups emerge from the way individuals generate and share information. In humans, information flows were initially shaped by natural selection yet are increasingly structured by emerging communication technologies. Our larger, more complex social networks now transfer high-fidelity information over vast distances at low cost. The digital age and the rise of social media have accelerated changes to our social systems, with poorly understood functional consequences. This gap in our knowledge represents a principal challenge to scientific progress, democracy, and actions to address global crises. We argue that the study of collective behavior must rise to a "crisis discipline" just as medicine, conservation, and climate science have, with a focus on providing actionable insight to policymakers and regulators for the stewardship of social systems.

A collective risk dilemma for tourism restrictions under the COVID-19 context

The current COVID-19 pandemic has impacted millions of people and the global economy. Tourism has been one the most affected economic sectors because of the mobility restrictions established by governments and uncoordinated actions from origin and destination regions. The coordination of restrictions and reopening policies could help control the spread of virus and enhance economies, but this is not an easy endeavor since touristic companies, citizens, and local governments have conflicting interests. We propose an evolutionary game model that reflects a collective risk dilemma behind these decisions. To this aim, we represent regions as players, organized in groups; and consider the perceived risk as a strict lock-down and null economic activity. The costs for regions when restricting their mobility are heterogeneous, given that the dependence on tourism of each region is diverse. Our analysis shows that, for both large populations and the EU NUTS2 case study, the existence of heterogeneous costs enhances global agreements. Furthermore, the decision on how to group regions to maximize the regions' agreement of the population is a relevant issue for decision makers to consider. We find out that a layout of groups based on similar costs of cooperation boosts the regions' agreements and avoid the risk of having a total lock-down and a negligible tourism activity. These findings can guide policy makers to facilitate agreements among regions to maximize the tourism recovery.

Learning enables adaptation in cooperation for multi-player stochastic games

Interactions among individuals in natural populations often occur in a dynamically changing environment. Understanding the role of environmental variation in population dynamics has long been a central topic in theoretical ecology and population biology. However, the key question of how individuals, in the middle of challenging social dilemmas (e.g. the 'tragedy of the commons'), modulate their behaviours to adapt to the fluctuation of the environment has not yet been addressed satisfactorily. Using evolutionary game theory, we develop a framework of stochastic games that incorporates the adaptive mechanism of reinforcement learning to investigate whether cooperative behaviours can evolve in the ever-changing group interaction environment. When the action choices of players are just slightly influenced by past reinforcements, we construct an analytical condition to determine whether cooperation can be favoured over defection. Intuitively, this condition reveals why and how the environment can mediate cooperative dilemmas. Under our model architecture, we also compare this learning mechanism with two non-learning decision rules, and we find that learning significantly improves the propensity for cooperation in weak social dilemmas, and, in sharp contrast, hinders cooperation in strong social dilemmas. Our results suggest that in complex social-ecological dilemmas, learning enables the adaptation of individuals to varying environments.

Timing Uncertainty in Collective Risk Dilemmas Encourages Group Reciprocation and Polarization

Social dilemmas are often shaped by actions involving uncertain returns only achievable in the future, such as climate action or voluntary vaccination. In this context, uncertainty may produce non-trivial effects. Here, we assess experimentally — through a collective risk dilemma — the effect of timing uncertainty, i.e. how uncertainty about when a target needs to be reached affects the participants' behaviors. We show that timing uncertainty prompts not only early generosity but also polarized outcomes, where participants' total contributions are distributed unevenly. Furthermore, analyzing participants' behavior under timing uncertainty reveals an increase in reciprocal strategies. A data-driven game-theoretical model captures the self-organizing dynamics underpinning these behavioral patterns. Timing uncertainty thus casts a shadow on the future that leads participants to respond early, whereas reciprocal strategies appear to be important for group success. Yet, the same uncertainty also leads to inequity and polarization, requiring the inclusion of new incentives handling these societal issues.

•

Timing uncertainty influences experimental observations in the collective risk game

•

It induces subjects to contribute earlier and in a polarized manner

•

Successful players adopt reciprocal strategies, responding in kind to past actions

•

Coordination gets more difficult under high timing uncertainty