Resource- and Neighbor-Aware Observation Transmission Scheme in Satellite Networks

The observation satellite can exploit its own storage and computational resources to reduce the transmission delay. However, too excessive usage of these resources can have negative effects on the queuing delay at the relay satellite and/or on conducting other tasks at each observation satellite. In this paper, we proposed a new resource- and neighbor-aware observation transmission scheme (RNA-OTS). In RNA-OTS, each observation satellite decides whether to use its resources and the resources of the relay satellite at each time epoch by considering its resource utilization and transmission policies of neighbor observation satellites. For the optimal decision of each observation satellite in a distributed manner, the operation of observation satellites is modeled by means of a constrained stochastic game, and a best-response-dynamics-based algorithm is devised to find the Nash equilibrium. The evaluation results demonstrate that RNA-OTS can decrease the delay to deliver the observation to the destination by up to 87% compared to a relay-satellite-based scheme while guaranteeing a sufficiently low average utilization of the resources of the observation satellite.

On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

Learning enables adaptation in cooperation for multi-player stochastic games

Interactions among individuals in natural populations often occur in a dynamically changing environment. Understanding the role of environmental variation in population dynamics has long been a central topic in theoretical ecology and population biology. However, the key question of how individuals, in the middle of challenging social dilemmas (e.g. the 'tragedy of the commons'), modulate their behaviours to adapt to the fluctuation of the environment has not yet been addressed satisfactorily. Using evolutionary game theory, we develop a framework of stochastic games that incorporates the adaptive mechanism of reinforcement learning to investigate whether cooperative behaviours can evolve in the ever-changing group interaction environment. When the action choices of players are just slightly influenced by past reinforcements, we construct an analytical condition to determine whether cooperation can be favoured over defection. Intuitively, this condition reveals why and how the environment can mediate cooperative dilemmas. Under our model architecture, we also compare this learning mechanism with two non-learning decision rules, and we find that learning significantly improves the propensity for cooperation in weak social dilemmas, and, in sharp contrast, hinders cooperation in strong social dilemmas. Our results suggest that in complex social-ecological dilemmas, learning enables the adaptation of individuals to varying environments.

Caring for the future can turn tragedy into comedy for long-term collective action under risk of collapse

We will need collective action to avoid catastrophic climate change, and this will require valuing the long term as well as the short term. Shortsightedness and uncertainty have hindered progress in resolving this collective action problem and have been recognized as important barriers to cooperation among humans. Here, we propose a coupled social-ecological dilemma to investigate the interdependence of three well-identified components of this cooperation problem: 1) timescales of collapse and recovery in relation to time preferences regarding future outcomes, 2) the magnitude of the impact of collapse, and 3) the number of actors in the collective. We find that, under a sufficiently severe and time-distant collapse, how much the actors care for the future can transform the game from a tragedy of the commons into one of coordination, and even into a comedy of the commons in which cooperation dominates. Conversely, we also find conditions under which even strong concern for the future still does not transform the problem from tragedy to comedy. For a large number of participating actors, we find that the critical collapse impact, at which these game regime changes happen, converges to a fixed value of collapse impact per actor that is independent of the enhancement factor of the public good, which is usually regarded as the driver of the dilemma. Our results not only call for experimental testing but also help explain why polarization in beliefs about human-caused climate change can threaten global cooperation agreements.

Uptake of Diagnostic Tests by Livestock Farmers: A Stochastic Game Theory Approach

Game theory examines strategic decision-making in situations of conflict, cooperation, and coordination. It has become an established tool in economics, psychology and political science, and more recently has been applied to disease control. Used to examine vaccination uptake in human medicine, game theory shows that when vaccination is voluntary some individuals will choose to "free-ride" on the protection provided by others, resulting in insufficient coverage for control of a vaccine-preventable disease. Here, we use game theory to examine farmer uptake of a new diagnostic ELISA test for sheep scab-a highly infectious disease with an estimated cost exceeding £8M per year to the UK industry. The stochastic game models decisions made by neighboring farmers when deciding whether to adopt the newly available test, which can detect subclinical infestation. A key element of the stochastic game framework is that it allows multiple states. Depending on infestation status and test adoption decisions in the previous year, a farm may be at high, medium or low risk of infestation this year-a status which influences the decision the farmer makes and the farmer payoffs. Ultimately, each farmer's decision depends on the costs of using the diagnostic test vs. the benefits of enhanced disease control, which may only accrue in the longer term. The extent to which a farmer values short-term over long-term benefits reflects external factors such as inflation or individual characteristics such as patience. Our results show that when using realistic parameters and with a test cost around 50% more than the current clinical diagnosis, the test will be adopted in the high-risk state, but not in the low-risk state. For the medium risk state, test adoption will depend on whether the farmer takes a long-term or short-term view. We show that these outcomes are relatively robust to change in test costs and, moreover, that whilst the farmers adopting the test would not expect to see large gains in profitability, substantial reduction in sheep scab (and associated welfare implications) could be achieved in a cost-neutral way to the industry.

A Cross-Layer Routing Protocol Based on Quasi-Cooperative Multi-Agent Learning for Multi-Hop Cognitive Radio Networks

Transmission latency minimization and energy efficiency improvement are two main challenges in multi-hop Cognitive Radio Networks (CRN), where the knowledge of topology and spectrum statistics are hard to obtain. For this reason, a cross-layer routing protocol based on quasi-cooperative multi-agent learning is proposed in this study. Firstly, to jointly consider the end-to-end delay and power efficiency, a comprehensive utility function is designed to form a reasonable tradeoff between the two measures. Then the joint design problem is modeled as a Stochastic Game (SG), and a quasi-cooperative multi-agent learning scheme is presented to solve the SG, which only needs information exchange with previous nodes. To further enhance performance, experience replay is applied to the update of conjecture belief to break the correlations and reduce the variance of updates. Simulation results demonstrate that the proposed scheme is superior to traditional algorithms leading to a shorter delay, lower packet loss ratio and higher energy efficiency, which is close to the performance of an optimum scheme.