Stochastic dynamic programming illuminates the link between environment, physiology, and evolution

Spatial evolutionary public goods game theory applied to optimal resource allocation and defense strategies in herbaceous plants

Empirical evidence suggests that the attractiveness of a plant to herbivores can be affected by the investment in defense by neighboring plants, as well as investment in defense by the focal plant. Thus, the payoff for allocating to defense may not only be influenced by the frequency and intensity of herbivory but also by defense strategies employed by other plants in the environment. We use a combination of spatial evolutionary game theory and stochastic dynamic programming to predict the proportion of plants in the population investing in defense (cooperators) and the proportion of plants that do not (defectors). Our model accounts for metabolic costs of maintenance of stored resources when predicting optimal resource allocation to growth, reproduction, and storage; this cost is not commonly accounted for in previous models. For both annual and perennial plants, our model predicts an evolutionarily stable proportion of cooperators and defectors (mixed stable strategy), but the proportion of cooperators is higher in a population of perennial plants than in a population of annual plants. We also show that including a metabolic cost of maintaining stored resources does not change the proportion of cooperators but does decrease plant fitness and allocation to overwinter storage.

Size-dependence of food intake and mortality interact with temperature and seasonality to drive diversity in fish life histories

Understanding how growth and reproduction will adapt to changing environmental conditions is a fundamental question in evolutionary ecology, but predicting the responses of specific taxa is challenging. Analyses of the physiological effects of climate change upon life history evolution rarely consider alternative hypothesized mechanisms, such as size-dependent foraging and the risk of predation, simultaneously shaping optimal growth patterns. To test for interactions between these mechanisms, we embedded a state-dependent energetic model in an ecosystem size-spectrum to ask whether prey availability (foraging) and risk of predation experienced by individual fish can explain observed diversity in life histories of fishes. We found that asymptotic growth emerged from size-based foraging and reproductive and mortality patterns in the context of ecosystem food web interactions. While more productive ecosystems led to larger body sizes, the effects of temperature on metabolic costs had only small effects on size. To validate our model, we ran it for abiotic scenarios corresponding to the ecological lifestyles of three tuna species, considering environments that included seasonal variation in temperature. We successfully predicted realistic patterns of growth, reproduction, and mortality of all three tuna species. We found that individuals grew larger when environmental conditions varied seasonally, and spawning was restricted to part of the year (corresponding to their migration from temperate to tropical waters). Growing larger was advantageous because foraging and spawning opportunities were seasonally constrained. This mechanism could explain the evolution of gigantism in temperate tunas. Our approach addresses variation in food availability and individual risk as well as metabolic processes and offers a promising approach to understand fish life-history responses to changing ocean conditions.

The influence of prey availability on behavioral decisions and reproductive success of a central-place forager during lactation

Marine central-place foragers are increasingly faced with altered prey landscapes, necessitating predictions of the impact of such changes on behavior, reproductive success, and population dynamics. We used state-dependent behavioral life history theory implemented via Stochastic Dynamic Programming (SDP) to explore the influence of changes in prey distribution and energy gain from foraging on the behavior and reproductive success of a central place forager during lactation. Our work is motivated by northern fur seals (Callorhinus ursinus) because of the ongoing population decline of the Eastern Pacific stock and projected declines in biomass of walleye pollock (Gadus chalcogrammus), a key fur seal prey species in the eastern Bering Sea. We also explored how changes in female and pup metabolic rates, body size, and lactation duration affected model output to provide insight into traits that might experience selective pressure in response to reductions in prey availability. Simulated females adopted a central-place foraging strategy after an initial extended period spent on land (4.7-8.3 days). Trip durations increased as the high energy prey patch moved farther from land or when the energy gain from foraging decreased. Increases in trip duration adversely affected pup growth rates and wean mass despite attempts to compensate by increasing land durations. Metabolic rate changes had the largest impacts on pup wean mass, with reductions in a pup's metabolic rate allowing females to successfully forage at distances of 600+ km from land for up to 15+ days. Our results indicate that without physiological adaptations, a rookery is unlikely to be viable if the primary foraging grounds are 400 km or farther from the rookery. To achieve pup growth rates characteristic of a population experiencing rapid growth, model results indicate the primary foraging grounds need to be <150 km from the rookery.

Strategy maps: Generalised giving-up densities for optimal foraging

Finding a common currency for benefits and hazards is a major challenge in optimal foraging theory, often requiring complex computational methods. We present a new analytic approach that builds on the Marginal Value Theorem and giving-up densities while incorporating the nonlinear effect of predation risk. We map the space of all possible environments into strategy regions, each corresponding to a discrete optimal strategy. This provides a generalised quantitative measure of the trade-off between foraging rewards and hazards. This extends a classic optimal diet choice rule-of-thumb to incorporate the hazard of waiting for better resources to appear. We compare the dynamics of optimal decision-making for three foraging life-history strategies: One in which fitness accrues instantly, and two with delays before fitness benefit is accrued. Foragers with delayed-benefit strategies are more sensitive to predation risk than resource quality, as they stand to lose more fitness from a predation event than instant-accrual foragers.

Importance of interindividual interactions in eco-evolutionary population dynamics: The rise of demo-genetic agent-based models

The study of eco-evolutionary dynamics, that is of the intertwinning between ecological and evolutionary processes when they occur at comparable time scales, is of growing interest in the current context of global change. However, many eco-evolutionary studies overlook the role of interindividual interactions, which are hard to predict and yet central to selective values. Here, we aimed at putting forward models that simulate interindividual interactions in an eco-evolutionary framework: the demo-genetic agent-based models (DG-ABMs). Being demo-genetic, DG-ABMs consider the feedback loop between ecological and evolutionary processes. Being agent-based, DG-ABMs follow populations of interacting individuals with sets of traits that vary among the individuals. We argue that the ability of DG-ABMs to take into account the genetic heterogeneity-that affects individual decisions/traits related to local and instantaneous conditions-differentiates them from analytical models, another type of model largely used by evolutionary biologists to investigate eco-evolutionary feedback loops. Based on the review of studies employing DG-ABMs and explicitly or implicitly accounting for competitive, cooperative or reproductive interactions, we illustrate that DG-ABMs are particularly relevant for the exploration of fundamental, yet pressing, questions in evolutionary ecology across various levels of organization. By jointly modelling the effects of management practices and other eco-evolutionary processes on interindividual interactions and population dynamics, DG-ABMs are also effective prospective and decision support tools to evaluate the short- and long-term evolutionary costs and benefits of management strategies and to assess potential trade-offs. Finally, we provide a list of the recent practical advances of the ABM community that should facilitate the development of DG-ABMs.

Stochastic optimal switching model for migrating population dynamics

An optimal switching control formalism combined with the stochastic dynamic programming is, for the first time, applied to modelling life cycle of migrating population dynamics with non-overlapping generations. The migration behaviour between habitats is efficiently described as impulsive switching based on stochastic differential equations, which is a new standpoint for modelling the biological phenomenon. The population dynamics is assumed to occur so that the reproductive success is maximized under an expectation. Finding the optimal migration strategy ultimately reduces to solving an optimality equation of the quasi-variational type. We show an effective linkage between our optimality equation and the basic reproduction number. Our model is applied to numerical computation of optimal migration strategy and basic reproduction number of an amphidromous fish Plecoglossus altivelis altivelis in Japan as a target species.

Modeling optimal responses and fitness consequences in a changing Arctic

Animals must balance a series of costs and benefits while trying to maximize their fitness. For example, an individual may need to choose how much energy to allocate to reproduction versus growth, or how much time to spend on vigilance versus foraging. Their decisions depend on complex interactions between environmental conditions, behavioral plasticity, reproductive biology, and energetic demands. As animals respond to novel environmental conditions caused by climate change, the optimal decisions may shift. Stochastic dynamic programming provides a flexible modeling framework with which to explore these trade-offs, but this method has not yet been used to study possible changes in optimal trade-offs caused by climate change. We created a stochastic dynamic programming model capturing trade-off decisions required by an individual adult female polar bear (Ursus maritimus) as well as the fitness consequences of her decisions. We predicted optimal foraging decisions throughout her lifetime as well as the energetic thresholds below which it is optimal for her to abandon a reproductive attempt. To explore the effects of climate change, we shortened the spring feeding period by up to 3 weeks, which led to predictions of riskier foraging behavior and higher reproductive thresholds. The resulting changes in fitness may be interpreted as a best-case scenario, where bears adapt instantaneously and optimally to new environmental conditions. If the spring feeding period was reduced by 1 week, her expected fitness declined by 15%, and if reduced by 3 weeks, expected fitness declined by 68%. This demonstrates an effective way to explore a species' optimal response to a changing landscape of costs and benefits and highlights the fact that small annual effects can result in large cumulative changes in expected lifetime fitness.

Reproductive skipping as an optimal life history strategy in the southern elephant seal, Mirounga leonina

Intermittent breeding by which organisms skip some current reproductive opportunities in order to enhance future reproductive success is a common life history trade-off among long-lived, iteroparous species. The southern elephant seal Mirounga leonina engages in intermediate breeding when body condition is low. While it is anticipated that this strategy may increase the lifetime reproductive output of this species, the conditions under which reproductive skipping are predicted to occur are not clear. Here I develop a dynamic state variable model based on published data that examines when southern elephant seals are predicted to optimally skip reproduction in order to maximize lifetime reproductive output as a function of current body mass, maternal age, and survivorship. I demonstrate that the optimal reproductive strategy for this species can include reproductive skipping, and that the conditions where this is optimal depend on patterns of mass-dependent adult female survival. I further show that intermittent breeding can increase lifetime reproductive output, and that the magnitude of this benefit increases with the ability of individual animals to replenish depleted body mass through foraging. Finally, I show that when the environment is variable and foraging is reduced in bad years, the benefit of adopting an optimal strategy that includes reproductive skipping increases asymptotically with the frequency of bad years. These results highlight the importance of characterizing the pattern of adult survival in this species, as well as the need to identify other factors that may influence the prevalence and benefits of reproductive skipping.

Dynamics of starvation and recovery predict extinction risk and both Damuth's law and Cope's rule

The eco-evolutionary dynamics of species are fundamentally linked to the energetic constraints of their constituent individuals. Of particular importance is the interplay between reproduction and the dynamics of starvation and recovery. To elucidate this interplay, here we introduce a nutritional state-structured model that incorporates two classes of consumers: nutritionally replete, reproducing consumers, and undernourished, nonreproducing consumers. We obtain strong constraints on starvation and recovery rates by deriving allometric scaling relationships and find that population dynamics are typically driven to a steady state. Moreover, these rates fall within a "refuge" in parameter space, where the probability of population extinction is minimized. We also show that our model provides a natural framework to predict steady state population abundances known as Damuth's law, and maximum mammalian body size. By determining the relative stability of an otherwise homogeneous population to a competing population with altered percent body fat, this framework provides a principled mechanism for a selective driver of Cope's rule.

Enriching behavioral ecology with reinforcement learning methods

This article focuses on the division of labor between evolution and development in solving sequential, state-dependent decision problems. Currently, behavioral ecologists tend to use dynamic programming methods to study such problems. These methods are successful at predicting animal behavior in a variety of contexts. However, they depend on a distinct set of assumptions. Here, we argue that behavioral ecology will benefit from drawing more than it currently does on a complementary collection of tools, called reinforcement learning methods. These methods allow for the study of behavior in highly complex environments, which conventional dynamic programming methods do not feasibly address. In addition, reinforcement learning methods are well-suited to studying how biological mechanisms solve developmental and learning problems. For instance, we can use them to study simple rules that perform well in complex environments. Or to investigate under what conditions natural selection favors fixed, non-plastic traits (which do not vary across individuals), cue-driven-switch plasticity (innate instructions for adaptive behavioral development based on experience), or developmental selection (the incremental acquisition of adaptive behavior based on experience). If natural selection favors developmental selection, which includes learning from environmental feedback, we can also make predictions about the design of reward systems. Our paper is written in an accessible manner and for a broad audience, though we believe some novel insights can be drawn from our discussion. We hope our paper will help advance the emerging bridge connecting the fields of behavioral ecology and reinforcement learning.

Leaving safety to visit a feeding site: is it optimal to hesitate while exposed?

Animals living in complex environments experience differing risks of predation depending upon their location within the landscape. An animal could reduce the risk it experiences by remaining in a refuge site, but it may need to emerge from its refuge and enter more dangerous sites for feeding and other activities. Here, I consider the actions of an animal choosing to travel a short distance between a safe refuge and a dangerous foraging site, such as a bird leaving cover to visit a feeder. Although much work has been conducted examining the choice between a refuge and a foraging site when faced with a trade-off between starvation and predation risk, the work presented here is the first to consider the travel behaviour between these locations. Using state-dependent stochastic dynamic programming, I illustrate that there are several forms of optimal behaviour that can emerge. In some situations, the animal should choose to travel without stopping between sites, but in other cases, it is optimal for the animal to travel hesitantly towards the food, and to stop its travel at a point before it reaches the refuge. I discuss how this hesitant 'dawdling' behaviour may be optimal, and suggest further work to test these predictions.

Modelling and interpreting fish bioenergetics: a role for behaviour, life-history traits and survival trade-offs

Bioenergetics is used as the mechanistic foundation of many models of fishes. As the context of a model gradually extends beyond pure bioenergetics to include behaviour, life-history traits and function and performance of the entire organism, so does the need for complementing bioenergetic measurements with trade-offs, particularly those dealing with survival. Such a broadening of focus revitalized and expanded the domain of behavioural ecology in the 1980s. This review makes the case that a similar change of perspective is required for physiology to contribute to the types of predictions society currently demands, e.g. regarding climate change and other anthropogenic stressors.

Avoiding tipping points in fisheries management through Gaussian process dynamic programming

Model uncertainty and limited data are fundamental challenges to robust management of human intervention in a natural system. These challenges are acutely highlighted by concerns that many ecological systems may contain tipping points, such as Allee population sizes. Before a collapse, we do not know where the tipping points lie, if they exist at all. Hence, we know neither a complete model of the system dynamics nor do we have access to data in some large region of state space where such a tipping point might exist. We illustrate how a Bayesian non-parametric approach using a Gaussian process (GP) prior provides a flexible representation of this inherent uncertainty. We embed GPs in a stochastic dynamic programming framework in order to make robust management predictions with both model uncertainty and limited data. We use simulations to evaluate this approach as compared with the standard approach of using model selection to choose from a set of candidate models. We find that model selection erroneously favours models without tipping points, leading to harvest policies that guarantee extinction. The Gaussian process dynamic programming (GPDP) performs nearly as well as the true model and significantly outperforms standard approaches. We illustrate this using examples of simulated single-species dynamics, where the standard model selection approach should be most effective and find that it still fails to account for uncertainty appropriately and leads to population crashes, while management based on the GPDP does not, as it does not underestimate the uncertainty outside of the observed data.