Uncertainty drives deviations in normative foraging decision strategies

Foraging animals use dynamic Bayesian updating to model meta-uncertainty in environment representations

Foraging theory predicts animal behavior in many contexts. In patch-based foraging behaviors, the marginal value theorem (MVT) gives the optimal strategy for deterministic environments whose parameters are fully known to the forager. In natural settings, environmental parameters exhibit variability and are only partially known to the animal based on its experience, creating uncertainty. Models of uncertainty in foraging are well established. However, natural environments also exhibit unpredicted changes in their statistics. As a result, animals must ascertain whether the currently observed quality of the environment is consistent with their internal models, or whether something has changed, creating meta-uncertainty. Behavioral strategies for optimizing foraging behavior under meta-uncertainty, and their neural underpinnings, are largely unknown. Here, we developed a novel behavioral task and computational framework for studying patch-leaving decisions in head-fixed and freely moving mice in conditions of meta-uncertainty. We stochastically varied between-patch travel time, as well as within-patch reward depletion rate. We find that, when uncertainty is minimal, mice adopt patch residence times in a manner consistent with the MVT and not explainable by simple ethologically motivated heuristic strategies. However, behavior in highly variable environments was best explained by modeling both first- and second-order uncertainty in environmental parameters, wherein local variability and global statistics are captured by a Bayesian estimator and dynamic prior, respectively. Thus, mice forage under meta-uncertainty by employing a hierarchical Bayesian strategy, which is essential for efficiently foraging in volatile environments. The results provide a foundation for understanding the neural basis of decision-making that exhibits naturalistic meta-uncertainty.

Information-Seeking Decision Strategies Mitigate Risk in Dynamic, Uncertain Environments

To survive in dynamic and uncertain environments, individuals must develop effective decision strategies that balance information gathering and decision commitment. Models of such strategies often prioritize either optimizing tangible payoffs, like reward rate, or gathering information to support a diversity of (possibly unknown) objectives. However, our understanding of the relative merits of these two approaches remains incomplete, in part because direct comparisons have been limited to idealized, static environments that lack the dynamic complexity of the real world. Here we compared the performance of normative reward- and information-seeking strategies in a dynamic foraging task. Both strategies show similar transitions between exploratory and exploitative behaviors as environmental uncertainty changes. However, we find subtle disparities in the actions they take, resulting in meaningful performance differences: whereas reward-seeking strategies generate slightly more reward on average, information-seeking strategies provide more consistent and predictable outcomes. Our findings support the adaptive value of information-seeking behaviors that can mitigate risk with minimal reward loss.

Modulation by NPY/NPF-like receptor underlies experience-dependent, sexually dimorphic learning

The evolutionary paths taken by each sex within a given species sometimes diverge, resulting in behavioral differences. Given their distinct needs, the mechanism by which each sex learns from a shared experience is still an open question. Here, we reveal sexual dimorphism in learning: C. elegans males do not learn to avoid the pathogenic bacteria PA14 as efficiently and rapidly as hermaphrodites. Notably, neuronal activity following pathogen exposure was dimorphic: hermaphrodites generate robust representations, while males, in line with their behavior, exhibit contrasting representations. Transcriptomic and behavioral analysis revealed that the neuropeptide receptor npr-5, an ortholog of the mammalian NPY/NPF-like receptor, regulates male learning by modulating neuronal activity. Furthermore, we show the dependency of the males' decision-making on their sexual status and demonstrate the role of npr-5 as a modulator of incoming sensory cues. Taken together, these findings illustrate how neuromodulators drive sex-specific behavioral plasticity in response to a shared experience.

Serial pattern learning: The anticipation of worsening conditions by pigeons

In general, animals are known to be sensitive to the immediacy of reinforcers. That is, they are generally impulsive and outcomes that occur in the future are generally heavily discounted. Furthermore, they should prefer alternatives that provide reinforcers that require less rather than greater effort to obtain. In the present research, pigeons were given a choice between (1) obtaining reinforcers on a progressively more difficult schedule of reinforcement; starting with four pecks, then eight pecks, then 16 pecks, then 32 pecks, and finally 64 pecks on each trial, and (2) a color signaling a number of pecks for a single reinforcer: red = six, green = 11, blue = 23, or yellow = 45. If pigeons choose optimally, most of the time they should choose the progressive schedule to obtain five reinforcers rather than switch to a color to receive only one. However, if they are sensitive primarily to the number of pecks to the next reinforcer, they should choose the progressive schedule once before switching to red, twice before switching to green, three times before switching to blue, and four times before switching to yellow. Instead, they systematically switched too early. Rather than choose based on the rate of reinforcement or even based on the time or effort to the next reinforcer, they appear to anticipate that the progressive schedule is going to get more difficult, and they base their choice suboptimally on the serial pattern of the worsening progressive schedule.

Grocery Shopping Under Simplified Marginal Value Theorem Predictions

This study examined whether supermarkets can be considered patches in the marginal value theorem (MVT) sense despite their particular features and whether they are models of human food foraging in resource-dense conditions. On the basis of the MVT, the quantitative relationship between gains in the Euro and patch residence time was modeled as an exponential growth function toward an upper asymptote, allowing the choice of an optimal strategy under diminishing returns. N = 61 participants were interviewed about their current shopping trip and contextual variables at a German supermarket and provided data to estimate relevant model parameters. A nonlinear model of the patch residence time and resulting gain based on an exponential function was fitted via nonlinear orthogonal distance regression. The results generally revealed the relationships predicted by the model, with some uncertainty regarding the estimation of the upper asymptote due to a lack of data from participants with long residence times. Despite this limitation, the data support the applicability of the MVT-based model. The results show that approaches from optimal foraging theory, such as the MVT, can be used successfully to model human shopping behavior even when participants' verbal reports are used.

No matter what you do, travel is travel in visual foraging

In visual foraging, foragers collect multiple items from a series of visual displays (or "patches"). When the goal is to maximize the total or the rate of collection of target items, foragers must decide when to leave a depleted patch given that "traveling" from one patch to another incurs a temporal cost. In three experiments, we investigated whether the interposition of a secondary task during travel between patches in visual foraging altered patch-leaving behavior. Over the course of 10- or 30-minute experiments, participants foraged in simulated "berry patches" and traveled to the next patch at will. While they traveled, they either actively performed a secondary task or simply observed passing visual stimuli. Travel time was varied across conditions. The addition of a secondary task, regardless of its relevance to visual foraging, to traveling, or to both, did not impact patch-leaving times in the primary visual foraging task. In Experiment 1 and more weakly in Experiment 2, the patch-leaving decision was based on how long the travel took as predicted by the Marginal Value Theorem (MVT). In Experiment 3, however, patch-leaving did not depend on travel time. Participants 'overharvested' in a manner that suggests that they may have adopted rules different from those of MVT. Across all three experiments, patch-leaving did not depend on the nature of the travel.

Probabilistic inference and Bayesian-like estimation in animals: Empirical evidence

Animals often make decisions without perfect knowledge of environmental parameters like the quality of an encountered food patch or a potential mate. Theoreticians often assume animals make such decisions using a Bayesian updating process that combines prior information about the frequency distribution of resources in the environment with sample information from an encountered resource; such a process leads to decisions that maximize fitness, given the available information. I examine three aspects of empirical work that shed light on the idea that animals can make such decisions in a Bayesian-like manner. First, many animals are sensitive to variance differences in behavioral options, one metric used to characterize frequency distributions. Second, several species use information about the relative frequency of preferred versus nonpreferred items in different populations to make probabilistic inferences about samples taken from populations in a manner that results in maximizing the likelihood of obtaining a preferred reward. Third, the predictions of Bayesian models often match the behavior of individuals in two main approaches. One approach compares behavior to models that make different assumptions about how individuals estimate the quality of an environmental parameter. The patch exploitation behavior of nine species of birds and mammals has matched the predictions of Bayesian models. The other approach compares the behavior of individuals who learn, through experience, different frequency distributions of resources in their environment. The behavior of three bird species and bumblebees exploiting food patches and fruit flies selecting mates is influenced by their experience learning different frequency distributions of food and mates, respectively, in ways consistent with Bayesian models. These studies lend support to the idea that animals may combine prior and sample information in a Bayesian-like manner to make decisions under uncertainty, but additional work on a greater diversity of species is required to better understand the generality of this ability.

Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Foraging Under Uncertainty Follows the Marginal Value Theorem with Bayesian Updating of Environment Representations

Foraging theory has been a remarkably successful approach to understanding the behavior of animals in many contexts. In patch-based foraging contexts, the marginal value theorem (MVT) shows that the optimal strategy is to leave a patch when the marginal rate of return declines to the average for the environment. However, the MVT is only valid in deterministic environments whose statistics are known to the forager; naturalistic environments seldom meet these strict requirements. As a result, the strategies used by foragers in naturalistic environments must be empirically investigated. We developed a novel behavioral task and a corresponding computational framework for studying patch-leaving decisions in head-fixed and freely moving mice. We varied between-patch travel time, as well as within-patch reward depletion rate, both deterministically and stochastically. We found that mice adopt patch residence times in a manner consistent with the MVT and not explainable by simple ethologically motivated heuristic strategies. Critically, behavior was best accounted for by a modified form of the MVT wherein environment representations were updated based on local variations in reward timing, captured by a Bayesian estimator and dynamic prior. Thus, we show that mice can strategically attend to, learn from, and exploit task structure on multiple timescales simultaneously, thereby efficiently foraging in volatile environments. The results provide a foundation for applying the systems neuroscience toolkit in freely moving and head-fixed mice to understand the neural basis of foraging under uncertainty.

Effort Foraging Task reveals positive correlation between individual differences in the cost of cognitive and physical effort in humans

Effort-based decisions, in which people weigh potential future rewards against effort costs required to achieve those rewards involve both cognitive and physical effort, though the mechanistic relationship between them is not yet understood. Here, we use an individual differences approach to isolate and measure the computational processes underlying effort-based decisions and test the association between cognitive and physical domains. Patch foraging is an ecologically valid reward rate maximization problem with well-developed theoretical tools. We developed the Effort Foraging Task, which embedded cognitive or physical effort into patch foraging, to quantify the cost of both cognitive and physical effort indirectly, by their effects on foraging choices. Participants chose between harvesting a depleting patch, or traveling to a new patch that was costly in time and effort. Participants' exit thresholds (reflecting the reward they expected to receive by harvesting when they chose to travel to a new patch) were sensitive to cognitive and physical effort demands, allowing us to quantify the perceived effort cost in monetary terms. The indirect sequential choice style revealed effort-seeking behavior in a minority of participants (preferring high over low effort) that has apparently been missed by many previous approaches. Individual differences in cognitive and physical effort costs were positively correlated, suggesting that these are perceived and processed in common. We used canonical correlation analysis to probe the relationship of task measures to self-reported affect and motivation, and found correlations of cognitive effort with anxiety, cognitive function, behavioral activation, and self-efficacy, but no similar correlations with physical effort.

Understanding patch foraging strategies across development

Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.

Acetylcholine and noradrenaline enhance foraging optimality in humans

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Stochastic dynamics of social patch foraging decisions

Animals typically forage in groups. Social foraging can help animals avoid predation and decrease their uncertainty about the richness of food resources. Despite this, theoretical mechanistic models of patch foraging have overwhelmingly focused on the behavior of single foragers. In this study, we develop a mechanistic model that accounts for the behavior of individuals foraging together and departing food patches following an evidence accumulation process. Each individual's belief about patch quality is represented by a stochastically accumulating variable, which is coupled to another's belief to represent the transfer of information. We consider a cohesive group, and model information sharing by considering both intermittent pulsatile coupling (only communicate decision to leave) and continuous diffusive coupling (communicate throughout the deliberation process). Groups employing pulsatile coupling can obtain higher foraging efficiency, which depends more strongly on the coupling parameter compared to those using diffusive coupling. Conversely, groups using diffusive coupling are more robust to changes and heterogeneities in belief weighting and departure criteria. Efficiency is measured by a reward rate function that balances the amount of energy accumulated against the time spent in a patch, computed by solving an ordered first passage time problem for the patch departures of each individual. Using synthetic departure time data, we can distinguish between the two modes of communication and identify the model parameters. Our model establishes a social patch foraging framework to identify deliberative decision strategies and forms of social communication, and to allow model fitting to field data from foraging animal groups.