Matching Behaviours and Rewards

Modular Maximization Theory: A functional account of economic behavior in laboratory animal models with applications to drug-seeking behavior

Substance abuse research depends on precise and sensitive assessments of reinforcer efficacy in animal models. However, conventional methods often lack theoretical rigor and specificity to support these assessments. To address these gaps, the Modular Maximization Theory (MMT) is introduced as a comprehensive framework for understanding instrumental behavior. Like earlier maximization theories, MMT posits that behavior is distributed across alternatives to maximize utility over time. This concept is structured through five foundational postulates that define alternative actions and rules for choosing between them as budget constraints and utility functions. A key innovation of MMT is its incorporation of reinforcer utilization-encompassing both consummatory and post-consummatory activities-into the budget-constraint function. A model of ratio-schedule performance is developed under the assumption that utilization is proportional to demand, with utility represented as an additive power function of reinforcer magnitude. This model, termed PURSPU (Proportional Utilization, Ratio Schedule, Power Utility), effectively explains how reinforcer magnitude, response effort, non-contingent reinforcement, and income influence demand curves, behavior-output functions, dose-response relationships, and progressive-ratio breakpoints, while accounting for rate-dependent effects. The model also offers novel insights into choice behavior, including concurrent-schedule performance, income dependency, and delay discounting, as well as post-reinforcement pauses and run rates. Variations in budget constraints and utility functions are proposed as alternative models. Potential theoretical advancements, more targeted assessments of drug abuse liability, and the broader role of MMT in understanding human drug abuse are explored.

The role of rat prelimbic cortex in decision making

The frontal cortex plays a critical role in decision-making. One specific frontal area, the anterior cingulate cortex, has been identified as crucial for setting a threshold for how much evidence is needed before a choice is made (Domenech & Dreher, 2010). Threshold is a key concept in drift diffusion models, a popular framework used to understand decision-making processes. Here, we investigated the role of the prelimbic cortex, part of the rodent cingulate cortex, in decision making. Male and female rats learned to choose between stimuli associated with high and low value rewards. Females learned faster, were more selective in their responses, and integrated information about the stimuli more quickly. By contrast, males learned more slowly and showed a decrease in their decision thresholds during choice learning. Inactivating the prelimbic cortex in female and male rats sped up decision making without affecting choice accuracy. Drift diffusion modeling found selective effects of prelimbic cortex inactivation on the decision threshold, which was reduced with increasing doses of the GABA-A agonist muscimol. Stimulating the prelimbic cortex through mu opioid receptors slowed the animals' choice latencies and increased the decision threshold. These findings provide the first causal evidence that the prelimbic cortex directly influences decision processes. Additionally, they suggest possible sex-based differences in early choice learning.

How do animals weigh conflicting information about reward sources over time? Comparing dynamic averaging models

Optimal foraging theory suggests that animals make decisions which maximize their food intake per unit time when foraging, but the mechanisms animals use to track the value of behavioral alternatives and choose between them remain unclear. Several models for how animals integrate past experience have been suggested. However, these models make differential predictions for the occurrence of spontaneous recovery of choice: a behavioral phenomenon in which a hiatus from the experimental environment results in animals reverting to a behavioral allocation consistent with a reward distribution from the more distant past, rather than one consistent with their most recently experienced distribution. To explore this phenomenon and compare these models, three free-operant experiments with rats were conducted using a serial reversal design. In Phase 1, two responses (A and B) were baited with pellets on concurrent variable interval schedules, favoring option A. In Phase 2, lever baiting was reversed to favor option B. Rats then entered a delay period, where they were maintained at weight in their home cages and no experimental sessions took place. Following this delay, preference was assessed using initial responding in test sessions where levers were presented, but not baited. Models were compared in performance, including an exponentially weighted moving average, the Temporal Weighting Rule, and variants of these models. While the data provided strong evidence of spontaneous recovery of choice, the form and extent of recovery was inconsistent with the models under investigation. Potential interpretations are discussed in relation to both the decision rule and valuation functions employed.

Models of Conditioned Reinforcement and Abnormal Behaviour in Captive Animals

Abnormal behaviours are common in captive animals, and despite a lot of research, the development, maintenance and alleviation of these behaviours are not fully understood. Here, we suggest that conditioned reinforcement can induce sequential dependencies in behaviour that are difficult to infer from direct observation. We develop this hypothesis using recent models of associative learning that include conditioned reinforcement and inborn facets of behaviour, such as predisposed responses and motivational systems. We explore three scenarios in which abnormal behaviour emerges from a combination of associative learning and a mismatch between the captive environment and inborn predispositions. The first model considers how abnormal behaviours, such as locomotor stereotypies, may arise from certain spatial locations acquiring conditioned reinforcement value. The second model shows that conditioned reinforcement can give rise to abnormal behaviour in response to stimuli that regularly precede food or other reinforcers. The third model shows that abnormal behaviour can result from motivational systems being adapted to natural environments that have different temporal structures than the captive environment. We conclude that models including conditioned reinforcement offer an important theoretical insight regarding the complex relationships between captive environments, inborn predispositions, and learning. In the future, this general framework could allow us to further understand and possibly alleviate abnormal behaviours.

Choice and rate-amount independence in a titration procedure

The generalized matching law or Law of Allocation proposed by Baum (2018a, 2018b) potentially provides a broad conceptual framework within which to understand the allocation of time among activities. In its simplest form, the law incorporates power-function induction of activities by variables such as rate and amount of delivered inducers. Whether these variables affect allocation independently of one another is a central issue, because independence of the variables would allow simple multiplication of power functions and would make quantitative prediction simple too. The present experiment used a titration procedure to test the independence of rate and amount of food in determining pigeons' allocation of pecking between two keys. Amount ratio was varied within sessions to engender different peck ratios. Rate ratio was varied across two series of conditions. The results conformed to the predictions of the simple version of the Law of Allocation by strongly supporting independence of rate and amount. The Law of Allocation may have broad application for understanding activities in natural settings and everyday life.

A Theory of the Extinction Burst

A preliminary theory of a temporary increase in the rate of an operant response with the transition to extinction (i.e., the extinction burst) is proposed. The theory assumes reinforcers are events permitting access to some valuable activity, and that such activity can compete for allocation with the target response under some conditions (e.g., very high reinforcement rates). With the transition to extinction, elimination of this competition for allocation can produce an increase in the the target response, but the increase is transient because the value of the target response decreases with exposure to extinction. The theory provides a way to understand why the extinction burst is not ubiquitous, seems more common following very small ratio schedules, occurs for a short period of time following the transition to extinction, and may be eliminated with the availability of alternative reinforcement. It appears to provide a reasonable starting point for a theory of the extinction burst that does not necessarily require inclusion of invigorating effects of frustration, and it is closely aligned with Resurgence as Choice theory. Additional research on factors modulating reinforcement-related activities and how they affect the extinction burst could help to further evaluate the theory.

Rate matching, probability matching, and optimization in concurrent ratio schedules

Much research has documented rate matching in concurrent variable-interval schedules, but comparatively little research has examined performance in concurrent variable-ratio schedules, except in discrete-trials procedures that sometimes produce probability matching. One should expect that the two types of schedule would result in different performances, because ratio schedules cannot improve with time the way interval schedules do; ratio schedules lack the temporal dynamics of interval schedules. The present experiment exposed rats to concurrent variable-ratio schedules. Seven unsignaled components were presented in random order within each daily session, with probability ratios ranging from 1:8 to 8:1. Three conditions were studied that varied the overall probability of food while leaving probability ratios the same. Choice appeared to conform to probability matching, because sensitivities in the rate-matching relation were close to 0.5, whereas sensitivities to probability ratio were close to 1.0. The sensitivities alone, however, could not confirm probability matching, because undermatching to rate occurs often. Analyses at smaller time scales supported the interpretation of probability matching. In particular, control by food deliveries was highly local in concurrent variable-ratio schedules, in contrast with concurrent variable-interval schedules, in which control is extended. Activity continued to switch between alternatives throughout components, contradicting optimal sampling theory.

Switching costs in stochastic environments drive the emergence of matching behaviour in animal decision-making through the promotion of reward learning strategies

A principle of choice in animal decision-making named probability matching (PM) has long been detected in animals, and can arise from different decision-making strategies. Little is known about how environmental stochasticity may influence the switching time of these different decision-making strategies. Here we address this problem using a combination of behavioral and theoretical approaches, and show, that although a simple Win-Stay-Loss-Shift (WSLS) strategy can generate PM in binary-choice tasks theoretically, budgerigars (Melopsittacus undulates) actually apply a range of sub-tactics more often when they are expected to make more accurate decisions. Surprisingly, budgerigars did not get more rewards than would be predicted when adopting a WSLS strategy, and their decisions also exhibited PM. Instead, budgerigars followed a learning strategy based on reward history, which potentially benefits individuals indirectly from paying lower switching costs. Furthermore, our data suggest that more stochastic environments may promote reward learning through significantly less switching. We suggest that switching costs driven by the stochasticity of an environmental niche can potentially represent an important selection pressure associated with decision-making that may play a key role in driving the evolution of complex cognition in animals.