Yoked variable-ratio and variable-interval responding in pigeons

Prisoner's dilemma and the free operant: John Nash, I'd like you to meet Fred Skinner

In separate chambers, responding by two pairs of pigeons was reinforced under concurrent random-ratio schedules of reinforcement. For each pair, the birds' schedules were coupled in such a manner that left- and right-key reinforcement probabilities were determined by the key being pecked by the other pigeon of the pair. In this way, a reinforcement matrix, like that of the popular Prisoner's Dilemma game of game theory, was created. The responding of all subjects soon gravitated to the choice combination identified by the mathematician John Nash as the equilibrium of the Prisoner's Dilemma game. This was found both before and after reversal of contingencies on the keys. In a second experiment, with a single pair of pigeons, stimuli signaling the choice of the paired pigeon had little lasting effect: responding again gravitated to the game's equilibrium. The results affirm earlier findings, demonstrating that Skinner's principle of positive reinforcement, together with Nashian mathematics, entirely accounts for iterative game-theoretic behavior. They extend these findings to the so-called free operant: to schedules of reinforcement in which responding is not constrained by stimulus-response sequencing (i.e., a trials procedure). The coupled schedule of reinforcement introduced here offers significant promise for the experimental analysis of economic and social behaviors.

Food Restriction Level and Reinforcement Schedule Differentially Influence Behavior during Acquisition and Devaluation Procedures in Mice

Behavioral strategies are often classified based on whether reinforcer value controls reinforcement. Value-sensitive behaviors, in which animals update their actions when reinforcer value is changed, are classified as goal-directed; conversely, value-insensitive actions, where behavior remains consistent when the reinforcer is removed or devalued, are considered habitual. Basic reinforcement schedules can help to bias behavior toward either process: random ratio (RR) schedules are thought to promote the formation of goal-directed behaviors while random intervals (RIs) promote habitual control. However, how the schedule-specific features of these tasks interact with other factors that influence learning to control behavior has not been well characterized. Using male and female mice, we asked how distinct food restriction levels, a strategy often used to increase task engagement, interact with RR and RI schedules to control performance during task acquisition and devaluation procedures. We determined that food restriction level has a stronger effect on the behavior of mice following RR schedules compared with RI schedules, and that it promotes a decrease in response rate during devaluation procedures that is best explained by the effects of extinction rather than devaluation. Surprisingly, food restriction accelerated the decrease in response rates observed following devaluation across sequential extinction sessions, but not within a single session. Our results support the idea that the relationships between schedules and behavioral control strategies are not clear-cut and suggest that an animal's engagement in a task must be accounted for, together with the structure of reinforcement schedules, to appropriately interpret the cognitive underpinnings of behavior.

Instrumental behavior in humans is sensitive to the correlation between response rate and reward rate

Recent theories of instrumental behavior postulate that the correlation between response and reward rate is a critical factor in instrumental goal-directed performance. However, it is still not clear whether human actions can be sensitive to rate correlation. Using a novel within-subject design, participants were trained under ratio and interval contingencies of reinforcement matching both reward probabilities and reward rates between conditions. The impact of rate correlation on performance was evident in the higher performance observed under ratio contingencies for both types of matching. Moreover, there was no difference in performance between two classes of interval schedules with equivalent correlational properties but different reward probabilities. These results are discussed in terms of a recent dual-system model of instrumental behavior.

Evidence for a dissociation between causal beliefs and instrumental actions

Human experiments have demonstrated that instrumental performance of an action and the causal beliefs of the effectiveness of an action in producing a reward are correlated and controlled by the probability of an action leading to a reward. The animal literature, however, shows that instrumental performance under free-operant training differs even when the reward probabilities are matched while subjects undergo training under ratio or interval schedules of reward. In two experiments, we investigated whether causal beliefs would correlate with instrumental performance under ratio and interval schedules for matched reward probabilities. In both experiments, we found that performance was higher under ratio than under interval training. However, causal beliefs were similar between these two conditions despite these differences in instrumental performance. When reward probabilities were increased by experimental manipulations in Experiment 2, the causal beliefs increased but performance decreased with respect to Experiment 1. This is evidence that instrumental performance and causal action-reward attribution may not follow from the same psychological process under free-operant training.

A re-examination of responding on ratio and regulated-probability interval schedules

The higher response rates observed on ratio than on matched interval reward schedules has been attributed to the differential reinforcement of longer inter-response times (IRTs) on the interval contingency. Some data, however, seem to contradict this hypothesis, showing that the difference is still observed when the role of IRT reinforcement is neutralized by using a regulated-probability interval schedule (RPI). Given the mixed evidence for these predictions, we re-examined this hypothesis by training three groups of rats to lever press under ratio, interval and RPI schedules across two phases while matching reward rates within triads. At the end of the first phase, the master ratio and RPI groups responded at similar rates. In the second phase, an interval group yoked to the same master ratio group of the first phase responded at a lower rate than the RPI group. Post-hoc analysis showed comparable reward rates for master and yoked schedules. The experienced response-outcome rate correlations were likewise similar and approached zero as training progressed. We discuss these results in terms of a contemporary dual-system model of instrumental conditioning.

The Effect of the Instrumental Training Contingency on Susceptibility to Reinforcer Devaluation

Two experiments investigated performance of instrumental lever pressing by rats following post-conditioning devaluation of the sucrose reinforcer produced by establishing an aversion to it. In Experiment I rats responded less in an extinction test after being averted from the sucrose following training on a ratio schedule, but not following an equivalent amount of training on an interval schedule. This was true even though the devalued sucrose would not act as an effective reinforcer on either the ratio or interval schedule. Experiment II provided a further investigation of the insensitivity of interval responding to reinforcer devaluation by comparing test performance under simple extinction with responding when the devalued reinforcer was presented on either a response-contingent or non-contingent schedule during the test. Once again simple extinction performance was unaffected by prior reinforcer devaluation. Furthermore, neither non-contingent nor contingent presentations of the devalued reinforcer significantly depressed responding below the level seen in the extinction condition. Ratio, but not interval performance appears to be controlled by knowledge about the instrumental contingency that encodes specific properties of the training reinforcer.

Human instrumental performance in ratio and interval contingencies: A challenge for associative theory

Associative learning theories regard the probability of reinforcement as the critical factor determining responding. However, the role of this factor in instrumental conditioning is not completely clear. In fact, free-operant experiments show that participants respond at a higher rate on variable ratio than on variable interval schedules even though the reinforcement probability is matched between the schedules. This difference has been attributed to the differential reinforcement of long inter-response times (IRTs) by interval schedules, which acts to slow responding. In the present study, we used a novel experimental design to investigate human responding under random ratio (RR) and regulated probability interval (RPI) schedules, a type of interval schedule that sets a reinforcement probability independently of the IRT duration. Participants responded on each type of schedule before a final choice test in which they distributed responding between two schedules similar to those experienced during training. Although response rates did not differ during training, the participants responded at a lower rate on the RPI schedule than on the matched RR schedule during the choice test. This preference cannot be attributed to a higher probability of reinforcement for long IRTs and questions the idea that similar associative processes underlie classical and instrumental conditioning.

Delayed reinforcement and fixed-ratio performance

Effects of delayed reinforcement on fixed-ratio (FR) maintained responding of pigeons were investigated. In Experiments 1-3, the delay of reinforcement was increased across successive sessions until pigeons paused for 300 s. Both signaled and unsignaled delays were studied across different conditions. Overall response rates and run rates (timed from the first to last response of a ratio) decreased and postreinforcement pauses increased as delays increased in each experiment. As delays increased, the likelihood of pausing during a ratio run also increased. When these measures were plotted as a function of obtained delays, signaled delays had less of an effect on the above measures than did unsignaled ones. In Experiment 2, delays had a greater effect on the above measures than did a control condition arranging equivalent interreinforcer intervals to those accompanying the delays. Experiments 3 and 4 examined the generality of the effects obtained in the first two experiments. In Experiment 3, delays imposed on FR or yoked-interval schedules had similar behavioral effects. In Experiment 4, effects similar to those found in Experiments 1-3 for 1, 10, and 20-s delays imposed on FR 50 schedules were found when the FR requirement increased across sessions. Despite the different contingencies relating response rate and reinforcement rates on interval and ratio schedules, delays of reinforcement generally affect performance on these schedules similarly.

Briefly delayed reinforcement effects on variable-ratio and yoked-interval schedule performance

Most investigations of briefly delayed reinforcement have involved schedules that arrange a time-plus-response requirement. The present experiment examined whether briefly delaying reinforcement on schedules that have a ratio requirement differs from results with schedules that have a time-plus-response requirement. Four pigeons responded on a two-component multiple schedule. One component arranged a variable-ratio (VR) 50 and the other a variable-interval (VI) schedule in which the distribution of reinforcers was yoked to the preceding VR schedule. Across a series of conditions, delays were imposed in both schedules. These delays were brief (0.25- or 0.5-s) unsignaled delays and, as control conditions, a 5-s unsignaled delay and a 0.5-s delay signaled by a blackout of the chamber. In the yoked-VI component, the brief unsignaled delay increased response rates in six of nine opportunities and increased the proportion of short interresponse times (IRTs) (<0.4 s) in eight of nine opportunities. In the VR component, the brief unsignaled delay increased response rates and the proportion of short IRTs in only two of nine opportunities. For two of the three pigeons that were exposed to the 5-s unsignaled delay, response rates and the proportion of short IRTs decreased in both of the components. The 0.5-s signaled delay did not systematically change response rates nor did it change the distribution of short IRTs relative to the immediate reinforcement condition. The results replicate effects reported with time-based schedules and extend these observations by showing that changes commonly observed in VI performance with briefly delayed reinforcement are not characteristic of VR responding.

Some Factors Modulating the Strength of Resurgence After Extinction of an Instrumental Behavior

In resurgence, an operant behavior that has undergone extinction can return ("resurge") when a second operant that has replaced it itself undergoes extinction. The phenomenon may provide insight into relapse that may occur after incentive or contingency management therapies in humans. Three experiments with rats examined the impact of several variables on the strength of the resurgence effect. In each, pressing one lever (L1) was first reinforced and then extinguished while pressing a second, alternative, lever (L2) was now reinforced. When L2 responding was then itself extinguished, L1 responses resurged. Experiment 1 found that resurgence was especially strong after an extensive amount of L1 training (12 as opposed to 4 training sessions) and after L1 was reinforced on a random ratio schedule as opposed to a variable interval schedule that was matched on reinforcement rate. Experiment 2 found that after 12 initial sessions of L1 training, 4, 12, or 36 sessions of Phase 2 each allowed substantial (and apparently equivalent) resurgence. Experiment 3 found no effect of changing the identity of the reinforcer (from grain pellet to sucrose pellet or sucrose to grain) on the amount of resurgence. The results suggest that resurgence can be robust; in the natural world, an operant behavior with an extensive reinforcement history may still resurge after extensive incentive-based therapy. The results are discussed in terms of current explanations of the resurgence effect.

Discrimination of variable schedules is controlled by interresponse times proximal to reinforcement

In Experiment 1, food-deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable-interval schedule; (b) a tandem variable-interval, differential-reinforcement-of-low-rate schedule; or (c) a tandem variable-interval, differential-reinforcement-of-high-rate schedule. Completion of a sample-lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample-lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional-discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable-ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable-ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.

Making sense of sensitivity in the human operant literature

Human operant behavior is often said to be controlled by different variables or governed by different processes than nonhuman operant behavior. Support for this claim within the operant literature comes from data suggesting that human behavior is often insensitive to schedules of reinforcement to which nonhuman behavior has been sensitive. The data that evoke the use of the terms sensitivity and insensitivity, however, result from both between-species and within-subject comparisons. We argue that because sensitivity is synonymous with experimental control, conclusions about sensitivity are best demonstrated through within-subject comparisons. Further, we argue that even when sensitivity is assessed using within-subject comparisons of performance on different schedules of reinforcement, procedural differences between studies of different species may affect schedule performance in important ways. We extend this argument to age differences as well. We conclude that differences across populations are an occasion for more precise experimental analyses and that it is premature to conclude that human behavior is controlled by different processes than nonhuman behavior.

Role of stimulus-stimulus pairing in matching-to-sample procedure: cross-species comparison of humans and pigeons

The present experiment examined whether, in a matching-to-sample (MTS) procedure, a relation between two stimuli, a sample and a comparison, could be established as a result of just stimulus-stimulus pairing, even if back up reinforcers were never provided for the conditional relation between the sample and comparison stimuli, but rather only for the comparison stimulus. A procedure called "pseudo matching-to-sample" was used in which, when S1 was presented as a sample stimulus, two comparison stimuli (C1 and C2) were presented, and only responses to C1 were reinforced. Conversely, when S2 was presented, only responses to C3 (and not C4) were reinforced. In other words, organisms experiencing this procedure could discriminate C1 from C2, and C3 from C4, by simple discrimination without regard to the conditional sample stimuli. In order to examine cross-species differences, responding by humans in this procedure was compared to that by pigeons. Although the humans developed a discriminative function for the sample stimuli, that is, the humans' responding was affected by both the sample stimuli and the reinforcers, responding by the pigeons was affected solely by the reinforcers. These data suggest that, in this procedure, humans (but not pigeons) are able to learn relations among stimuli simply as a result of stimulus-stimulus pairing.

Contingency and stimulus change in chained schedules of reinforcement

Higher rates of pecking were maintained by pigeons in the middle component of three-component chained fixed-interval schedules than in that component of corresponding multiple schedules (two extinction components followed by a fixed-interval component). This rate difference did not occur in equivalent tandem and mixed schedules, in which a single stimulus was correlated with the three components. The higher rates in components of chained schedules demonstrate a reinforcing effect of the stimulus correlated with the next component; the acquired functions of this stimulus make the vocabulary of conditioned reinforcement appropriate. Problems in defining conditioned reinforcement arise not from difficulties in demonstrating reinforcing effects but from disagreements about which experimental operations allow such reinforcing effects to be called conditioned.

Response-rate differences in variable-interval and variable-ratio schedules: An old problem revisited

In Experiment 1, a variable-ratio 10 schedule became, successively, a variable-interval schedule with only the minimum interreinforcement intervals yoked to the variable ratio, or a variable-interval schedule with both interreinforcement intervals and reinforced interresponse times yoked to the variable ratio. Response rates in the variable-interval schedule with both interreinforcement interval and reinforced interresponse time yoking fell between the higher rates maintained by the variable-ratio schedule and the lower rates maintained by the variable-interval schedule with only interreinforcement interval yoking. In Experiment 2, a tandem variable-interval 15-s variable-ratio 5 schedule became a yoked tandem variable-ratio 5 variable-interval x-s schedule, and a tandem variable-interval 30-s variable-ratio 10 schedule became a yoked tandem variable-ratio 10 variable-interval x-s schedule. In the yoked tandem schedules, the minimum interreinforcement intervals in the variable-interval components were those that equated overall interreinforcement times in the two phases. Response rates did not decline in the yoked schedules even when the reinforced interresponse times became longer. Experiment 1 suggests that both reinforced interresponse times and response rate-reinforcement rate correlations determine response-rate differences in variable-ratio 10 and yoked variable-interval schedules in rats. Experiment 2 suggests a minimal role for the reinforced interresponse time in determining response rates on tandem variable-interval 30-s variable-ratio 10 and yoked tandem variable-ratio 10 variable-interval x-s schedules in rats.

Instructed versus shaped human verbal behavior: Interactions with nonverbal responding

Undergraduate students' presses on left and right buttons occasionally made available points exchangeable for money. Blue lights over the buttons were correlated with multiple random-ratio random-interval components; usually, the random-ratio schedule was assigned to the left button and the random-interval to the right. During interruptions on the multiple schedule, students filled out sentence-completion guess sheets (e.g., The way to earn points with the left button is to...). For different groups, guesses were shaped with differential points also worth money (e.g., successive approximations to "press fast" for the left button), or were instructed (e.g., Write "press slowly" for the left button), or were simply collected. Control of rate of pressing by guesses was examined in individual cases by reversing shaped or instructed guesses, by instructing pressing rates, and/or by reversing multiple-schedule contingencies. Shaped guesses produced guess-consistent pressing even when guessed rates opposed those characteristic of the contingencies (e.g., slow random-ratio and fast random-interval rates), whereas guesses and rates of pressing rarely corresponded after unsuccessful shaping of guesses or when guessing had no differential consequences. Instructed guesses and pressing were inconsistently related. In other words, when verbal responses were shaped (contingency-governed), they controlled nonverbal responding. When they were instructed (rule-governed), their control of nonverbal responding was inconsistent: the verbal behavior sometimes controlled, sometimes was controlled by, and sometimes was independent of the nonverbal behavior.

Dissociation of value and response strength

Four pigeons were exposed to multiple schedules and later to concurrent-chains schedules, with terminal links that had previously been multiple-schedule components. For 2 birds, the terminal-link schedules arranged an inverse relationship between response rate and reinforcement rate; for the other 2 birds a direct corresponding relationship was arranged. Those response rates were further modified by differentially reinforcing either longer or shorter interresponse times, relative to the current means. Although the birds' initial-link responses indicated preferences for terminal links with higher rates of reinforcement, in half the cases the birds responded during the terminal links in such a way as to produce lower rates of reinforcement, rates their initial-link behavior indicated they did not prefer. That outcome is inconsistent with maximization theory, but consistent with a strengthening analysis of behavior on single-key schedules.

Performances on ratio and interval schedules of reinforcement: Data and theory

TWO DIFFERENCES BETWEEN RATIO AND INTERVAL PERFORMANCE ARE WELL KNOWN: (a) Higher rates occur on ratio schedules, and (b) ratio schedules are unable to maintain responding at low rates of reinforcement (ratio "strain"). A third phenomenon, a downturn in response rate at the highest rates of reinforcement, is well documented for ratio schedules and is predicted for interval schedules. Pigeons were exposed to multiple variable-ratio variable-interval schedules in which the intervals generated in the variable-ratio component were programmed in the variable-interval component, thereby "yoking" or approximately matching reinforcement in the two components. The full range of ratio performances was studied, from strained to continuous reinforcement. In addition to the expected phenomena, a new phenomenon was observed: an upturn in variable-interval response rate in the midrange of rates of reinforcement that brought response rates on the two schedules to equality before the downturn at the highest rates of reinforcement. When the average response rate was corrected by eliminating pausing after reinforcement, the downturn in response rate vanished, leaving a strictly monotonic performance curve. This apparent functional independence of the postreinforcement pause and the qualitative shift in response implied by the upturn in variable-interval response rate suggest that theoretical accounts will require thinking of behavior as partitioned among at least three categories, and probably four: postreinforcement activity, other unprogrammed activity, ratio-typical operant behavior, and interval-typical operant behavior.

Optimization and the matching law as accounts of instrumental behavior

The interaction between instrumental behavior and environment can be conveniently described at a molar level as a feedback system. Two different possible theories, the matching law and optimization, differ primarily in the reference criterion they suggest for the system. Both offer accounts of most of the known phenomena of performance on concurrent and single variable-interval and variable-ratio schedules. The matching law appears stronger in describing concurrent performances, whereas optimization appears stronger in describing performance on single schedules.

Performance of children under a multiple random-ratio random-interval schedule of reinforcement

Three children, aged 1.5, 2.5, and 4.5 years, pressed telegraph keys under a two-component multiple random-ratio random-interval schedule of reinforcement. In the first condition, responses on the left key were reinforced under a random-interval schedule and responses on the right key were reinforced under a random-ratio schedule. In the second condition, the schedule components were reversed. In the third condition, the original arrangement was reinstated. For all subjects, rates of responding were higher in the random-ratio component despite higher rates of reinforcement in the random-interval component. The average interreinforcement interval of the random-interval component was increased in the fourth condition, resulting in more similar rates of reinforcement for both schedule components, and then returned to its original value in the fifth condition. In both conditions, all subjects continued to exhibit higher rates of responding in the ratio component than in the interval component. Although these observations are consistent with results from studies with pigeons, it is argued that the response-rate differences between the interval and ratio schedule components are sufficient to demonstrate schedule sensitivity.

Single-sample discrimination of different schedules' reinforced interresponse times

Food-deprived rats in Experiment 1 responded to one of two tandem schedules that were, with equal probability, associated with a sample lever. The tandem schedules' initial links were different random-interval schedules. Their values were adjusted to approximate equality in time to completing each tandem schedule's response requirements. The tandem schedules differed in their terminal links: One reinforced short interresponse times; the other reinforced long ones. Tandem-schedule completion presented two comparison levers, one of which was associated with each tandem schedule. Pressing the lever associated with the sample-lever tandem schedule produced a food pellet. Pressing the other produced a blackout. The difference between terminal-link reinforced interresponse times varied across 10-trial blocks within a session. Conditional-discrimination accuracy increased with the size of the temporal difference between terminal-link reinforced interresponse times. In Experiment 2, one tandem schedule was replaced by a random ratio, while the comparison schedule was either a tandem schedule that only reinforced long interresponse times or a random-interval schedule. Again, conditional-discrimination accuracy increased with the temporal difference between the two schedules' reinforced interresponse times. Most rats mastered the discrimination between random ratio and random interval, showing that the interresponse times reinforced by these schedules can serve to discriminate between these schedules.

Differential outcomes enhance accuracy of delayed matching to sample but not resistance to change

Three experiments assessed the relation between the differential outcomes effect and resistance to change of delayed matching-to-sample performance. Pigeons produced delayed matching-to-sample trials by responding on variable interval schedules in two components of a multiple schedule. In the same-outcome component, the probability of reinforcement was the same for both samples (.9 in Experiments 1 and 2, .5 in Experiment 3); in the different-outcomes component, the probability of reinforcement was .9 for one sample and .1 for the other. In all three experiments, the forgetting functions in the different-outcomes component were higher and shallower than in the same-outcomes component. When total reinforcement was greater in the same-outcomes component (Experiments 1 and 2), resistance to disruption by prefeeding, intercomponent food, extinction, or flashing lights typically was greater in that component. In Experiment 3, when total reinforcement was equated, resistance to disruption was similar across components. Thus, the level and slope of forgetting functions depended on differential reinforcement correlated with the samples, but the resistance to change of forgetting functions depended on total reinforcement in a component. Both aspects of the results can be explained by a model of delayed matching to sample performance.

On the primacy of molecular processes in determining response rates under variable-ratio and variable-interval schedules

This study focused on variables that may account for response-rate differences under variable-ratio (VR) and variable-interval (VI) schedules of reinforcement. Four rats were exposed to VR, VI, tandem VI differential-reinforcement-of-high-rate, regulated-probability-interval, and negative-feedback schedules of reinforcement that provided the same rate of reinforcement. Response rates were higher under the VR schedule than the VI schedule, and the rates on all other schedules approximated those under the VR schedule. The median reinforced interresponse time (IRT) under the VI schedule was longer than for the other schedules. Thus, differences in reinforced IRTs correlated with differences in response rate, an outcome suggestive of the molecular control of response rate. This conclusion was complemented by the additional finding that the differences in molar reinforcement-feedback functions had little discernible impact on responding.

Schedule discrimination in a mixed schedule: implications for models of the variable-ratio, variable-interval rate difference

In Experiment 1, each of three humans knowledgeable about operant schedules used mouse clicks to respond to a "work key" presented on a monitor. On a random half of the presentations, work-key responses that completed a variable ratio (VR) 12 produced a tone. After five tones, the work key was replaced by two report keys. Pressing the right or left report key, respectively, added or subtracted yen50 from a counter and produced the work key. On the other half of the presentations, a variable interval (VI) associated with the work key was defined so its interreinforcer intervals approximated the time it took to complete the variable ratio. After five tone-producing completions of this schedule, the report keys were presented. Left or right report-key presses, respectively, added or subtracted yen50 from the counter. Subjects achieved high yen totals. In Experiment 2, the procedure was changed by requiring an interresponse time after completion of the variable interval that approximated the duration of the reinforced interresponse time on the variable ratio. Prior to beginning, subjects were shown how a sequence of response bouts and pauses could be used to predict schedule type. Subjects again achieved high levels of accuracy. These results show humans can discriminate ratio from interval schedules even when those schedules provide the same rate of reinforcement and reinforced interresponse times.

Cost, benefit, tonic, phasic: what do response rates tell us about dopamine and motivation?

The role of dopamine in decision making has received much attention from both the experimental and computational communities. However, because reinforcement learning models concentrate on discrete action selection and on phasic dopamine signals, they are silent as to how animals decide upon the rate of their actions, and they fail to account for the prominent effects of dopamine on response rates. We suggest an extension to reinforcement learning models in which response rates are optimally determined by balancing the tradeoff between the cost of fast responding and the benefit of rapid reward acquisition. The resulting behavior conforms well with numerous characteristics of free-operant responding. More importantly, this framework highlights a role for a tonic signal corresponding to the net rate of rewards, in determining the optimal rate of responding. We hypothesize that this critical quantity is conveyed by tonic levels of dopamine, explaining why dopaminergic manipulations exert a global affect on response rates. We further suggest that the effects of motivation on instrumental rates of responding are mediated through its influence on the net reward rate, implying a tight coupling between motivational states and tonic dopamine. The relationships between phasic and tonic dopamine signaling, and between directing and energizing effects of motivation, as well as the implications for motivational control of habitual and goal-directed instrumental action selection, are discussed.

Tonic dopamine: opportunity costs and the control of response vigor

RATIONALE

Dopamine neurotransmission has long been known to exert a powerful influence over the vigor, strength, or rate of responding. However, there exists no clear understanding of the computational foundation for this effect; predominant accounts of dopamine's computational function focus on a role for phasic dopamine in controlling the discrete selection between different actions and have nothing to say about response vigor or indeed the free-operant tasks in which it is typically measured.

OBJECTIVES

We seek to accommodate free-operant behavioral tasks within the realm of models of optimal control and thereby capture how dopaminergic and motivational manipulations affect response vigor.

METHODS

We construct an average reward reinforcement learning model in which subjects choose both which action to perform and also the latency with which to perform it. Optimal control balances the costs of acting quickly against the benefits of getting reward earlier and thereby chooses a best response latency.

RESULTS

In this framework, the long-run average rate of reward plays a key role as an opportunity cost and mediates motivational influences on rates and vigor of responding. We review evidence suggesting that the average reward rate is reported by tonic levels of dopamine putatively in the nucleus accumbens.

CONCLUSIONS

Our extension of reinforcement learning models to free-operant tasks unites psychologically and computationally inspired ideas about the role of tonic dopamine in striatum, explaining from a normative point of view why higher levels of dopamine might be associated with more vigorous responding.

The operant reserve: A computer simulation in (accelerated) real time

In Skinner's Reflex Reserve theory, reinforced responses added to a reserve depleted by responding. It could not handle the finding that partial reinforcement generated more responding than continuous reinforcement, but it would have worked if its growth had depended not just on the last response but also on earlier responses preceding a reinforcer, each weighted by delay. In that case, partial reinforcement generates steady states in which reserve decrements produced by responding balance increments produced when reinforcers follow responding. A computer simulation arranged schedules for responses produced with probabilities proportional to reserve size. Each response subtracted a fixed amount from the reserve and added an amount weighted by the reciprocal of the time to the next reinforcer. Simulated cumulative records and quantitative data for extinction, random-ratio, random-interval, and other schedules were consistent with those of real performances, including some effects of history. The model also simulated rapid performance transitions with changed contingencies that did not depend on molar variables or on differential reinforcement of inter-response times. The simulation can be extended to inhomogeneous contingencies by way of continua of reserves arrayed along response and time dimensions, and to concurrent performances and stimulus control by way of different reserves created for different response classes.

Facts, interpretations, and explanations: a review of Evelyn Fox Keller's Making sense of life

The job of a researcher is to explain the phenomenon that he or she is seeking to understand. To do this requires the accumulation of facts. These facts are then interpreted to arrive at explanations. However, individual researchers often interpret facts in different ways and arrive at disparate explanations. In her book, Making Sense of Life, Evelyn Fox Keller (2002) outlines various approaches used by developmental biologists to understand the animate systems we call life. In this review, I note several parallels between biology and behavior analysis in how facts are discovered, what is an acceptable interpretation of data, and how explanations are arrived at.

The effects of reinforcement frequency and response requirements on the maintenance of behavior

Six rats responded under fixed-interval and tandem fixed-interval fixed-ratio schedules of food reinforcement. Basic fixed-interval schedules alternated over experimental conditions with tandem fixed-interval fixed-ratio schedules with the same fixed-interval value. Fixed-interval length was varied within subjects over pairs of experimental conditions; the ratio requirement of the tandem schedules was varied across subjects. For both subjects with a ratio requirement of 10, overall response rates and running response rates typically were higher under the tandem schedules than under the corresponding basic fixed-interval schedules. For all subjects with ratio requirements of 30 or 60, overall response rates and running response rates were higher under the tandem schedules than under the corresponding basic fixed-interval schedules only with relatively short fixed intervals. At longer fixed intervals, higher overall response rates and running rates were maintained by the basic fixed-interval schedules than by the tandem schedules. These findings support Zeiler and Buchman's (1979) reinforcement-theory account of response strength as an increasing monotonic function of both the response requirement and reinforcement frequency. Small response requirements added in tandem to fixed-interval schedules have little effect on reinforcement frequency and so their net effect is to enhance responding. Larger response requirements reduce reinforcement frequency more substantially; therefore their net effect depends on the length of the fixed interval, which limits overall reinforcement frequency. At the longest fixed intervals studied in the present experiment, reinforcement frequency under the tandem schedules was sufficiently low that responding weakened or ceased altogether.

The linear system theory's account of behavior maintained by variable-ratio schedules

The mathematical theory of linear systems, which has been used successfully to describe behavior maintained by variable-interval schedules, is extended to describe behavior maintained by variable-ratio schedules. The result of the analysis is a pair of equations, one of which expresses response rate on a variable-ratio schedule as a function of the mean ratio requirement (n) that the schedule arranges. The other equation expresses response rate on a variable-ratio schedule as a function of reinforcement rate. Both equations accurately describe existing data from variable-ratio schedules. The theory accounts for two additional characteristics of behavior maintained by variable-ratio schedules; namely, the appearance of strained, two-valued (i.e., zero or very rapid) responding at large ns, and the abrupt cessation of responding at a boundary n. The theory also accounts for differences between behavior on variable-interval and variable-ratio schedules, including (a) the occurrence of strained responding on variable-ratio but not on variable-interval schedules, (b) the abrupt cessation of responding on occurrence of higher response rates on variable-ratio than on variable-interval schedules. Furthermore, given data from a series of variable-interval schedules and from a series of concurrent variable-ratio variable-interval schedules, the theory permits quantitative prediction of many properties of behavior on single-alternative variable-ratio schedules. The linear system theory's combined account of behavior on variable-interval and variable-ratio schedules is superior to existing versions of six other mathematical theories of variable-interval and variable-ratio responding.

Variable-ratio schedules as variable-interval schedules with linear feedback loops

The mathematical theory of linear systems has been used successfully to describe responding on variable-interval (VI) schedules. In the simplest extension of the theory to the variable-ratio (VR) case, VR schedules are treated as if they were VI schedules with linear feedback loops. The assumption entailed by this approach, namely, that VR and VI-plus-linear-feedback schedules are equivalent, was tested by comparing responding on the two types of schedule. Four human subjects' lever pressing produced monetary reinforcers on five VR schedules, and on five VI schedules with linear feedback loops that reproduced the feedback properties of the VR schedules. Pressing was initiated by instructions in 2 subjects, and was shaped by successive approximation in the other 2. The different methods of response initiation did not have differential effects on behavior. For each of the 4 subjects, the VR and the comparable VI-plus-linear-feedback schedules generated similar average response rates and similar response patterns. The subjects' behavior on both types of schedule was similar to that of avian and rodent species on VR schedules. These results indicate that the assumption entailed by the VI-plus-linear-feedback approach to the VR case is valid and, consequently, that the approach is worth pursuing. The results also confute interresponse-time theories of schedule performance, which require interval and ratio contingencies to produce different response rates.

Early life undernutrition and operant responding in the rat: the effect of the reinforcement schedule employed

Rats were either undernourished from birth to 43 days and thereafter well fed (previously undernourished, PU) or well nourished throughout. When behaviour was tested in adulthood it was found that significant differences between the groups in rate of lever-pressing for food occurred when they were tested under a variable-interval 60-sec schedule of reinforcement, but not when reward was delivered according to a fixed-interval 60-sec or variable-ratio ten schedule. The results of a second experiment suggested that the rate difference might reduce or disappear with extended exposure to the schedule. The third experiment exposed rats to fixed-interval 60-sec and mixed fixed-interval 10-sec fixed-interval 110 sec schedules. Response rate differences between the PU and control groups occurred only under the mixed schedule, a result interpreted as showing that temporal irregularity of reward delivery plays some role in the genesis of more rapid operant responding in PU rats. When rats received a larger variable-ratio schedule, requiring 40 responses for reward, no significant rate differences between the groups were found over the whole experimental condition. It is suggested that schedules on which there are significant differences have some special characteristic, possibly sensitivity to differences in food motivation between the groups.