Algorithmic shaping and misbehavior in the acquisition of token deposit by rats

Grounding applied animal behavior practices in the experimental analysis of behavior

Some of the earliest applications outside the laboratory of principles derived from the experimental analysis of behavior (EAB), such as the pioneering work of Keller and Marian Breland, involved animals. This translational tradition continues to the present as EAB-related behavior principles are applied with increasing frequency to behavior management and training practices with animals in nonlaboratory settings. Such translations, and those populations to which they are applied, benefit from a rigorous experimental analysis of practices that are promulgated in popular outlets. These translations both affirm the generality of those principles and serve as goads for laboratory and field research that can further articulate extant principles, develop new ones, and refine methods of application and assessment. This review considered several areas of basic EAB research and contemporary applied animal behavior (AAB) practices in relation to one another: (1) response establishment and maintenance, (2) response reduction and elimination, (3) chaining and conditioned reinforcement, and (4) discriminative stimulus control. Within each topic, a selection of processes and procedures in both EAB and AAB work were reviewed in relation to one another.

Training Petting Zoo Sheep to Act Like Petting Zoo Sheep: An Empirical Evaluation of Response-Independent Schedules and Shaping with Negative Reinforcement

Simple Summary

The present study examines the use of response-independent food schedules (food delivered independent of what an animal is doing) and negative reinforcement in the form of a trainer removing themselves from the presence of an animal to effectively train approach and contact behaviors in petting zoo sheep. All sheep had previously been reported to avoid/escape the presence of zoo staff and caretakers, as well as refuse to eat in the presence of any persons. In Experiment 1, we were able to effectively use response-independent schedules to produce both approach and contact behaviors in a Hampshire sheep. In Experiment 2, a Jacob sheep was trained for approach/contact via a shaping procedure that initially rewarded approach behaviors by removing the presence of the trainer (negative reinforcement), and was later switched to rewarding responses directly with food (positive reinforcement) once the sheep began to regularly eat in the trainer’s presence. The results are discussed with respect to how both procedures are effective in examining the applied approach/avoidance behaviors with zoo animals, as well as the greater ethical considerations involved in using negative vs. positive reinforcement to train animals.

Abstract

Shaping through differential reinforcement of successive approximations to a target response has been a cornerstone procedure for the training of novel behavior. However, much of how it has traditionally been implemented occurs through informal observation, rather than any direct, systematic measurement. In the present study, we examine the use of response-independent food schedules and shaping for increasing approach and contact behaviors in petting zoo sheep. In Experiment 1, a fixed-time (FT) 15 s food schedule was used to effectively increase approach and contact behaviors in one sheep. In Experiment 2, negative reinforcement in the form of removal of the presence of a trainer was made contingent on the successful completion of approximations within a shaping procedure and later switched to food rewards. A changing-criterion design was used to empirically examine the effects of the shaping procedure during each step of the program. The result is one of the first studies to demonstrate the utility of using negative reinforcement within a shaping procedure to successfully intervene on approach/avoidance behaviors in an applied animal setting.

Rats' choices with token stimuli in concurrent variable-interval schedules

Four rats responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights arranged vertically above each of two side levers). During exchange periods, each token could be exchanged for one food pellet by responding on a center lever, with one response required for each pellet delivery. In different conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response levers. The experiments were closely patterned after research with pigeons by Mazur and Biondi (2013), and the results from the rats in the present experiment were similar. Response percentages on the two levers changed as each additional token was earned, and these patterns indicated that choice was controlled by both the time to the exchange periods and the number of food pellets that were delivered in the exchange period. In some conditions, the exchange requirement was three tokens for each lever, but the token lights were not turned on as they were earned for one of the two keys. The rats showed a slight preference for the lever without the token lights, which may indicate that the token lights were not serving as conditioned reinforcers (a result also found by Mazur and Biondi with pigeons). Overall, these results suggest that, in this choice procedure, the token stimuli served primarily as discriminative stimuli that signaled the temporal proximity and quantity of the primary reinforcer, food.

Effects of reinforcing preselected approximations on the topography of the rat's bar press

The effects of different shaping approximations on the topography of the rat's bar press were investigated in two experiments. Behavior was classified into discrete components, and changes in components and their sequential organization were analyzed. Experiment 1 examined response form early in training and found that specific components reinforced during shaping were incorporated into press sequences. Experiment 2 investigated how response form changed when a shaping contingency was relaxed later in training. Two topographies were selected for reinforcement, and both appeared in the press sequences of all subjects by the end of shaping. Subsequently, all variations of bar pressing were reinforced, and neither topography was necessary to satisfy the contingency. Although the frequency of the topographies reinforced during shaping declined for 3 of 4 subjects during this phase, the most frequent press sequence for 2 rats at the end of training included both unnecessary topographies. Variability in press topographies declined when all emitted variants were reinforced. However, all subjects emitted novel response forms throughout training. The results demonstrate that specific response-reinforcer contingencies influence response form by modulating component availability and organization.

Token reinforcement: a review and analysis

Token reinforcement procedures and concepts are reviewed and discussed in relation to general principles of behavior. The paper is divided into four main parts. Part I reviews and discusses previous research on token systems in relation to common behavioral functions--reinforcement, temporal organization, antecedent stimulus functions, and aversive control--emphasizing both the continuities with other contingencies and the distinctive features of token systems. Part II describes the role of token procedures in the symmetrical law of effect, the view that reinforcers (gains) and punishers (losses) can be measured in conceptually analogous terms. Part III considers the utility of token reinforcement procedures in cross-species analysis of behavior more generally, showing how token procedures can be used to bridge the methodological gulf separating research with humans from that with other animals. Part IV discusses the relevance of token systems to the field of behavioral economics. Token systems have the potential to significantly advance research and theory in behavioral economics, permitting both a more refined analysis of the costs and benefits underlying standard economic models, and a common currency more akin to human monetary systems. Some implications for applied research and for broader theoretical integration across disciplines will also be considered.

Maintenance of responding when reinforcement becomes delayed

In four experiments with rats, we examined the persistence of behavior when reinforcement was switched from immediate to delayed. In Experiment 1, lever pressing elicited by instrumental training with immediate reinforcement continued when a 20-sec delay of reinforcement was introduced (easy-to-hard condition), whereas when the delay condition was introduced from the start (hard-to-hard condition), responding remained low throughout. A similar result was obtained in Experiment 2, in which lever pressing was elicited by a classical conditioning (autoshaping) procedure. In Experiment 3, rats initially trained with delayed reinforcement continued to respond at a low rate when switched to immediate reinforcement (hard-to-easy condition). By measuring magazine entry (goal tracking) as well as lever pressing (sign tracking) in Experiment 4, we confirmed that such transfer effects at least partly involve the persistence of whatever type of behavior was initially dominant.

Second-order schedules of token reinforcement with pigeons: effects of fixed- and variable-ratio exchange schedules

Pigeons' key pecks produced food under second-order schedules of token reinforcement, with light-emitting diodes serving as token reinforcers. In Experiment 1, tokens were earned according to a fixed-ratio 50 schedule and were exchanged for food according to either fixed-ratio or variable-ratio exchange schedules, with schedule type varied across conditions. In Experiment 2, schedule type was varied within sessions using a multiple schedule. In one component, tokens were earned according to a fixed-ratio 50 schedule and exchanged according to a variable-ratio schedule. In the other component, tokens were earned according to a variable-ratio 50 schedule and exchanged according to a fixed-ratio schedule. In both experiments, the number of responses per exchange was varied parametrically across conditions, ranging from 50 to 400 responses. Response rates decreased systematically with increases in the fixed-ratio exchange schedules, but were much less affected by changes in the variable-ratio exchange schedules. Response rates were consistently higher under variable-ratio exchange schedules than tinder comparable fixed-ratio exchange schedules, especially at higher exchange ratios. These response-rate differences were due both to greater pre-ratio pausing and to lower local rates tinder the fixed-ratio exchange schedules. Local response rates increased with proximity to food under the higher fixed-ratio exchange schedules, indicative of discriminative control by the tokens.

Beyond the "hot-and-cold" game: a demonstration of computer-controlled shaping

Shaping, or the method of successive approximations, is widely taught in introductory psychology and the psychology of learning as a procedure for establishing new behavior. This article illustrates a computer-controlled shaping demonstration that allows the user to specify several critical parameters of the shaping process and that then shapes the user's mouse movements toward an arbitrary virtual (invisible) target on the computer screen. The relative effectiveness of different shaping parameters can be assessed by examining several dependent measures, such as the distance of the cursor from the target across time and the rate at which reinforcers were earned. This demonstration allows students to move beyond the notion that shaping is simply the application of the "hot-and-cold" game and to understand that there is a science underlying the art of shaping.