Human observing: Maintained by stimuli correlated with reinforcement but not extinction

Environment tracking and signal following in a reinforcer-ratio reversal procedure

Several studies suggest that the degree of control by reinforcer ratios (environment tracking) and by exteroceptive stimuli that signal future reinforcer availability (signal following) depends on environmental certainty: As reinforcers become more likely at one location, environmental contingencies exert stronger control and exteroceptive stimuli exert weaker control. This research has not yet been extended to environments in which reinforcer availability changes across time, even though such changes are present in most natural environments. Thus, in the present experiment, we examined environment tracking and signal following in a concurrent schedule in which the reinforcer ratio reversed to its reciprocal 30 s after a reinforcer delivery and keylight-color stimuli signaled the likely or definite time or location of the next reinforcer. Across conditions, we manipulated environmental certainty by varying the probability of reinforcer deliveries on the locally richer key. This made the location of future reinforcers at a particular time more or less certain, but did not change the overall reinforcer ratio. Changes in local environmental certainty had little to no effect on environment tracking and signal following; in all conditions, keylight-color stimuli strongly controlled choice and reinforcer ratios exerted weak control. The present findings suggest that the extent of environment tracking and signal following is primarily determined by global, not local, environmental certainty.

How I Learned to Stop Worrying and Love Replication Failures

Worries about the reproducibility of experiments in the behavioral and social sciences arise from evidence that many published reports contain false positive results. Misunderstanding and misuse of statistical procedures are key sources of false positives. In behavior analysis, however, statistical procedures have not been used much. Instead, the investigator must show that the behavior of an individual is consistent over time within an experimental condition, that the behavior changes systematically across conditions, and that these changes can be reproduced - and then the whole pattern must be shown in additional individuals. These high standards of within- and between-subject replication protect behavior analysis from the publication of false positive findings. When a properly designed and executed experiment fails to replicate a previously published finding, the failure exposes flaws in our understanding of the phenomenon under study - perhaps in recognizing the boundary conditions of the phenomenon, identifying the relevant variables, or bringing the variables under sufficient control. We must accept the contradictory findings as valid and pursue an experimental analysis of the possible reasons. In this way, we resolve the contradiction and advance our science. To illustrate, two research programs are described, each initiated because of a replication failure.

Reprint of "Suboptimal choice by pigeons: an analog of human gambling behavior"

Human gambling often involves the choice of a low probability but high valued outcome over a high probability (certain) low valued outcome (not gambling) that is economically more optimal. We have developed an analog of gambling in which pigeons prefer a suboptimal alternative that infrequently provides a signal for a high probability (or high magnitude) of reinforcement over an optimal alternative that always provides a signal for a lower probability (or lower magnitude) of reinforcement. We have identified two mechanisms that may be responsible for this suboptimal behavior. First, the effect of nonreinforcement results in considerably less inhibition of choice than ideally it should. Second, the frequency of the occurrence of the signal for a high probability or high magnitude of reinforcement is less important than ideally it should. Also analogous to human gambling is the finding that pigeons that are normally food restricted choose suboptimally, whereas those that are minimally food restricted choose optimally. In addition, pigeons that are singly housed choose suboptimally, whereas those that are exposed to a more enriched environment choose less suboptimally. We believe that these findings have implications for the understanding and treatment of problem gambling behavior.

Suboptimal choice by pigeons: an analog of human gambling behavior

Human gambling often involves the choice of a low probability but high valued outcome over a high probability (certain) low valued outcome (not gambling) that is economically more optimal. We have developed an analog of gambling in which pigeons prefer a suboptimal alternative that infrequently provides a signal for a high probability (or high magnitude) of reinforcement over an optimal alternative that always provides a signal for a lower probability (or lower magnitude) of reinforcement. We have identified two mechanisms that may be responsible for this suboptimal behavior. First, the effect of nonreinforcement results in considerably less inhibition of choice than ideally it should. Second, the frequency of the occurrence of the signal for a high probability or high magnitude of reinforcement is less important than ideally it should. Also analogous to human gambling is the finding that pigeons that are normally food restricted choose suboptimally, whereas those that are minimally food restricted choose optimally. In addition, pigeons that are singly housed choose suboptimally, whereas those that are exposed to a more enriched environment choose less suboptimally. We believe that these findings have implications for the understanding and treatment of problem gambling behavior.

Do pigeons prefer information in the absence of differential reinforcement?

Prior research has indicated that pigeons do not prefer an alternative that provides a sample (for matching to sample) over an alternative that does not provide a sample (i.e., there is no indication of which comparison stimulus is correct). However, Zentall and Stagner (Journal of Experimental Psychology. Animal Behavior Processes 36:506-509, 2010) showed that when delay of reinforcement was controlled, pigeons had a strong preference for matching over pseudomatching (i.e., there was a sample, but it did not indicate which comparison stimulus was correct). Experiment 1 of the present study replicated and extended the results of the Zentall and Stagner (Journal of Experimental Psychology. Animal Behavior Processes 36:506-509, 2010) study by including an identity relation between the sample and one of the comparison stimuli in both the matching and pseudomatching tasks. In Experiment 2, in which we asked whether the pigeons would still prefer matching if we equated the two tasks for probability of reinforcement, we found no systematic preference for matching over pseudomatching. Thus, it appears that in the absence of differential reinforcement, the information provided by a sample that signals which of the two comparison stimuli is correct is insufficient to produce a preference for that alternative.

Individual variation in resisting temptation: implications for addiction

When exposed to the sights, sounds, smells and/or places that have been associated with rewards, such as food or drugs, some individuals have difficulty resisting the temptation to seek out and consume them. Others have less difficulty restraining themselves. Thus, Pavlovian reward cues may motivate maladaptive patterns of behavior to a greater extent in some individuals than in others. We are just beginning to understand the factors underlying individual differences in the extent to which reward cues acquire powerful motivational properties, and therefore, the ability to act as incentive stimuli. Here we review converging evidence from studies in both human and non-human animals suggesting that a subset of individuals are more "cue reactive", in that certain reward cues are more likely to attract these individuals to them and motivate actions to get them. We suggest that those individuals for whom Pavlovian reward cues become especially powerful incentives may be more vulnerable to impulse control disorders, such as binge eating and addiction.

Judgment and decision making: Behavioral approaches

The area of judgment and decision making has given rise to the study of many interesting phenomena, including reasoning fallacies, which are also of interest to behavior analysts. Indeed, techniques and principles of behavior analysis may be applied to study these fallacies. This article reviews research from a behavioral perspective that suggests that humans are not the information-seekers we sometimes suppose ourselves to be. Nor do we utilize information effectively when it is presented. This is shown from the results of research utilizing matching to sample and other behavioral tools (monetary reward, feedback, instructional control) to study phenomena such as the conjunction fallacy, base-rate neglect, and probability matching. Research from a behavioral perspective can complement research from other perspectives in furthering our understanding of judgment and decision making.

Choice, conditioned reinforcement, and the prius effect

Psychologists have long been intrigued with the rationales that underlie our decisions. Similarly, the concept of conditioned reinforcement has a venerable history, particularly in accounting for behavior not obviously maintained by primary reinforcers. The studies of choice and of conditioned reinforcement have often developed in lockstep. Many contemporary approaches to these fundamental topics share an emphasis on context and on relative value. We trace the evolution of thinking about the potency of conditioned reinforcers from stimuli that were thought to acquire their value from pairings with more fundamental reinforcers to stimuli that acquire their value by being differentially correlated with these more fundamental reinforcers. We discuss some seminal experiments (including several that have been underappreciated) and some ongoing data, all of which have propelled us to the conclusion that the strength of conditioned reinforcers is determined by their signaling a relative improvement in the organism's relation to reinforcement.

Revisiting the role of bad news in maintaining human observing behavior

Results from studies of observing responses have suggested that stimuli maintain observing owing to their special relationship to primary reinforcement (the conditioned-reinforcement hypothesis), and not because they predict the availability and nonavailability of reinforcement (the information hypothesis). The present article first reviews a study that challenges that conclusion and then reports a series of five brief experiments that provide further support for the conditioned-reinforcement view. In Experiments 1 through 3, participants preferred occasional good news (a stimulus correlated with reinforcement) or no news (a stimulus uncorrelated with reinforcement) to occasional bad news (a stimulus negatively correlated with reinforcement). In Experiment 4 bad news was preferred to no news when the absence of stimulus change following a response to the bad-news option was reliably associated with good news. When this association was weakened in Experiment 5 the results were intermediate. The results support the conclusion that information is reinforcing only when it is positive or useful. As required by the conditioned-reinforcement hypothesis, useless information does not maintain observing.

Pavlovian-instrumental interaction in 'observing behavior'

Subjects typically choose to be presented with stimuli that predict the existence of future reinforcements. This so-called ‘observing behavior’ is evident in many species under various experimental conditions, including if the choice is expensive, or if there is nothing that subjects can do to improve their lot with the information gained. A recent study showed that the activities of putative midbrain dopamine neurons reflect this preference for observation in a way that appears to challenge the common prediction-error interpretation of these neurons. In this paper, we provide an alternative account according to which observing behavior arises from a small, possibly Pavlovian, bias associated with the operation of working memory.

The theory of Reinforcement Learning (RL) has been influential in explaining basic learning and behavior in humans and other animals, and in accounting for key features of the activity of dopamine neurons. However, perhaps due to this very success, paradigms that challenge RL are at a premium. One case concerns so-called ‘observing behavior’, in which, at least in some versions, animals elect to observe cues that are predictive of future rewarding outcomes, although the observations themselves have no direct behavioral relevance. In a recent experiment on observing, the activity of monkey dopaminergic neurons was also found to be incompatible with classic RL. However, as is often the case, this was a task that allowed for potential interactions from a secondary behavioral system in which responses are directly triggered by values. In this paper we show that a model incorporating a next order of refinement associated with such Pavlovian interactions can explain this type of observing behavior.

Observing responses and serial stimuli: searching for the reinforcing properties of the S-

The control exerted by a stimulus associated with an extinction component (S-) on observing responses was determined as a function of its temporal relation with the onset of the reinforcement component. Lever pressing by rats was reinforced on a mixed random-interval extinction schedule. Each press on a second lever produced stimuli associated with the component of the schedule in effect. In Experiment 1 a response-dependent clock procedure that incorporated different stimuli associated with an extinction component of a variable duration was used. When a single S- was presented throughout the extinction component, the rate of observing remained relatively constant across this component. In the response-dependent clock procedure, observing responses increased from the beginning to the end of the extinction component. This result was replicated in Experiment 2, using a similar clock procedure but keeping the number of stimuli per extinction component constant. We conclude that the S- can function as a conditioned reinforcer, a neutral stimulus or as an aversive stimulus, depending on its temporal location within the extinction component.

Effects of different accessibility of reinforcement schedules on choice in humans

Based on the delay-reduction hypothesis, a less profitable schedule should be rejected if its duration exceeds the mean delay to reinforcement. It should be accepted if its duration is shorter than the mean delay. This was tested for humans, using a successive-choice schedule. The accessibility of the less profitable (variable-interval 18 s) schedule was varied by changing the duration (in terms of a fixed interval) of the waiting-time component preceding its presentation. Forty-eight students were randomly assigned to three groups. In Phase 1, the duration of the less profitable schedule equaled the mean delay to reinforcement in all groups. In Phase 2, waiting time preceding the less profitable schedule was reduced in Group 1 and increased in Group 2. Thus, the schedule was correlated either with a relative delay increase (Group 1) or a delay reduction (Group 2). In Group 3, conditions remained unchanged. As predicted, acceptance of the less profitable schedule decreased in Group 1 and increased in Group 2. The increased acceptance in Group 2 was accompanied by a decreased acceptance of the more profitable (variable-interval 3 s) schedule, resembling a pattern of negative contrast. Response rates were higher under the component preceding (a) the more profitable schedule in Group 1 and (b) the less profitable schedule in Group 2. Implications for the modification of human choice behavior are discussed.

On conditioned reinforcing effects of negative discriminative stimuli

Observing responses by pigeons were studied during sessions in which a food key and an observing key were available continuously. A variable-interval schedule and extinction alternated randomly on the food key. In one condition, food-key pecking during extinction decreased reinforcement frequency during the next variable-interval component, and in the other condition such pecking did not affect reinforcement frequency. Observing responses either changed both keylight colors from white to green (S+) or to red (S-) depending on the condition on the food key, or the observing responses never produced the S+ but produced the S- when extinction was in effect on the food key. Observing responses that produced only S- were maintained only when food-key pecking during extinction decreased reinforcement frequency in the subsequent variable-interval component. The red light conformed to conventional definitions of a negative discriminative stimulus, rendering results counter to previous findings that production of S- alone does not maintain observing. Rather than offering support for an informational account of conditioned reinforcement, the results are discussed in terms of a molar analysis to account for how stimuli acquire response-maintaining properties.

Conditioned reinforcement of human observing behavior by descriptive and arbitrary verbal stimuli

College students earned monetary reinforcers by pressing a key according to a compound schedule with variable-interval and extinction components. Pressing additional keys occasionally produced displays of either of two verbal stimuli; one was uncorrelated with the schedule components, and the other was correlated with the extinction component. In Experiments 1 and 2, the display area of the apparatus was blank unless an observing key was pressed, whereupon a descriptive message appeared. Most students preferred an uncorrelated stimulus stating that "Some of this time scores are TWICE AS LIKELY as normal, and some of this time NO SCORES can be earned" over a stimulus stating that "At this time NO SCORES can be earned." In Experiment 3, the display area indicated that "The Current Status of the Program is: NOT SHOWN." Presses on the observing keys replaced this message with stimuli that provided arbitrary labels for the schedule conditions. All of the students preferred a stimulus stating that "The Current Status of the Program is: B" over an uncorrelated stimulus stating that "The Current Status of the Program is: either A or B." Thus, under some circumstances, observing was maintained by a stimulus correlated with extinction-a finding that poses a challenge for Pavolvian accounts of conditioned reinforcement. Differences in the maintenance of observing by the descriptive and arbitrary stimuli may be attributed to differences in either the strength or nature of the instructional control exerted by the verbal stimuli.

Separating the effects of salience and disparity on the rate of observing

Pigeons producing deliveries of grain on a mixed variable-interval, extinction schedule by pecking a center key could also produce discriminative stimuli on concurrent variable-interval schedules by pecking the left or right observing key. The stimuli produced by each observing key were varied independently. In the first experiment, the negative discriminative stimulus was at the far end of the spectrum from the key illumination accompanying the mixed schedule and from the positive discriminative stimulus. When the magnitude of the difference between the latter two stimuli (salience) was varied, more pecks occurred on the observing key producing the larger of the two differences than on the key producing the smaller difference. In the second experiment, the stimulus accompanying the mixed schedule was at the far end of the spectrum, and the magnitude of the difference between the two discriminative stimuli (disparity) was varied. The proportion of pecks occurring on each observing key shifted systematically in the direction of the key producing the larger difference. The salience of the discriminative stimuli and their disparity each has an independent influence on the frequency of observing when the other is controlled, but the effect of the salience appears to be the more substantial.

Observing responses: maintained by good news only?

Observing responses are those that produce stimuli correlated with the availability (S+) or non-availability (S-) of reinforcement but that has no influence on the actual delivery or timing of reinforcement. Prior research has shown that observing is maintained by the occasional production of the S+ ("good news") and not by production of the equally informative S- ("bad news"). However, for both humans and rats the S- maintains observing when it is at least implicitly correlated with good news. In the present study, pigeons could obtain both good and bad news by responding during the appropriate key color. In one condition, the bad news was actually more informative about reinforcement than was the good news. Nevertheless, a preponderance of the birds' responses was made on the nominally good-news option. The present results offer further support for the central role of good news in maintaining observing responses and are entirely consistent with the traditional conditioned-reinforcement (or classical conditioning) interpretation of observing.

Classical conditioning: The new hegemony

Converging data from different disciplines are showing the role of classical conditioning processes in the elaboration of human and animal behavior to be larger than previously supposed. Restricted views of classically conditioned responses as merely secretory, reflexive, or emotional are giving way to a broader conception that includes problem-solving, and other rule-governed behavior thought to be the exclusive province of either operant conditiońing or cognitive psychology. These new views have been accompanied by changes in the way conditioning is conducted and evaluated. Data from a number of seemingly unrelated phenomena such as relapse to drug abuse by postaddicts, the placebo effect, and the immune response appear to involve classical conditioning processes. Classical conditioning, moreover, has been found to occur in simpler and simpler organisms and recently even demonstrated in brain slices and in utero. This target article will integrate the several research areas that have used the classical conditioning process as an explanatory model; it will challenge teleological interpretations of the classically conditioned CR and offer some basic principles for testing conditioning in diverse areas.

A framework for the functional analysis of behaviour

We present a general framework for analyzing the contribution to reproductive success of a behavioural action. An action may make a direct contribution to reproductive success, but even in the absence of a direct contribution it may make an indirect contribution by changing the animal's state. We consider actions over a period of time, and define a reward function that characterizes the relationship between the animal's state at the end of the period and its future reproductive success. Working back from the end of the period using dynamic programming, the optimal action as a function of state and time can be found. The procedure also yields a measure of the cost, in terms of future reproductive success, of a suboptimal action. These costs provide us with a common currency for comparing activities such as eating and drinking, or eating and hiding from predators. The costs also give an indication of the robustness of the conclusions that can be drawn from a model. We review how our framework can be used to analyze optimal foraging decisions in a stochastic environment. We also discuss the modelling of optimal daily routines and provide an illustration based on singing to attract a mate. We use the model to investigate the features that can produce a dawn song burst in birds. State is defined very broadly so that it includes the information an animal has about its environment. Thus, exploration and learning can be included within the framework.