Income maximizing in concurrent interval-ratio schedules

Towards an animal model of consciousness based on the platform theory

The evolution of intellectual capacities has brought forth a continuum of consciousness levels subserved by neuronal networks of varying complexity. Brain pathologies, neurodegenerative, and mental diseases affect conscious cognition and behavior. Although impairments in consciousness are among the most devastating consequences of neurological and mental diseases, valid and reliable animal models of consciousness, that could be used for preclinical research are missing. The platform theory holds that the brain enters a conscious operation mode, whenever mental representations of stimuli, associations, concepts, memories, and experiences are effortfully maintained (in working memory) and actively manipulated. We used the platform theory as a framework and evaluation standard to categorize behavioral paradigms with respect to the level of consciousness involved in task performance. According to the platform theory, a behavioral paradigm involves conscious cognitive operations, when the problem posed is unexpected, novel or requires the maintenance and manipulation of a large amount of information to perform cognitive operations on them. Conscious cognitive operations are associated with a relocation of processing resources and the redirection of attentional focus. A consciousness behavioral test battery is proposed that is composed of tests which are assumed to require higher levels of consciousness as compared to other tasks and paradigms. The consciousness test battery for rodents includes the following tests: Working memory in the radial arm maze, episodic-like memory, prospective memory, detour test, and operant conditioning with concurrent variable-interval variable-ratio schedules. Performance in this test battery can be contrasted with the performance in paradigms and tests that require lower levels of consciousness. Additionally, a second more comprehensive behavioral test battery is proposed to control for behavioral phenotypes not related to consciousness. Our theory could serve as a guidance for the decryption of the neurobiological basis of consciousness.

Income maximizing on concurrent ratio-interval schedules of reinforcement

Three experiments examined the effect of food availability on pigeons' choice behavior under concurrent schedules of reinforcement. In Experiment 1, 3 pigeons earned their daily food ration by choosing, in 30-min sessions, between concurrent variable-ratio 30 variable-interval 40-s schedules. Food presentations during both schedules lasted 2 or 12 s, depending upon the condition. Relative variable-ratio response rate was inversely related to hopper duration. In Experiment 2, 4 pigeons received their daily feeding by responding on the same schedule pair as in Experiment 1 (with 4-s food presentations) in sessions that varied in length from 10 to 30 min, depending on the condition. The length of a vertical slit projected on a response key increased with time so that "passage of time" might be more easily discriminable. As session duration decreased, relative variable-ratio response rate increased. In Experiment 3, 4 pigeons chose between two variable-interval 40-s schedules. One schedule operated without regard to the schedule selected, whereas the other operated only when the subject responded in its presence (dependent). Although these schedules had the same feedback function, preference for the dependent variable interval increased as session duration decreased from 30 to 10 min. The preference changes in these studies reveal the operation of an income-maximizing process in choice.

Maximizing versus matching on concurrent variable-interval schedules

Maximization and matching predictions were examined for a time-based analogue of the concurrent variable-interval variable-ratio schedule. One alternative was a variable interval whose time base operated relatively independent of the schedule chosen, and the other was a discontinuous variable interval for which timing progressed only when selected. Pigeons switched between schedules by pecking a changeover key. The maximization hypothesis predicts that subjects will show a bias toward the discontinuous variable interval and undermatching; however the obtained results conformed closely to the predictions of the matching law. Finally, a quantitative comparison was made of the bias and sensitivity estimates obtained in published concurrent variable-interval variable-ratio analogue studies. Results indicated that only the ratio-based analogue of the concurrent variable interval variable ratio studied by Green, Rachlin, and Hanson (1983) produced significant bias toward the variable-ratio alternative and undermatching, as predicted by reinforcement maximization.

The Actor-Critic Learning Is Behind the Matching Law: Matching Versus Optimal Behaviors

The ability to make a correct choice of behavior from various options is crucial for animals' survival. The neural basis for the choice of behavior has been attracting growing attention in research on biological and artificial neural systems. Alternative choice tasks with variable ratio (VR) and variable interval (VI) schedules of reinforcement have often been employed in studying decision making by animals and humans. In the VR schedule task, alternative choices are reinforced with different probabilities, and subjects learn to select the behavioral response rewarded more frequently. In the VI schedule task, alternative choices are reinforced at different average intervals independent of the choice frequencies, and the choice behavior follows the so-called matching law. The two policies appear robustly in subjects' choice of behavior, but the underlying neural mechanisms remain unknown. Here, we show that these seemingly different policies can appear from a common computational algorithm known as actor-critic learning. We present experimentally testable variations of the VI schedule in which the matching behavior gives only a suboptimal solution to decision making and show that the actor-critic system exhibits the matching behavior in the steady state of the learning even when the matching behavior is suboptimal. However, it is found that the matching behavior can earn approximately the same reward as the optimal one in many practical situations.

Computational algorithms and neuronal network models underlying decision processes

Animals or humans often encounter such situations in which they must choose their behavioral responses to be made in the near or distant future. Such a decision is made through continuous and bidirectional interactions between the environment surrounding the brain and its internal state or dynamical processes. Therefore, decision making may provide a unique field of researches for studying information processing by the brain, a biological system open to information exchanges with the external world. To make a decision, the brain must analyze pieces of information given externally, past experiences in a similar situation, possible behavioral responses, and predicted outcomes of the individual responses. In this article, we review results of recent experimental and theoretical studies of neuronal substrates and computational algorithms for decision processes.

An olfactory discrimination procedure for mice

This paper describes an olfactory discrimination procedure for mice that is inexpensively implemented and leads to rapid discrimination learning. Mice were first trained to dig in small containers of sand to retrieve bits of buried chocolate. For discrimination training, two containers were presented simultaneously for eight trials per session. One container held sand mixed with cinnamon, and the other held sand mixed with nutmeg. Both containers were baited with chocolate buried in the sand. One odor was designated S+, and mice were allowed to dig and retrieve the chocolate from this container. The other odor was S-, and both containers were removed immediately if subjects began to dig in an S- container. After meeting a two-session acquisition criterion, subjects were given a series of discrimination reversals. In Experiment 1, 12 Swiss-Webster mice (6 male and 6 female) acquired the olfactory discrimination in three to five sessions and completed 3 to 10 successive discrimination reversals within a 50-session testing limit. In Experiment 2, subjects were 14 Pah(enu2) mice, the mouse mutant for phenylketonuria; 7 were homozygotes in which the disorder was expressed (PKU), and 7 were heterozygotes with normal metabolism (non-PKU). Thirteen mice completed pretraining in four to seven sessions, acquisition required 3 to 12 sessions, and all mice completed at least three reversals. Learning rates were similar in PKU and non-PKU mice. We discuss issues related to implementation and several potentially useful procedural variations.

Stock optimizing: maximizing reinforcers per session on a variable-interval schedule

In Experiment 1, 2 monkeys earned their daily food ration by pressing a key that delivered food according to a variable-interval 3-min schedule. In Phases 1 and 4, sessions ended after 3 hr. In Phases 2 and 3, sessions ended after a fixed number of responses that reduced food intake and body weights from levels during Phases 1 and 4. Monkeys responded at higher rates and emitted more responses per food delivery when the food earned in a session was reduced. In Experiment 2, monkeys earned their daily food ration by depositing tokens into the response panel. Deposits delivered food according to a variable-interval 3-min schedule. When the token supply was unlimited (Phases 1, 3, and 5), sessions ended after 3 hr. In Phases 2 and 4, sessions ended after 150 tokens were deposited, resulting in a decrease in food intake and body weight. Both monkeys responded at lower rates and emitted fewer responses per food delivery when the food earned in a session was reduced. Experiment 1's results are consistent with a strength account, according to which the phases that reduced body weights increased food's value and therefore increased subjects' response rates. The results of Experiment 2 are consistent with an optimizing strategy, because lowering response rates when food is restricted defends body weight on variable-interval schedules. These contrasting results may be attributed to the discriminability of the contingency between response number and the end of a session being greater in Experiment 2 than in Experiment 1. In consequence, subjects lowered their response rates in order to increase the number of reinforcers per session (stock optimizing).

Human choice on concurrent variable-interval variable-ratio schedules

Each of 5 adult male humans sat in a 4 degrees C room where they could warm themselves by illuminating six heat lamps for 10-second periods according to a concurrent variable-interval variable-ratio schedule. Left-button presses on a response panel switched between the schedules and started a 2-second changeover delay. Right-button presses illuminated the heat lamps if assigned by the associated schedule and if the changeover delay had timed out. Panel lights identified the schedule in effect and each effective right-button press. A discrimination procedure--either a multiple variable-interval variable-ratio schedule or the presentation of each schedule individually on alternate days--preceded exposure to the choice procedure for some subjects. For subjects not exposed to a discrimination procedure prior to exposure to choice, or if such exposure failed to result in higher rates to the ratio than to the interval schedule, relative response rates matched relative reinforcement rates. However, if subjects responded at higher rates to the ratio schedule than to the interval schedule during a prior discrimination procedure, relative rates on a subsequent choice procedure deviated from matching in the direction of reinforcement-rate maximizing. In eight of 11 conditions, choice appeared to be governed by maximizing processes. In all cases, human concurrent ratio-interval performances differed from those of nonhumans in that matching was never obtained with local ratio-interval rate differences.

Behavioral economics of drug self-administration and drug abuse policy

The concepts of behavioral economics have proven useful for understanding the environmental control of overall levels of responding for a variety of commodities, including reinforcement by drug self-administration. These general concepts are summarized for application to the analysis of drug-reinforced behavior and proposed as the basis for future applications. This behavioral agenda includes the assessment of abuse liability, the assay of drug-reinforcer interactions, the design of drug abuse interventions, and the formulation of drug abuse public policy. These separate domains of investigation are described as part of an overall strategy for designing model projects to control drug use and testing public policy initiatives.