Local analysis of behaviour in the adjusting-delay task for assessing choice of delayed reinforcement

Using type-2 fuzzy ontology to improve semantic interoperability for healthcare and diagnosis of depression

Ontology enhances semantic interoperability through integrating health data from heterogeneous sources and sharing information in a meaningful way. In the field of smart health services, semantic interoperability means the exchange and interpretation of data without ambiguity and uncertainty. However, existing classical ontologies are not able to represent vague and uncertain knowledge, especially in contexts of mental health disorders which are associated with varying degrees of uncertainty and inaccuracy of diagnosis, and in this case, the treatment is a complex and common mental process necessitating to share information accurately and unambiguously. Type-2 fuzzy set theory can offer a fruitful solution in order to control uncertainty or express ambiguous concepts in a dynamic and complex environment such as healthcare systems. Herein, a semantic framework for healthcare, and also monitoring mental health disorders using type-2 fuzzy set theory based on the Internet of Thing (IoT) is suggested, in which all depression-related concepts are semantically annotated to share detailed information with the treatment staff. This framework not only paved the way to increasing the accuracy of medical diagnosis and decision-making but also provides the possibility of inference and semantic reasoning using the languages of SPARQL query and DL query.

Generalizability of time-based interventions: Effects of choice procedure and smaller-sooner delay

Interventions exposing rats to delayed-reward contingencies attenuate suboptimal impulsive choices, a preference for a smaller-sooner (SS) over a larger-later (LL) reward. Interventions may potentially improve delay-tolerance, timing of delays, and/or discrimination of reward magnitudes. Generalization from the intervention to impulsive choice under different procedures can provide insights into the processes that underlie the intervention effects. Experiment 1 tested intervention effects on systematic-delay (SYS) and adjusting-delay (ADJ) procedures, predicting that intervention effects would be more effective on the SYS procedure with predictable delays. The ADJ procedure did not benefit significantly from intervention, but the SYS procedure, unexpectedly, showed greater impulsive choices following intervention. Experiment 2 tested whether short (5 s) SS intervention delays may have promoted greater impulsivity in the SYS impulsive choice procedure in Experiment 1. Short SS delays in choice and intervention procedures increased impulsive choices in comparison to longer (10 s) delays. Incongruent SS delays in the intervention/choice procedures resulted in negative intervention effects. The results suggest that short SS delays are detrimental to self-control and that specific temporal information generalizes from the intervention to the SYS choice task, but not the ADJ choice task.

Computational validity: using computation to translate behaviours across species

We propose a new conceptual framework (computational validity) for translation across species and populations based on the computational similarity between the information processing underlying parallel tasks. Translating between species depends not on the superficial similarity of the tasks presented, but rather on the computational similarity of the strategies and mechanisms that underlie those behaviours. Computational validity goes beyond construct validity by directly addressing questions of information processing. Computational validity interacts with circuit validity as computation depends on circuits, but similar computations could be accomplished by different circuits. Because different individuals may use different computations to accomplish a given task, computational validity suggests that behaviour should be understood through the subject's point of view; thus, behaviour should be characterized on an individual level rather than a task level. Tasks can constrain the computational algorithms available to a subject and the observed subtleties of that behaviour can provide information about the computations used by each individual. Computational validity has especially high relevance for the study of psychiatric disorders, given the new views of psychiatry as identifying and mediating information processing dysfunctions that may show high inter-individual variability, as well as for animal models investigating aspects of human psychiatric disorders. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

Prolonged Consumption of Sweetened Beverages Lastingly Deteriorates Cognitive Functions and Reward Processing in Mice

We investigated the detrimental effects of chronic consumption of sweet or sweetened beverages in mice. We report that consumption of beverages containing small amounts of sucrose during several weeks impaired reward systems. This is evidenced by robust changes in the activation pattern of prefrontal brain regions associated with abnormal risk-taking and delayed establishment of decision-making strategy. Supporting these findings, we find that chronic consumption of low doses of artificial sweeteners such as saccharin disrupts brain regions' activity engaged in decision-making and reward processes. Consequently, this leads to the rapid development of inflexible decisions, particularly in a subset of vulnerable individuals. Our data also reveal that regular consumption, even at low doses, of sweet or sweeteners dramatically alters brain neurochemistry, i.e., dopamine content and turnover, and high cognitive functions, while sparing metabolic regulations. Our findings suggest that it would be relevant to focus on long-term consequences on the brain of sweet or sweetened beverages in humans, especially as they may go metabolically unnoticed.

Discounting of food and water in rats shows trait- and state-like characteristics

Delay discounting is the loss in value of an outcome as a function of its delay. The present study focused on examining a trait-like characteristic of delay discounting in a preclinical animal model. Specifically, we were interested in whether there was a positive relation between discounting of 2 different outcomes in rats. That is, would rats that discount delayed food steeply also discount delayed water steeply? In addition, we examined how session-to-session variability in discounting could be attributed to differences between subjects (trait variability) and to differences within subjects (state variability). Finally, we measured discounting from early- to mid-adulthood, allowing us to examine changes in discounting as a function of age. Overall, we found a moderate, positive correlation between discounting of food and discounting of water in rats, providing further evidence that the relative consistency with which individuals discount different outcomes is a trait-like characteristic. In addition, we found a high degree of within-subject variability in discounting, indicating strong state-like differences from session to session. Finally, overall, discounting decreased as a function of age; however, individual-subject data showed variability in how discounting changed across time. Overall, our results show that differences in delay discounting between individuals reflect variability in both trait- and state-like characteristics.

Effects of morphine/CP55940 mixtures on an impulsive choice task in rhesus monkeys

µ-Opioid receptor agonists are commonly used to treat pain despite their adverse effects. In preclinical studies, cannabinoid receptor agonists increase the potency of opioids for producing antinociceptive but not reinforcing effects. It is unknown whether other adverse effects of these drugs, such as impairment of complex behavior, are enhanced by their co-administration. This study characterized the effects of morphine (µ-opioid receptor agonist; 0.32-5.6 mg/kg, subcutaneously) and CP55940 (CB1/CB2 cannabinoid receptor agonist; 0.0032-0.32 mg/kg, subcutaneously), alone and in mixtures, in monkeys (n=3) choosing between one pellet delivered immediately and two pellets delivered after a delay. Two consecutive choices of the immediate or delayed reward decreased or increased, respectively, the delay. The median adjusted delay, indicating indifference between the immediate and delayed reinforcers, was increased by morphine (3.2 mg/kg) and CP55940 (0.01-0.032 mg/kg). Performance after administration of morphine (0.32 and 1 mg/kg)/CP55940 (0.0032-0.032 mg/kg) mixtures was not different from performance after CP55940 alone. Neither morphine, CP55940, nor mixtures decreased the median adjusted delay (i.e. increased impulsivity). These findings failed to confirm previous studies showing that morphine increases impulsivity, perhaps because of procedural differences among studies. Treatment of pain often requires repeated drug administration; thus, it remains to be determined whether the present findings predict the effects of chronically administered morphine/CP5540 mixtures on impulsive choice.

Cognitive and behavioral training interventions to promote self-control

This review article discusses various cognitive and behavioral interventions that have been developed with the goal of promoting self-controlled responding. Self-control can exert a significant impact on human health and impulsive behaviors are associated with a wide range of diseases and disorders, leading to the suggestion that impulsivity is a trans-disease process. The self-control interventions include effort exposure, reward discrimination, reward bundling, interval schedules of reinforcement, impulse control training, and mindfulness training. Most of the interventions have been consistently shown to increase self-control, except for mindfulness training. Some of the successful interventions are long-lasting, whereas others may be transient. Most interventions are domain-specific, targeting specific cognitive and behavioral processes that relate to self-control rather than targeting overall self-control. For example, effort exposure appears to primarily increase effort tolerance, which in turn can improve self-control. Similarly, interval schedules primarily target interval timing, which promotes self-controlled responses. A diagram outlining a proposed set of intervention effects on self-control is introduced to motivate further research in this area. The diagram suggests that the individual target processes of the interventions may potentially summate to produce general self-control, or perhaps even produce synergistic effects. In addition, it is suggested that developing a self-control profile may be advantageous for aligning specific interventions to mitigate specific deficits. Overall, the results indicate that interventions are a promising avenue for promoting self-control and may help to contribute to changing health outcomes associated with a wide variety of diseases and disorders. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

An improved within-session self-adjusting delay discounting procedure for the study of choice impulsivity in rats

RATIONALE

Delay-discounting procedures involving choice between small immediate rewards and large delayed rewards are used to study impulsivity in rodents. Improving existing procedures may provide new insights into the neurobiological mechanisms underlying decision-making processes.

OBJECTIVES

To develop a novel delay-discounting procedure that adjusts the delay value within individual sessions based on the rat's most recent choices.

METHODS

Compared to previously developed procedure, we required a more consistent demonstration of preference, five consecutive choices of the large or small reward, a criterion that is more likely to reflect deliberate choice by the animal, as opposed to two consecutive choices. In addition, delays were changed in steps of 5 s (rather than 1 s), because 5-s increments should be more easily discriminated and may produce a more distinct effect on choice. We characterized the procedure behaviorally by manipulating the duration of the session and the consecutive choice criterion, and we investigated the stability of the behavior upon interruption of training. We also characterized the procedure pharmacologically by investigating the effects of dopaminergic compounds.

RESULTS

Our procedures allowed obtaining two complementary measures of delay discounting: (1) the percentage of choices of the delay option and (2) the mean adjusting delay, an index of the delay that animals choose more frequently. We found that our procedure rapidly establishes a baseline of choice behavior that remains stable over time and is highly sensitive to manipulations of the dopaminergic system.

CONCLUSIONS

This procedure may provide a useful tool for investigating the neurobiology of inter-temporal choice and decision-making.

Dissecting drug effects in preclinical models of impulsive choice: emphasis on glutamatergic compounds

RATIONALE

Impulsive choice is often measured with delay discounting paradigms. Because there are multiple discounting procedures, as well as different statistical analyses that can be applied to data generated from these paradigms, there are some inconsistencies in the literature regarding drug effects on impulsive choice.

OBJECTIVES

The goal of the current paper is to review the methodological and analytic approaches used to measure discounting and to discuss how these differences can account for differential drug effects observed across studies.

RESULTS

Because some procedures/analyses use a single data point as the dependent variable, changes in this value following pharmacological treatment may be interpreted as alterations in sensitivity to delayed reinforcement, but when other procedures/analyses are used, no changes in behavior are observed. Even when multiple data points are included, some studies show that the statistical analysis (e.g., ANOVA on raw proportion of responses vs. using hyperbolic/exponential functions) can lead to different interpretations. Finally, procedural differences (e.g., delay presentation order, signaling the delay to reinforcement, etc.) in the same discounting paradigm can alter how drugs affect sensitivity to delayed reinforcement.

CONCLUSIONS

Future studies should utilize paradigms that allow one to observe alterations in responding at each delay (e.g., concurrent-chains schedules). Concerning statistical analyses, using parameter estimates derived from nonlinear functions or incorporating the generalized matching law can allow one to determine if drugs affect sensitivity to delayed reinforcement or impair discrimination of the large and small magnitude reinforcers. Using these approaches can help further our understanding of the neurochemical underpinnings of delay discounting.

Nucleus accumbens core lesions induce sub-optimal choice and reduce sensitivity to magnitude and delay in impulsive choice tasks

The nucleus accumbens core (NAc) has long been recognized as an important contributor to the computation of reward value that is critical for impulsive choice behavior. Impulsive choice refers to choosing a smaller-sooner (SS) over a larger-later (LL) reward when the LL is more optimal in terms of the rate of reward delivery. Two experiments examined the role of the NAc in impulsive choice and its component processes of delay and magnitude processing. Experiment 1 delivered an impulsive choice task with manipulations of LL reward magnitude, followed by a reward magnitude discrimination task. Experiment 2 tested impulsive choice under manipulations of LL delay, followed by temporal bisection and progressive interval tasks. NAc lesions, in comparison to sham control lesions, produced suboptimal preferences that resulted in lower reward earning rates, and led to reduced sensitivity to magnitude and delay within the impulsive choice task. The secondary tasks revealed intact reward magnitude and delay discrimination abilities, but the lesion rats persisted in responding more as the progressive interval increased during the session. The results suggest that the NAc is most critical for demonstrating good sensitivity to magnitude and delay, and adjusting behavior accordingly. Ultimately, the NAc lesions induced suboptimal choice behavior rather than simply promoting impulsive choice, suggesting that an intact NAc is necessary for optimal decision making.

Impulsivity in rodents with a genetic predisposition for excessive alcohol consumption is associated with a lack of a prospective strategy

Increasing evidence supports the hypothesis that impulsive decision-making is a heritable risk factor for an alcohol use disorder (AUD). Clearly identifying a link between impulsivity and AUD risk, however, is complicated by the fact that both AUDs and impulsivity are heterogeneous constructs. Understanding the link between the two requires identifying the underlying cognitive factors that lead to impulsive choices. Rodent models have established that a family history of excessive drinking can lead to the expression of a transgenerational impulsive phenotype, suggesting heritable alterations in the decision-making process. In the present study, we explored the cognitive processes underlying impulsive choice in a validated, selectively bred rodent model of excessive drinking-the alcohol-preferring ("P") rat. Impulsivity was measured via delay discounting (DD), and P rats exhibited an impulsive phenotype as compared to their outbred foundation strain-Wistar rats. Steeper discounting in P rats was associated with a lack of a prospective behavioral strategy, which was observed in Wistar rats and was directly related to DD. To further explore the underlying cognitive factors mediating these observations, a drift diffusion model of DD was constructed. These simulations supported the hypothesis that prospective memory of the delayed reward guided choice decisions, slowed discounting, and optimized the fit of the model to the experimental data. Collectively, these data suggest that a deficit in forming or maintaining a prospective behavioral plan is a critical intermediary to delaying reward, and by extension, may underlie the inability to delay reward in those with increased AUD risk.

Differential effects of social and novelty enrichment on individual differences in impulsivity and behavioral flexibility

Early life experience profoundly impacts behavior and cognitive functions in rats. The present study investigated how the presence of conspecifics and/or novel objects, could independently influence individual differences in impulsivity and behavioral flexibility. Twenty-four rats were reared in an isolated condition, an isolated condition with a novel object, a pair-housed social condition, or a pair-housed social condition with a novel object. The rats were then tested on an impulsive choice task, a behavioral flexibility task, and an impulsive action task. Novelty enrichment produced an overall increase in impulsive choice, while social enrichment decreased impulsive choice in the absence of novelty enrichment and also produced an overall increase in impulsive action. In the behavioral flexibility task, social enrichment increased regressive errors, whereas both social and novelty enrichment reduced never-reinforced errors. Individual differences analyses indicated a significant relationship between performance in the behavioral flexibility and impulsive action tasks, which may reflect a common psychological correlate of action inhibition. Moreover, there was a relationship between delay sensitivity in the impulsive choice task and performance on the DRL and behavioral flexibility tasks, suggesting a dual role for timing and inhibitory processes in driving the interrelationship between these tasks. Overall, these results indicate that social and novelty enrichment produce distinct effects on impulsivity and adaptability, suggesting the need to parse out the different elements of enrichment in future studies. Further research is warranted to better understand how individual differences in sensitivity to enrichment affect individuals' interactions with and the resulting consequences of the rearing environment.

Dissociable frontostriatal white matter connectivity underlies reward and motor impulsivity

Dysfunction of cognitive control often leads to impulsive decision-making in clinical and healthy populations. Some research suggests that a generalized cognitive control mechanism underlies the ability to modulate various types of impulsive behavior, while other evidence suggests different forms of impulsivity are dissociable, and rely on distinct neural circuitry. Past research consistently implicates several brain regions, such as the striatum and portions of the prefrontal cortex, in impulsive behavior. However the ventral and dorsal striatum are distinct in regards to function and connectivity. Nascent evidence points to the importance of frontostriatal white matter connectivity in impulsivity, yet it remains unclear whether particular tracts relate to different control behaviors. Here we used probabilistic tractography of diffusion imaging data to relate ventral and dorsal frontostriatal connectivity to reward and motor impulsivity measures. We found a double dissociation such that individual differences in white matter connectivity between the ventral striatum and the ventromedial prefrontal cortex and dorsolateral prefrontal cortex was associated with reward impulsivity, as measured by delay discounting, whereas connectivity between dorsal striatum and supplementary motor area was associated with motor impulsivity, but not vice versa. Our findings suggest that (a) structural connectivity can is associated with a large amount of behavioral variation; (b) different types of impulsivity are driven by dissociable frontostriatal neural circuitry.

Adolescent methylmercury exposure affects choice and delay discounting in mice

The developing fetus is vulnerable to low-level exposure to methylmercury (MeHg), an environmental neurotoxicant, but the consequences of exposure during the adolescent period remain virtually unknown. The current experiments were designed to assess the effects of low-level MeHg exposure during adolescence on delay discounting, preference for small, immediate reinforcers over large, delayed ones, using a mouse model. Thirty-six male C57BL/6n mice were exposed to 0, 0.3, or 3.0ppm mercury (as MeHg) via drinking water from postnatal day 21 through 59, encompassing the murine adolescent period. As adults, mice lever pressed for a 0.01-cc droplet of milk solution delivered immediately or four 0.01-cc droplets delivered after a delay. Delays ranged from 1.26 to 70.79s, and all were presented within a session. A model based on the Generalized Matching Law indicated that sensitivity to reinforcer magnitude was lower for MeHg-exposed mice relative to controls, indicating that responding in MeHg-exposed mice was relatively indifferent to the larger reinforcer. Sensitivity to reinforcer delay was reduced (delay discounting was decreased) in the 0.3-ppm group, but not in the 3.0-ppm group, compared to controls. Adolescence is a developmental period during which the brain and behavior may be vulnerable to MeHg exposure. As with gestational MeHg exposure, the effects are reflected in the impact of reinforcing stimuli.

Delay discounting: Pigeon, rat, human--does it matter?

Delay discounting refers to the decrease in subjective value of an outcome as the time to its receipt increases. Across species and situations, animals discount delayed rewards, and their discounting is well-described by a hyperboloid function. The current review begins with a comparison of discounting models and the procedures used to assess delay discounting in nonhuman animals. We next discuss the generality of discounting, reviewing the effects of different variables on the degree of discounting delayed reinforcers by nonhuman animals. Despite the many similarities in discounting observed between human and nonhuman animals, several differences have been proposed (e.g., the magnitude effect; nonhuman animals discount over a matter of seconds whereas humans report willing to wait months, if not years before receiving a reward), raising the possibility of fundamental species differences in intertemporal choice. After evaluating these differences, we discuss delay discounting from an adaptationist perspective. The pervasiveness of discounting across species and situations suggests it is a fundamental process underlying decision making.

Behavioral correlates of the decision process in a dynamic environment: post-choice latencies reflect relative value and choice evaluation

One characteristic of natural environments is that outcomes vary across time. Animals need to adapt to these environmental changes and adjust their choices accordingly. In this experiment, we investigated the sensitivity with which rats could detect, and adapt to, multiple changes in the environment. Rats chose between two spouts which delivered 5% sucrose rewards with distinct probabilities. Across three phases, reward probabilities changed in size (large or small) and direction (increase or decrease). A discrete trial-structure was used, which allowed the choice process to be decomposed into three distinct response latency measures (choice execution latency, spout sampling duration, and trial-initiation latency). We found that a large decrease in reward probabilities rapidly produced the greatest change in choice proportions. The time taken to execute a choice reflected the differences in reward probabilities across the two spouts in some cases, but also reflected training history. By contrast, the amount of time rats spent responding at reward spouts in anticipation of reward consistently reflected the relative likelihood of reward across the two spouts and not the absolute probability of reward. The latency to initiate the subsequent trial reflected choice evaluation. These three response latencies thus indexed key behavioral correlates of the choice process as it unfolds in time. We discuss how this paradigm can be used to assess the corresponding neural correlates of decision-making.

Aging impairs deliberation and behavioral flexibility in inter-temporal choice

Inter-temporal choice depends on multiple, interacting systems, some of which may be compromised with age. Some of these systems may be responsible for ongoing trial-by-trial choice strategies. Some may represent the consequences of action. Some may be necessary for the coupling between anticipated consequences and strategies currently in use, flexibly guiding behavior. When faced with a difficult decision, rats will orient back and forth, a behavior termed "vicarious trial and error" (VTE). Recent experiments have linked the occurrence of VTE to hippocampal search processes and behavioral flexibility. We tested 5 month (n = 6), 9 month (n = 8) and over-27 month-old (n = 10) rats on a Spatial Adjusting Delay Discounting task to examine how aging impacted lap-by-lap strategies and VTE during inter-temporal choice. Rats chose between spatially separated food goals that provided a smaller-sooner or larger-later reward. On each lap, the delay to the larger-later reward was adjusted as a function of the rat's decisions, increasing by 1 s after delayed-side choices and decreasing by 1 s after non-delayed side choices. The strategies that aged rats used differed from those used in young and adult rats. Moreover, aged rats produced reliably more VTE behaviors, for protracted periods of time, uncoupled from behavioral flexibility.

Do the adjusting-delay and increasing-delay tasks measure the same construct: delay discounting?

Delay discounting describes the subjective devaluation of a reward when it is delayed. In animals, the adjusting-delay (AD) and increasing-delay (ID) tasks often are used to assess individual differences in, and drug effects on, delay discounting. No study to date, however, has compared systematically the measures of discounting produced in these tasks. The current study examined the correlation between measures of delay discounting derived from AD and ID procedures. Twenty rats completed 30 sessions under each task (order counterbalanced across rats). Quantitative measures of delay discounting produced by the two tasks were positively correlated, suggesting that the AD and ID tasks measure the same underlying facet of impulsive choice (i.e. individual or conjoint sensitivities to reward delay and magnitude). The measures derived from either task, however, depended on the sequences in which the tasks were experienced. That is, pre-exposure to one task decreased discounting of delayed rewards in the second task. Consistent with other published findings, exposure to delayed consequences during the initial discounting assessment might explain this effect. Despite the observed correlation between ID and AD indifference delays, we suggest that the ID procedure might be a more appropriate procedure for pharmacological studies.

Measurement of impulsive choice in rats: same- and alternate-form test-retest reliability and temporal tracking

Impulsive choice is typically measured by presenting smaller-sooner (SS) versus larger-later (LL) rewards, with biases towards the SS indicating impulsivity. The current study tested rats on different impulsive choice procedures with LL delay manipulations to assess same-form and alternate-form test-retest reliability. In the systematic-GE procedure (Green & Estle, 2003), the LL delay increased after several sessions of training; in the systematic-ER procedure (Evenden & Ryan, 1996), the delay increased within each session; and in the adjusting-M procedure (Mazur, 1987), the delay changed after each block of trials within a session based on each rat's choices in the previous block. In addition to measuring choice behavior, we also assessed temporal tracking of the LL delays using the median times of responding during LL trials. The two systematic procedures yielded similar results in both choice and temporal tracking measures following extensive training, whereas the adjusting procedure resulted in relatively more impulsive choices and poorer temporal tracking. Overall, the three procedures produced acceptable same form test-retest reliability over time, but the adjusting procedure did not show significant alternate form test-retest reliability with the other two procedures. The results suggest that systematic procedures may supply better measurements of impulsive choice in rats.

Homeostatic regulation of memory systems and adaptive decisions

While it is clear that many brain areas process mnemonic information, understanding how their interactions result in continuously adaptive behaviors has been a challenge. A homeostatic-regulated prediction model of memory is presented that considers the existence of a single memory system that is based on a multilevel coordinated and integrated network (from cells to neural systems) that determines the extent to which events and outcomes occur as predicted. The “multiple memory systems of the brain” have in common output that signals errors in the prediction of events and/or their outcomes, although these signals differ in terms of what the error signal represents (e.g., hippocampus: context prediction errors vs. midbrain/striatum: reward prediction errors). The prefrontal cortex likely plays a pivotal role in the coordination of prediction analysis within and across prediction brain areas. By virtue of its widespread control and influence, and intrinsic working memory mechanisms. Thus, the prefrontal cortex supports the flexible processing needed to generate adaptive behaviors and predict future outcomes. It is proposed that prefrontal cortex continually and automatically produces adaptive responses according to homeostatic regulatory principles: prefrontal cortex may serve as a controller that is intrinsically driven to maintain in prediction areas an experience-dependent firing rate set point that ensures adaptive temporally and spatially resolved neural responses to future prediction errors. This same drive by prefrontal cortex may also restore set point firing rates after deviations (i.e. prediction errors) are detected. In this way, prefrontal cortex contributes to reducing uncertainty in prediction systems. An emergent outcome of this homeostatic view may be the flexible and adaptive control that prefrontal cortex is known to implement (i.e. working memory) in the most challenging of situations. Compromise to any of the prediction circuits should result in rigid and suboptimal decision making and memory as seen in addiction and neurological disease. © 2013 The Authors. Hippocampus Published by Wiley Periodicals, Inc.

Assessing delay discounting in mice

Delay discounting (also intertemporal choice or impulsive choice) is the process by which delayed outcomes, such as delayed food delivery, are valued less than the same outcomes delivered immediately or with a shorter delay. This process is of interest because many psychopathologies, including substance dependence, pathological gambling, attention deficit hyperactivity disorder, and conduct disorder, are characterized by heightened levels of delay discounting. Some of these disorders are heritable, and data indicate that delay discounting also has a genetic component. To identify the genes underlying the delay discounting decision-making process and genetic correlates of heightened discounting, researchers have used mouse models. This unit describes a protocol for generating delay discounting behavior in mice and discusses analysis techniques for such behavior.

Subjective costs drive overly patient foraging strategies in rats on an intertemporal foraging task

Laboratory studies of decision making often take the form of two-alternative, forced-choice paradigms. In natural settings, however, many decision problems arise as stay/go choices. We designed a foraging task to test intertemporal decision making in rats via stay/go decisions. Subjects did not follow the rate-maximizing strategy of choosing only food items associated with short delays. Instead, rats were often willing to wait for surprisingly long periods, and consequently earned a lower rate of food intake than they might have by ignoring long-delay options. We tested whether foraging theory or delay discounting models predicted the behavior we observed but found that these models could not account for the strategies subjects selected. Subjects' behavior was well accounted for by a model that incorporated a cost for rejecting potential food items. Interestingly, subjects' cost sensitivity was proportional to environmental richness. These findings are at odds with traditional normative accounts of decision making but are consistent with retrospective considerations having a deleterious influence on decisions (as in the "sunk-cost" effect). More broadly, these findings highlight the utility of complementing existing assays of decision making with tasks that mimic more natural decision topologies.

Effect of orbitofrontal cortex lesions on temporal discounting in rats

Although choices of both humans and animals are more strongly influenced by immediate than delayed rewards, methodological limitations have made it difficult to estimate the precise form of temporal discounting in animals. In the present study, we sought to characterize temporal discounting in rats and to test the role of the orbitofrontal cortex (OFC) in this process. Rats were trained in a novel intertemporal choice task in which the sequence of delay durations was randomized across trials. The animals tended to choose a small immediate reward more frequently as the delay for a large reward increased, and, consistent with previous findings in other species, their choice behavior was better accounted for by hyperbolic than exponential discount functions. In addition, model comparisons showed that the animal's choice behavior was better accounted for by more complex discount functions with an additional parameter than a hyperbolic discount function. Following bilateral OFC lesions, rats extensively trained in this task showed no significant change in their intertemporal choice behavior. Our results suggest that the rodent OFC may not always play a role in temporal discounting when delays are randomized and/or after extensive training.

The effects of the previous outcome on probabilistic choice in rats

This study examined the effects of previous outcomes on subsequent choices in a probabilistic-choice task. Twenty-four rats were trained to choose between a certain outcome (1 or 3 pellets) versus an uncertain outcome (3 or 9 pellets), delivered with a probability of .1, .33, .67, and .9 in different phases. Uncertain outcome choices increased with the probability of uncertain food. Additionally, uncertain choices increased with the probability of uncertain food following both certain-choice outcomes and unrewarded uncertain choices. However, following uncertain-choice food outcomes, there was a tendency to choose the uncertain outcome in all cases, indicating that the rats continued to "gamble" after successful uncertain choices, regardless of the overall probability or magnitude of food. A subsequent manipulation, in which the probability of uncertain food varied within each session as a function of the previous uncertain outcome, examined how the previous outcome and probability of uncertain food affected choice in a dynamic environment. Uncertain-choice behavior increased with the probability of uncertain food. The rats exhibited increased sensitivity to probability changes and a greater degree of win-stay/lose-shift behavior than in the static phase. Simulations of two sequential choice models were performed to explore the possible mechanisms of reward value computations. The simulation results supported an exponentially decaying value function that updated as a function of trial (rather than time). These results emphasize the importance of analyzing global and local factors in choice behavior and suggest avenues for the future development of sequential-choice models.

Interactions between deliberation and delay-discounting in rats

When faced with decisions, rats sometimes pause and look back and forth between possible alternatives, a phenomenon termed vicarious trial and error (VTE). When it was first observed in the 1930s, VTE was theorized to be a mechanism for exploration. Later theories suggested that VTE aided the resolution of sensory or neuroeconomic conflict. In contrast, recent neurophysiological data suggest that VTE reflects a dynamic search and evaluation process. These theories make unique predictions about the timing of VTE on behavioral tasks. We tested these theories of VTE on a T-maze with return rails, where rats were given a choice between a smaller reward available after one delay or a larger reward available after an adjustable delay. Rats showed three clear phases of behavior on this task: investigation, characterized by discovery of task parameters; titration, characterized by iterative adjustment of the delay to a preferred interval; and exploitation, characterized by alternation to hold the delay at the preferred interval. We found that VTE events occurred during adjustment laps more often than during alternation laps. Results were incompatible with theories of VTE as an exploratory behavior, as reflecting sensory conflict, or as a simple neuroeconomic valuation process. Instead, our results were most consistent with VTE as reflecting a search process during deliberative decision making. This pattern of VTE that we observed is reminiscent of current navigational theories proposing a transition from a deliberative to a habitual decision-making mechanism.

Differences in impulsivity on a delay-discounting task predict self-administration of a low unit dose of methylphenidate in rats

There is controversy about the abuse liability of methylphenidate (MPH) in humans, and MPH has yet to be established fully as a reinforcer in rats. The present experiment examined whether intravenous MPH served as a reinforcer in rats, and how individual differences in impulsivity impacted MPH self-administration. Rats were exposed to a delay-discounting procedure, and then were implanted with an intravenous catheter to assess self-administration of 0.56 mg/kg/infusion MPH at different fixed ratio values. Self-administration rates of different MPH doses (0.03-1.0 mg/kg/infusion) were also examined. Both high and low impulsive rats acquired MPH self-administration at the same rate. All rats pressed more on the active lever than the inactive lever regardless of MPH dose, and pressed more for MPH than for saline. High impulsive rats self-administered more MPH than low impulsive rats at a low unit dose (0.1 mg/kg/infusion), though not at higher doses, indicating that individual differences in impulsive choice influence the dose-dependent reinforcing effects of MPH.

Valuation of uncertain and delayed rewards in primate prefrontal cortex

Humans and animals often must choose between rewards that differ in their qualities, magnitudes, immediacy, and likelihood, and must estimate these multiple reward parameters from their experience. However, the neural basis for such complex decision making is not well understood. To understand the role of the primate prefrontal cortex in determining the subjective value of delayed or uncertain reward, we examined the activity of individual prefrontal neurons during an inter-temporal choice task and a computer-simulated competitive game. Consistent with the findings from previous studies in humans and other animals, the monkey's behaviors during inter-temporal choice were well accounted for by a hyperbolic discount function. In addition, the activity of many neurons in the lateral prefrontal cortex reflected the signals related to the magnitude and delay of the reward expected from a particular action, and often encoded the difference in temporally discounted values that predicted the animal's choice. During a computerized matching pennies game, the animals approximated the optimal strategy, known as Nash equilibrium, using a reinforcement learning algorithm. We also found that many neurons in the lateral prefrontal cortex conveyed the signals related to the animal's previous choices and their outcomes, suggesting that this cortical area might play an important role in forming associations between actions and their outcomes. These results show that the primate lateral prefrontal cortex plays a central role in estimating the values of alternative actions based on multiple sources of information.

Cortico-limbic-striatal circuits subserving different forms of cost-benefit decision making

Research on the neural basis that underlies decision making in humans has revealed that these processes are mediated by distributed neural networks that incorporate different regions of the frontal lobes, the amygdala, the ventral striatum, and the dopamine system. In the present article, we review recent studies in rodents investigating the contribution of these systems to different forms of cost-benefit decision making and focus on evaluations related to delays, effort, or risks associated with certain rewards. Anatomically distinct regions of the medial and orbital prefrontal cortex make dissociable contributions to different forms of decision making, although lesions of these regions can induce variable effects, depending on the type of tasks used to assess these functions. The basolateral amygdala and the nucleus accumbens play a more fundamental role in these evaluations, helping an organism overcome different costs to obtain better rewards. Dopamine activity biases behavior toward more costly yet larger rewards, although abnormal increases in dopamine transmission can exert opposing actions on different types of decision making. The fact that similar neural circuits are recruited to solve these types of problems in both humans and animals suggests that animal models of decision making will prove useful in elucidating the mechanisms mediating these processes.

Steady-state assessment of impulsive choice in Lewis and Fischer 344 rats: between-condition delay manipulations

Previous research has shown that Lewis rats make more impulsive choices than Fischer 344 rats. Such strain-related differences in choice are important as they may provide an avenue for exploring genetic and neurochemical contributions to impulsive choice. The present systematic replication was designed to determine if these findings could be reproduced using a procedure less susceptible to within- or between-session carry-over effects that may have affected previous findings. Specifically, delays to the larger-later food reinforcer were manipulated between conditions following steady-state assessments of choice, and the order of delays across conditions was mixed. The results confirmed previous findings that Lewis rats made significantly more impulsive choices than Fischer 344 rats. Fischer 344 rats' preference for the larger-later reinforcer, on the other hand, was less extreme than reported in prior research, which may be due to carry-over effects inherent to the commonly used technique of systematically increasing delays within session. Previously reported across-strain motor differences were reproduced as Lewis rats had shorter latencies than Fischer 344 rats, although these latencies were not correlated with impulsive choice. Parallels between reduced dopamine function in Lewis rats and clinical reports of impulse-control disorders following treatment of Parkinson patients with selective D2/D3 dopamine agonists are discussed.

Effects of altering reinforcer magnitude and reinforcement schedule on phencyclidine (PCP) self-administration in monkeys using an adjusting delay task

Compared to nondrug reinforcers, few studies have examined delay discounting for drug reinforcers. The purpose of the present study was to examine delay discounting in rhesus monkeys using orally delivered phencyclidine (PCP) as the reinforcer and to examine the effects of manipulating reinforcer magnitude and cost on delay discounting for PCP using an adjusting delay task. Monkeys could choose between a single delivery of PCP available immediately or a bundle of PCP deliveries available following a titrated delay. The average of the delays, or the mean adjusted delay (MAD), served as the quantitative measure of delay discounting. In Experiment 1, reinforcer magnitude was manipulated by varying the PCP concentration and the size of the delayed reinforcer (6 or 12 deliveries). The concentration-effect curve for PCP deliveries assumed an inverted U-shaped function, but varying PCP concentration had little effect on MAD values or the choice between immediate and delayed reinforcers. Increasing the size of the delayed reinforcer produced an upward and leftward shift in the concentration-effect curve. In Experiment 2, the cost of reinforcers was manipulated by increasing the fixed ratio (FR) requirement for each choice. Increasing the FR led to increased MAD values and decreased PCP self-administration.

Prefrontal coding of temporally discounted values during intertemporal choice

Reward from a particular action is seldom immediate, and the influence of such delayed outcome on choice decreases with delay. It has been postulated that when faced with immediate and delayed rewards, decision makers choose the option with maximum temporally discounted value. We examined the preference of monkeys for delayed reward in an intertemporal choice task and the neural basis for real-time computation of temporally discounted values in the dorsolateral prefrontal cortex. During this task, the locations of the targets associated with small or large rewards and their corresponding delays were randomly varied. We found that prefrontal neurons often encoded the temporally discounted value of reward expected from a particular option. Furthermore, activity tended to increase with [corrected] discounted values for targets [corrected] presented in the neuron's preferred direction, suggesting that activity related to temporally discounted values in the prefrontal cortex might determine the animal's behavior during intertemporal choice.

Delay discounting and impulsive choice in the rat

Delay discounting refers to the degree to which immediate outcomes exhibit more influence over behavior than outcomes which are delayed. Impulsive choice, in the context of delay discounting, is generally considered as an increased preference for immediate over delayed outcomes, even where the delayed outcomes are more advantageous. In the past decade, there has been increasing use of delay-discounting paradigms to elucidate the physiological, pharmacological, and behavioral aspects of the putative neural circuitry underlying impulsive choice. This unit describes the assessment of impulsive choice in the rat using a delay-discounting procedure involving an operant response choice between a small reinforcer delivered immediately and a larger reinforcer delivered after a delay, which is progressively increased within a session. Variations of some of the main task parameters are also discussed, as well as their significance and interpretation.

Low-serotonin levels increase delayed reward discounting in humans

Previous animal experiments have shown that serotonin is involved in the control of impulsive choice, as characterized by high preference for small immediate rewards over larger delayed rewards. Previous human studies under serotonin manipulation, however, have been either inconclusive on the effect on impulsivity or have shown an effect in the speed of action-reward learning or the optimality of action choice. Here, we manipulated central serotonergic levels of healthy volunteers by dietary tryptophan depletion and loading. Subjects performed a "dynamic" delayed reward choice task that required a continuous update of the reward value estimates to maximize total gain. By using a computational model of delayed reward choice learning, we estimated the parameters governing the subjects' reward choices in low-, normal, and high-serotonin conditions. We found an increase of proportion in small reward choices, together with an increase in the rate of discounting of delayed rewards in the low-serotonin condition compared with the control and high-serotonin conditions. There were no significant differences between conditions in the speed of learning of the estimated delayed reward values or in the variability of reward choice. Therefore, in line with previous animal experiments, our results show that low-serotonin levels steepen delayed reward discounting in humans. The combined results of our previous and current studies suggest that serotonin may adjust the rate of delayed reward discounting via the modulation of specific loops in parallel corticobasal ganglia circuits.

Iron deprivation during fetal development changes the behavior of juvenile rhesus monkeys

Sensitive periods for induction of behavioral impairments by developmental iron deficiency were studied in a nonhuman primate model. Rhesus monkey infants were deprived of iron prenatally (n = 14) via the dam's diet (10 microg Fe/g) or postnatally (birth-4 mo, n = 12) via infant formula (1.5 mg Fe/L). They were compared with controls (n = 12) with adequate dietary iron throughout development in a series of cognitive tests and related assessments from 6 to 12 mo of age, a developmental stage corresponding approximately to 2-4 y of age in humans. Health, growth, and hematological status were not affected. Auditory brainstem response and white matter volumes in the cerebrum were similarly unaffected. Male infants in the prenatally deprived group had reduced spontaneous daytime activity relative to controls, as monitored by actimeter. On cognitive tests, prenatally deprived juveniles had similar level of correct responding, but showed more completed trials, and shorter latencies during early phases of the tests. Juveniles deprived of iron as infants showed a similar pattern of behavioral change, but most differences from controls were not as great. Inadequate iron nutrition during pregnancy was reflected in the juvenile period primarily as attenuated inhibitory response. This finding may be relevant to individual differences in temperament or to behavior disorders in children involving reduced inhibitory control.

Delay aversion: Effects of 7-OH-DPAT, 5-HT1A/1B-receptor stimulation and d-cycloserine

Impulsive individuals often display an aversion to waiting for rewards. Delay aversion can be quantified in rats in a delayed reward task, in which animals choose between an immediately available, small reward, and a large reward available after a delay. In previous research conducted at our laboratory and in literature, positive correlations between delay aversion and aggression, substance abuse and persistence during extinction of conditioned responses were found. The correlations suggest a possible shared pharmacology. Therefore, we tested drugs with known effects on these behaviors for possible effects on delay aversion: the dopamine D(3)-receptor agonist 7-OH-DPAT, the 5-HT(1A)-receptor agonist flesinoxan, the 5HT(1A/1B)-receptor agonist eltoprazine, and the NMDA-receptor agonist d-cycloserine. The results show that 7-OH-DPAT slightly decreased choice for the large reward. Flesinoxan disrupted task execution by lowering choice for the large reward even at a delay of 0 s. Eltoprazine slightly increased choice for the large reward, but the 5-HT(1B)-antagonist GR127935 had no effect. Administration of D-cycloserine also had no effect on choice behavior. The data suggest the dopamine D(3)-receptor and the 5-HT(1B)-receptor are interesting targets for treating delay aversion impulsivity. These targets were correctly predicted by the positive correlation between delay aversion and aggressive behavior and the intimate links between delay aversion and substance abuse disorders.

Effects of reinforcer probability, delay, and response requirements on the choices of rats and pigeons: possible species differences

In Experiment 1 with rats, a left lever press led to a 5-s delay and then a possible reinforcer. A right lever press led to an adjusting delay and then a certain reinforcer. This delay was adjusted over trials to estimate an indifference point, or a delay at which the two alternatives were chosen about equally often. Indifference points increased as the probability of reinforcement for the left lever decreased. In some conditions with a 20% chance of food, a light above the left lever was lit during the 5-s delay on all trials, but in other conditions, the light was only lit on those trials that ended with food. Unlike previous results with pigeons, the presence or absence of the delay light on no-food trials had no effect on the rats' indifference points. In other conditions, the rats showed less preference for the 20% alternative when the time between trials was longer. In Experiment 2 with rats, fixed-interval schedules were used instead of simple delays, and the presence or absence of the fixed-interval requirement on no-food trials had no effect on the indifference points. In Experiment 3 with rats and Experiment 4 with pigeons, the animals chose between a fixed-ratio 8 schedule that led to food on 33% of the trials and an adjusting-ratio schedule with food on 100% of the trials. Surprisingly, the rats showed less preference for the 33% alternative in conditions in which the ratio requirement was omitted on no-food trials. For the pigeons, the presence or absence of the ratio requirement on no-food trials had little effect. The results suggest that there may be differences between rats and pigeons in how they respond in choice situations involving delayed and probabilistic reinforcers.

Impulsivity (delay discounting) as a predictor of acquisition of IV cocaine self-administration in female rats

RATIONALE

Previous research in humans suggests a relationship between drug abuse and impulsivity as shown by selection of a smaller immediate reward over a larger delayed reward. However, it is not clear whether impulsivity precedes drug abuse or drug abuse influences impulsivity.

OBJECTIVE

The hypothesis of the present experiment was that rats selected for choosing smaller, immediate over larger, delayed food would acquire IV cocaine self-administration faster than those choosing larger, delayed food rewards.

METHODS

Female rats were screened for locomotor activity and trained on a delay discounting procedure that allowed them access to two response levers and a food pellet dispenser. Under a fixed-ratio (FR) 1 schedule, responding on one lever resulted in immediate delivery of one 45 mg pellet, while responding on the other lever resulted in delivery of three 45 mg pellets after a variable delay that increased after responses on the delay lever and decreased after responses on the immediate lever. For each rat, a mean adjusted delay (MAD) was calculated for each daily session, and stability was defined as MADs varying less than 5 s across 5 days. Based on their average MADs, rats were separated into low impulsive (LoI) and high impulsive (HiI) groups, implanted with an indwelling IV catheter, and trained to lever press for cocaine (0.2 mg/kg) under an FR1 schedule.

RESULTS

There were no differences in locomotor activity between the LoI and HiI groups; however, a greater percentage of the HiI group acquired cocaine self-administration, and they did so at a significantly faster rate than the LoI rats.

CONCLUSIONS

Performance on the delay discounting model of impulsivity predicted vulnerability to subsequent acquisition of cocaine self-administration.