Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.

Interpretable deep learning for deconvolutional analysis of neural signals

The widespread adoption of deep learning to build models that capture the dynamics of neural populations is typically based on "black-box" approaches that lack an interpretable link between neural activity and function. Here, we propose to apply algorithm unrolling, a method for interpretable deep learning, to design the architecture of sparse deconvolutional neural networks and obtain a direct interpretation of network weights in relation to stimulus-driven single-neuron activity through a generative model. We characterize our method, referred to as deconvolutional unrolled neural learning (DUNL), and show its versatility by applying it to deconvolve single-trial local signals across multiple brain areas and recording modalities. To exemplify use cases of our decomposition method, we uncover multiplexed salience and reward prediction error signals from midbrain dopamine neurons in an unbiased manner, perform simultaneous event detection and characterization in somatosensory thalamus recordings, and characterize the responses of neurons in the piriform cortex. Our work leverages the advances in interpretable deep learning to gain a mechanistic understanding of neural dynamics.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

Tonic dopamine and biases in value learning linked through a biologically inspired reinforcement learning model

A hallmark of various psychiatric disorders is biased future predictions. Here we examined the mechanisms for biased value learning using reinforcement learning models incorporating recent findings on synaptic plasticity and opponent circuit mechanisms in the basal ganglia. We show that variations in tonic dopamine can alter the balance between learning from positive and negative reward prediction errors, leading to biased value predictions. This bias arises from the sigmoidal shapes of the dose-occupancy curves and distinct affinities of D1- and D2-type dopamine receptors: changes in tonic dopamine differentially alters the slope of the dose-occupancy curves of these receptors, thus sensitivities, at baseline dopamine concentrations. We show that this mechanism can explain biased value learning in both mice and humans and may also contribute to symptoms observed in psychiatric disorders. Our model provides a foundation for understanding the basal ganglia circuit and underscores the significance of tonic dopamine in modulating learning processes.

Multi-timescale reinforcement learning in the brain

To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents2-6 and at characterizing the firing of dopamine neurons in the midbrain7-9. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, controlled by the discount factor. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, a mechanistic basis for the empirical observation that humans and animals use non-exponential discounts in many situations10-14, and open new avenues for the design of more efficient reinforcement learning algorithms.

Glutamate inputs send prediction error of reward but not negative value of aversive stimuli to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs) but the mechanisms underlying RPE computation, particularly contributions of different neurotransmitters, remain poorly understood. Here we used a genetically-encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons. We found that glutamate inputs exhibit virtually all of the characteristics of RPE, rather than conveying a specific component of RPE computation such as reward or expectation. Notably, while glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli toward more positive responses, while excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Competitive integration of time and reward explains value-sensitive foraging decisions and frontal cortex ramping dynamics

The ability to make advantageous decisions is critical for animals to ensure their survival. Patch foraging is a natural decision-making process in which animals decide when to leave a patch of depleting resources to search for a new one. To study the algorithmic and neural basis of patch foraging behavior in a controlled laboratory setting, we developed a virtual foraging task for head-fixed mice. Mouse behavior could be explained by ramp-to-threshold models integrating time and rewards antagonistically. Accurate behavioral modeling required inclusion of a slowly varying "patience" variable, which modulated sensitivity to time. To investigate the neural basis of this decision-making process, we performed dense electrophysiological recordings with Neuropixels probes broadly throughout frontal cortex and underlying subcortical areas. We found that decision variables from the reward integrator model were represented in neural activity, most robustly in frontal cortical areas. Regression modeling followed by unsupervised clustering identified a subset of neurons with ramping activity. These neurons' firing rates ramped up gradually in single trials over long time scales (up to tens of seconds), were inhibited by rewards, and were better described as being generated by a continuous ramp rather than a discrete stepping process. Together, these results identify reward integration via a continuous ramping process in frontal cortex as a likely candidate for the mechanism by which the mammalian brain solves patch foraging problems.

Emergence of belief-like representations through reinforcement learning

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming "beliefs"-optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN's learned representation encodes belief information, but only when the RNN's capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

A Hypothalamic Circuit Underlying the Dynamic Control of Social Homeostasis

Social grouping increases survival in many species, including humans1,2. By contrast, social isolation generates an aversive state (loneliness) that motivates social seeking and heightens social interaction upon reunion3-5. The observed rebound in social interaction triggered by isolation suggests a homeostatic process underlying the control of social drive, similar to that observed for physiological needs such as hunger, thirst or sleep3,6. In this study, we assessed social responses in multiple mouse strains and identified the FVB/NJ line as exquisitely sensitive to social isolation. Using FVB/NJ mice, we uncovered two previously uncharacterized neuronal populations in the hypothalamic preoptic nucleus that are activated during social isolation and social rebound and that orchestrate the behavior display of social need and social satiety, respectively. We identified direct connectivity between these two populations of opposite function and with brain areas associated with social behavior, emotional state, reward, and physiological needs, and showed that animals require touch to assess the presence of others and fulfill their social need, thus revealing a brain-wide neural system underlying social homeostasis. These findings offer mechanistic insight into the nature and function of circuits controlling instinctive social need and for the understanding of healthy and diseased brain states associated with social context.

Emergence of belief-like representations through reinforcement learning

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming "beliefs"-optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN's learned representation encodes belief information, but only when the RNN's capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

Author Summary

Natural environments are full of uncertainty. For example, just because my fridge had food in it yesterday does not mean it will have food today. Despite such uncertainty, animals can estimate which states and actions are the most valuable. Previous work suggests that animals estimate value using a brain area called the basal ganglia, using a process resembling a reinforcement learning algorithm called TD learning. However, traditional reinforcement learning algorithms cannot accurately estimate value in environments with state uncertainty (e.g., when my fridge's contents are unknown). One way around this problem is if agents form "beliefs," a probabilistic estimate of how likely each state is, given any observations so far. However, estimating beliefs is a demanding process that may not be possible for animals in more complex environments. Here we show that an artificial recurrent neural network (RNN) trained with TD learning can estimate value from observations, without explicitly estimating beliefs. The trained RNN's error signals resembled the neural activity of dopamine neurons measured during the same task. Importantly, the RNN's activity resembled beliefs, but only when the RNN had enough capacity. This work illustrates how animals could estimate value in uncertain environments without needing to first form beliefs, which may be useful in environments where computing the true beliefs is too costly.

Spontaneous behaviour is structured by reinforcement without explicit reward

Spontaneous animal behaviour is built from action modules that are concatenated by the brain into sequences1,2. However, the neural mechanisms that guide the composition of naturalistic, self-motivated behaviour remain unknown. Here we show that dopamine systematically fluctuates in the dorsolateral striatum (DLS) as mice spontaneously express sub-second behavioural modules, despite the absence of task structure, sensory cues or exogenous reward. Photometric recordings and calibrated closed-loop optogenetic manipulations during open field behaviour demonstrate that DLS dopamine fluctuations increase sequence variation over seconds, reinforce the use of associated behavioural modules over minutes, and modulate the vigour with which modules are expressed, without directly influencing movement initiation or moment-to-moment kinematics. Although the reinforcing effects of optogenetic DLS dopamine manipulations vary across behavioural modules and individual mice, these differences are well predicted by observed variation in the relationships between endogenous dopamine and module use. Consistent with the possibility that DLS dopamine fluctuations act as a teaching signal, mice build sequences during exploration as if to maximize dopamine. Together, these findings suggest a model in which the same circuits and computations that govern action choices in structured tasks have a key role in sculpting the content of unconstrained, high-dimensional, spontaneous behaviour.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning

A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multiple trials. Here we demonstrate that such a gradual shift occurs both at the level of dopaminergic cellular activity and dopamine release in the ventral striatum in mice. Our results establish a long-sought link between dopaminergic activity and the TD learning algorithm, providing fundamental insights into how the brain associates cues and rewards that are separated in time.

O-015 Results of urological consultation in the setting of IVF clinic

Study question

In the management of male infertility, we investigated whether urological consultation could improve the live birth rate, and who should visit urologists in the setting of IVF clinic.

Summary answer

Urologic consultation resulted in improvement of semen quality and live birth rate with more IVF use in those with adverse semen parameters.

What is known already

Male factor infertility exists in about a half of infertility couples. This accounts for about 8% in male reproductive age. Therefore, ideally every male partner of infertility couples attempting conception should have a urological evaluation. However, it is not very easy to access urologists who specialized in reproductive medicine in Japan because we have very few of such urologists. One the other hand, a certain number of couples are wasting their time during IVF failure without urological evaluation.

Study design, size, duration

This is a single-institution retrospective study. We enrolled male partners of infertility couples who visited Kameda IVF clinic Makuhari, Chiba, Japan, between May 2016 and December 2020 and followed at least one year. Live birth rate and the frequency of IVF use were investigated according to semen quality and urological consultation status. Chi-square tests and T tests were used to compare the results between groups.

Participants/materials, setting, methods

Among 2225 couples who visited Kameda IVF clinic Makuhari, 803 male partners (Group A, 36.0%) were evaluated by urologists who were specialized in male reproductive medicine. Remaining 1422 patients did not (Group B, 64.0%). Lifestyle evaluation, physical examination, semen analyses, scrotal ultrasonography, blood test including sexual hormones and zinc concentration were performed in Group A. Semen analyses and lifestyle evaluation were performed in Group B. Urological treatments were done according to factors of male infertility.

Main results and the role of chance

Semen quality was worse in Group A as compared to Group B (sperm motility, 28.5±16.9% vs. 46.0±17.0%; total sperm count, 105±108 million/mL vs. 176±155; total motile sperm count, 34±49 vs.87±98; mean±S.D.; p = 0.0001, 0.0001, 0.0001, A vs. B, respectively). After urologic consultation and managements, sperm motility was improved to 34±18% (p = 0.001). Live birth rate in groups A and B were similar (56.0% vs. 57.2%), however couples who obtained a child in Group A used IVF more often than those in Group B (70% vs. 49.9%, p < 0.001). Among those with adverse semen quality (total motile sperm count less than 15.6 million/mL, n = 472), 350 visited urologists (Group 1, 74.2%) and remaining 122 did not (Group 2, 25.8%). Live birth rate in Group 1 was significantly better than in Group 2 (65.3% vs. 54.1%, p = 0.0359). Use of IVF was significantly more frequent in Group 1 than Group 2 (79.3% vs. 63.6%, p = 0.0359) among who obtained a child. In those with better semen quality (motile sperm count >50 million, n = 900), 119 visited urologist (31.1%, Group 3) and 781 did not (Group 4). Live birth rate and the use of IVF were not different between Groups 3 and 4 (51.1% vs.60.9%; 50.4% vs. 62.9%).

Limitations, reasons for caution

This study is a single-institution, retrospective study in the setting of IVF clinic. There may be a selection bias since men first visit gynecologists. These could affect the study results.

Wider implications of the findings

In the setting of IVF clinic, urologic consultation resulted in improved semen quality and better live birth rate with the use of IVF, especially in those who have adverse semen parameters. The results of this study encourage patients to see urologists and physicians to introduce urologist to patients.

Trial registration number

not applicable

P-199 Effect of the area of oocyte perivitelline space on the fertilization and embryo development following intracytoplasmic sperm injection

Study question

Does the area of oocyte perivitelline space have an effect on fertilization and embryo development following intracytoplasmic sperm injection?

Summary answer

The area of oocyte perivitelline space has not an effect on the fertilization but the embryo development following intracytoplasmic sperm injection.

What is known already

Oocyte perivitelline space has a lot of variation at intracytoplasmic sperm injection (ICSI). Some researchers reported that the characteristics of perivitelline space (large or small) affect embryo development, pregnancy, and implantation. However, these studies did not accurately calculate the area of perivitelline space. Therefore, little information is available on the effect of the area of oocyte perivitelline space on fertilization and embryo development following ICSI. The purpose of this study was to calculate and classify the area of oocyte perivitelline space and investigate the effect of the area of perivitelline space on fertilization and embryo development following ICSI.

Study design, size, duration

1. We retrospectively investigated 634 mature oocytes that were conducted ICSI between January 2021 and December 2021. The area of each oocyte perivitelline space was defined from between the area of circle calculated from the inner layer of zona pellucida and cytoplasm and divided into 3 groups (-9%, 10-19%, 20%-).

2. We retrospectively calculated the diameter of an inner layer of zona pellucida and cytoplasm and compared it with the 3 groups (-9%, 10-19%, 20%-).

Participants/materials, setting, methods

1. The fertilization, survival, good quality day-3 embryo, blastocyst, good quality blastocyst rates following ICSI were compared with the 3 groups (-9%, 10-19%, 20%-).

2. The average diameter of an inner layer of zona pellucida and cytoplasm of each oocyte for the 3 groups (-9%, 10-19%, 20%-) were compared.

The data were analyzed by Fisher’s exact test, residual analysis, one-way ANOVA test, with Bonferroni correction as appropriate to determine the statistical differences among groups.

Main results and the role of chance

1. The survival rates of perivitelline space -9%, 10-19%, 20%- groups were 100% (109/109), 96% (363/378), 94% (138/147), the fertilization rates were 89% (97/109), 88% (331/378), 86% (127/147), the good quality day-3 embryo rates were 56% (54/97), 70% (232/331), 70% (89/127) respectively. No significant difference was observed between these comparison items. The blastocyst rates of perivitelline space -9%, 10-19%, 20%- groups were 51% (47/92), 69% (222/321), 82% (93/114), the good quality blastocyst rates were 22% (20/92), 40% (129/321), 52% (59/114) respectively. The blastocyst and good quality blastocyst rates of perivitelline space -9% group showed significantly lower results. On the other hand, the blastocyst and good quality blastocyst rates of perivitelline space 20%- group showed significantly higher results.

2. The average diameter of an inner layer of zona pellucida of perivitelline space -9%, 10-19%, 20%- groups were 125 ± 4 µm, 129 ± 5 µm, 136 ± 6 µm, the average diameter of the cytoplasm of perivitelline space were 121 ± 4 µm, 119 ± 4 µm, 118 ± 4 µm respectively. Significant differences were observed in all pairs of groups of the average diameter of an inner layer of zona pellucida and cytoplasm.

Limitations, reasons for caution

The area of oocyte perivitelline space was calculated at only one plane.

Wider implications of the findings

Oocytes with narrow perivitelline space might have a wide region of adhesive between the cytoplasm surface and an inner layer of the zona pellucida which resulted in a smaller diameter of the zona pellucida and lower blastocyst rate by forming cytoplasmic fragments (Yumoto K et al. JARG. 2020 ;37(6):1349-1354.).

Trial registration number

Not Applicable

Dopamine signals as temporal difference errors: recent advances

In the brain, dopamine is thought to drive reward-based learning by signaling temporal difference reward prediction errors (TD errors), a 'teaching signal' used to train computers. Recent studies using optogenetic manipulations have provided multiple pieces of evidence supporting that phasic dopamine signals function as TD errors. Furthermore, novel experimental results have indicated that when the current state of the environment is uncertain, dopamine neurons compute TD errors using 'belief states' or a probability distribution over potential states. It remains unclear how belief states are computed but emerging evidence suggests involvement of the prefrontal cortex and the hippocampus. These results refine our understanding of the role of dopamine in learning and the algorithms by which dopamine functions in the brain.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

A Unified Framework for Dopamine Signals across Timescales

Rapid phasic activity of midbrain dopamine neurons is thought to signal reward prediction errors (RPEs), resembling temporal difference errors used in machine learning. However, recent studies describing slowly increasing dopamine signals have instead proposed that they represent state values and arise independent from somatic spiking activity. Here we developed experimental paradigms using virtual reality that disambiguate RPEs from values. We examined dopamine circuit activity at various stages, including somatic spiking, calcium signals at somata and axons, and striatal dopamine concentrations. Our results demonstrate that ramping dopamine signals are consistent with RPEs rather than value, and this ramping is observed at all stages examined. Ramping dopamine signals can be driven by a dynamic stimulus that indicates a gradual approach to a reward. We provide a unified computational understanding of rapid phasic and slowly ramping dopamine signals: dopamine neurons perform a derivative-like computation over values on a moment-by-moment basis.

Distributional Reinforcement Learning in the Brain

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Reinforcement biases subsequent perceptual decisions when confidence is low, a widespread behavioral phenomenon

Learning from successes and failures often improves the quality of subsequent decisions. Past outcomes, however, should not influence purely perceptual decisions after task acquisition is complete since these are designed so that only sensory evidence determines the correct choice. Yet, numerous studies report that outcomes can bias perceptual decisions, causing spurious changes in choice behavior without improving accuracy. Here we show that the effects of reward on perceptual decisions are principled: past rewards bias future choices specifically when previous choice was difficult and hence decision confidence was low. We identified this phenomenon in six datasets from four laboratories, across mice, rats, and humans, and sensory modalities from olfaction and audition to vision. We show that this choice-updating strategy can be explained by reinforcement learning models incorporating statistical decision confidence into their teaching signals. Thus, reinforcement learning mechanisms are continually engaged to produce systematic adjustments of choices even in well-learned perceptual decisions in order to optimize behavior in an uncertain world.

The slow earthquake spectrum in the Japan Trench illuminated by the S-net seafloor observatories

Investigating slow earthquake activity in subduction zones provides insight into the slip behavior of megathrusts, which can provide important clues about the rupture extent of future great earthquakes. Using the S-net ocean-bottom seismograph network along the Japan Trench, we mapped a detailed distribution of tectonic tremors, which coincided with very-low-frequency earthquakes and a slow slip event. Compiling these and other related observations, including repeating earthquakes and earthquake swarms, we found that the slow earthquake distribution is complementary to the Tohoku-Oki earthquake rupture. We used our observations to divide the megathrust in the Japan Trench into three along-strike segments characterized by different slip behaviors. We found that the rupture of the Tohoku-Oki earthquake, which nucleated in the central segment, was terminated by the two adjacent segments.

Believing in dopamine

Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.

Multiple Dopamine Systems: Weal and Woe of Dopamine

The ability to predict future outcomes increases the fitness of the animal. Decades of research have shown that dopamine neurons broadcast reward prediction error (RPE) signals-the discrepancy between actual and predicted reward-to drive learning to predict future outcomes. Recent studies have begun to show, however, that dopamine neurons are more diverse than previously thought. In this review, we will summarize a series of our studies that have shown unique properties of dopamine neurons projecting to the posterior "tail" of the striatum (TS) in terms of anatomy, activity, and function. Specifically, TS-projecting dopamine neurons are activated by a subset of negative events including threats from a novel object, send prediction errors for external threats, and reinforce avoidance behaviors. These results indicate that there are at least two axes of dopamine-mediated reinforcement learning in the brain-one learning from canonical RPEs and another learning from threat prediction errors. We argue that the existence of multiple learning systems is an adaptive strategy that makes possible each system optimized for its own needs. The compartmental organization in the mammalian striatum resembles that of a dopamine-recipient area in insects (mushroom body), pointing to a principle of dopamine function conserved across phyla.

Dopamine neurons projecting to the posterior striatum reinforce avoidance of threatening stimuli

Midbrain dopamine neurons are well known for their role in reward-based reinforcement learning. We found that the activity of dopamine axons in the posterior tail of the striatum (TS) scaled with the novelty and intensity of external stimuli, but did not encode reward value. We demonstrated that the ablation of TS-projecting dopamine neurons specifically inhibited avoidance of novel or high-intensity stimuli without affecting animals' initial avoidance responses, suggesting a role in reinforcement rather than simply in avoidance itself. Furthermore, we found that animals avoided optogenetic activation of dopamine axons in TS during a choice task and that this stimulation could partially reinstate avoidance of a familiar object. These results suggest that TS-projecting dopamine neurons reinforce avoidance of threatening stimuli. More generally, our results indicate that there are at least two axes of reinforcement learning using dopamine in the striatum: one based on value and one based on external threat.

Publisher Correction: Shedding new light on the Crab with polarized X-rays

A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has not been fixed in the paper.

Belief state representation in the dopamine system

Learning to predict future outcomes is critical for driving appropriate behaviors. Reinforcement learning (RL) models have successfully accounted for such learning, relying on reward prediction errors (RPEs) signaled by midbrain dopamine neurons. It has been proposed that when sensory data provide only ambiguous information about which state an animal is in, it can predict reward based on a set of probabilities assigned to hypothetical states (called the belief state). Here we examine how dopamine RPEs and subsequent learning are regulated under state uncertainty. Mice are first trained in a task with two potential states defined by different reward amounts. During testing, intermediate-sized rewards are given in rare trials. Dopamine activity is a non-monotonic function of reward size, consistent with RL models operating on belief states. Furthermore, the magnitude of dopamine responses quantitatively predicts changes in behavior. These results establish the critical role of state inference in RL.

Radionuclide Assessment of the Natural Heart Ejection Fraction before and after LVAD Implantation

Complete pressure unloading of the ventricles can preserve ischemically damaged myocardium. Most clinical left heart assist device (LVAD) systems used after ischemic injury of the heart apply atrial cannulation which does not ensure pressure unloading. In order to assess the effect of the implantation of an intracorporeal LVAD on the function of the natural heart, we determined the ejection fraction (EF) in four male Holstein calves (90–105 kg) before and after insertion of a Cleveland Clinic pneumatic LVAD. A gated blood pool scan was obtained with a gamma camera after injection of 40 mCi Tc-labelled albumin. The animals were restrained in a sling to avoid movement artifacts.

All animals showed a drop of 65 ± 12% to 42 ± 14% EF in the first postoperative (p.o.) week. Left ventricular output did not maintain sufficient blood pressure as assessed by pump-off tests. Systolic blood pressure dropped from 122 ±6.5 mm Hg to 81 ± 6 mm Hg without pump support on the morning of the first p.o. day.

Apical coring and possible restrained heart movement by the implanted LVAD may lead to impaired myocardial function that renders the individual LVAD dependent until adaptative corrections take place.

The Medial Prefrontal Cortex Shapes Dopamine Reward Prediction Errors under State Uncertainty

Animals make predictions based on currently available information. In natural settings, sensory cues may not reveal complete information, requiring the animal to infer the "hidden state" of the environment. The brain structures important in hidden state inference remain unknown. A previous study showed that midbrain dopamine neurons exhibit distinct response patterns depending on whether reward is delivered in 100% (task 1) or 90% of trials (task 2) in a classical conditioning task. Here we found that inactivation of the medial prefrontal cortex (mPFC) affected dopaminergic signaling in task 2, in which the hidden state must be inferred ("will reward come or not?"), but not in task 1, where the state was known with certainty. Computational modeling suggests that the effects of inactivation are best explained by a circuit in which the mPFC conveys inference over hidden states to the dopamine system. VIDEO ABSTRACT.

Can the Artificial Heart Make the Circulation Become Fractal?

In order to analyze the hemodynamic parameters in prosthetic circulation as an entity and not as decomposed parts, non linear mathematical analyzing techniques, including the fractal dimension analyzing theory, were utilized. Two pneumatically actuated ventricular assist devices were implanted, as biventricular bypasses (BVB), in chronic animal experiments, using four healthy adult goats. For the comparison between the natural and prosthetic circulation in the same animals, the BVB type complete prosthetic circulation model with ventricular fibrillation, was adopted. All hemodynamic parameters with natural and prosthetic circulation were recorded under awake conditions, and calculated with a personal computer system. Using the non-linear mathematical technique, the arterial blood pressure waveform was embedded into the return map as the beat-to-beat time series data and fractal dimension analysis were performed to analyze the reconstructed attractor. By the use of the Box counting method, fractal dimension analysis of the hemodynamics was performed. Return map of the hemodynamics during natural and artificial circulation showed fractal characteristics, and fractal dimension analysis of the arterial blood pressure revealed the fact that lower dimensional fractal dynamics were evident during prosthetic circulation. Fractal time series data is suggested to have robustness and error resistance, thus our results suggest that the circulatory regulatory system with an artificial heart may have these desired characteristics.

A Self-Killing Rabies Virus That Leaves a Trace on the DNA

Although modified rabies viruses have emerged as a powerful tool for tracing the inputs to genetically defined populations of neurons, the toxicity of the virus has limited its utility. A recent study employed a self-inactivating rabies (SiR) virus that enables recording or manipulation of targeted neurons for months.

Somatosensory Cortex Plays an Essential Role in Forelimb Motor Adaptation in Mice

Our motor outputs are constantly re-calibrated to adapt to systematic perturbations. This motor adaptation is thought to depend on the ability to form a memory of a systematic perturbation, often called an internal model. However, the mechanisms underlying the formation, storage, and expression of such models remain unknown. Here, we developed a mouse model to study forelimb adaptation to force field perturbations. We found that temporally precise photoinhibition of somatosensory cortex (S1) applied concurrently with the force field abolished the ability to update subsequent motor commands needed to reduce motor errors. This S1 photoinhibition did not impair basic motor patterns, post-perturbation completion of the action, or their performance in a reward-based learning task. Moreover, S1 photoinhibition after partial adaptation blocked further adaptation, but did not affect the expression of already-adapted motor commands. Thus, S1 is critically involved in updating the memory about the perturbation that is essential for forelimb motor adaptation.

Impact of the presence of HLA 1-locus mismatch and the use of low-dose antithymocyte globulin in unrelated bone marrow transplantation

HLA 1-locus-mismatched unrelated donors (1MMUD) have been used in allogeneic hematopoietic stem cell transplantation (allo-HCT) for patients who lack an HLA-matched donor. We retrospectively analyzed 3313 patients with acute leukemia or myelodysplastic syndrome who underwent bone marrow transplantation from an HLA allele-matched unrelated donor (MUD) or 1MMUD between 2009 and 2014. We compared the outcomes of MUD (n=2089) and 1MMUD with antithymocyte globulin (ATG) (1MM-ATG(+); n=109) with those of 1MMUD without ATG (1MM-ATG(-); n=1115). The median total dose of ATG (thymoglobulin) was 2.5 mg/kg (range 1.0-11.0 mg/kg) in the 1MM-ATG(+) group. The rates of grade III-IV acute GvHD, non-relapse mortality (NRM) and overall mortality were significantly lower in the MUD group than in the 1MM-ATG(-) group (hazard ratio (HR) 0.77, P=0.016; HR 0.74; P<0.001; and HR 0.87, P=0.020, respectively). Likewise, the rates of grade III-IV acute GVHD, NRM and overall mortality were significantly lower in the 1MM-ATG(+) group than in the 1MM-ATG(-) group (HR 0.42, P=0.035; HR 0.35, P<0.001; and HR 0.71, P=0.042, respectively). The outcome of allo-HCT from 1MM-ATG(-) was inferior to that of allo-HCT from MUD even in the recent cohort. However, the negative impact of 1MMUD disappeared with the use of low-dose ATG without increasing the risk of relapse.

Neural Circuitry of Reward Prediction Error

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

Slow motion

Optogenetic stimulation of serotonin neurons in the dorsal raphe causes mice to move more slowly without causing any apparent motor deficits or anxiety-like effects.

Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice

Dopamine neurons are thought to encode novelty in addition to reward prediction error (the discrepancy between actual and predicted values). In this study, we compared dopamine activity across the striatum using fiber fluorometry in mice. During classical conditioning, we observed opposite dynamics in dopamine axon signals in the ventral striatum (‘VS dopamine’) and the posterior tail of the striatum (‘TS dopamine’). TS dopamine showed strong excitation to novel cues, whereas VS dopamine showed no responses to novel cues until they had been paired with a reward. TS dopamine cue responses decreased over time, depending on what the cue predicted. Additionally, TS dopamine showed excitation to several types of stimuli including rewarding, aversive, and neutral stimuli whereas VS dopamine showed excitation only to reward or reward-predicting cues. Together, these results demonstrate that dopamine novelty signals are localized in TS along with general salience signals, while VS dopamine reliably encodes reward prediction error.

DOI:http://dx.doi.org/10.7554/eLife.21886.001

New experiences trigger a variety of responses in animals. We are surprised by, move towards, and often explore new objects. But how does the brain control these responses?

Dopamine is a molecule that controls many processes in the brain and plays critical roles in various mental disorders, diseases that affect movement, and addiction. Rewarding experiences (like a glass of cold water on a hot day) can trigger dopamine neurons and studies have also shown that dopamine neurons respond to new experiences. This suggested that novelty may be rewarding in itself, or that novelty may signal the potential for future reward. On the other hand, it may be that different groups of dopamine neurons play different roles in responding to new or rewarding experiences.

In 2015, it was reported that dopamine neurons connected to the rear part of an area in the brain called the striatum receive signals from different parts of the brain than most other dopamine neurons. The dopamine neurons connected to this “tail” of the striatum preferentially received inputs from regions involved in arousal rather than reward, suggesting that they may have a unique role and transmit a different type of information.

Now, Menegas et al. have shown that dopamine signals in different areas of the striatum separate reward from novelty and other signals in mice. The results demonstrate that new odors activate dopamine neurons projecting to the tail of the striatum, but that this activity fades as the novelty wears off (as the mice learn to associate the odor with a particular outcome). By contrast, dopamine neurons projecting to the front of the striatum do not respond to novelty, but rather become more active as mice learn which odors accompany rewards (only responding to odors that predict reward). The experiments also show that dopamine neurons in the tail of the striatum encode information about the importance of a stimulus.

Together, these findings indicate that some of the roles dopamine plays in the brain may not be related to reward, but are instead linked to the novelty and importance of the stimulus. The next challenge will be to find out how the separate reward and novelty signals in dopamine neurons relate to the animals’ behavior. This may help us to better understand dopamine-related psychiatric conditions, such as depression and addiction.

DOI:http://dx.doi.org/10.7554/eLife.21886.002

Low-dose thymoglobulin as second-line treatment for steroid-resistant acute GvHD: an analysis of the JSHCT

A nationwide retrospective study for the clinical outcomes of 99 patients who had received thymoglobulin at a median total dose of 2.5 mg/kg (range, 0.5-18.5 mg/kg) as a second-line treatment for steroid-resistant acute GvHD was conducted. Of the 92 evaluable patients, improvement (complete or partial response) was observed in 55 patients (60%). Multivariate analysis demonstrated that male sex and grade III and IV acute GvHD were associated with a lower improvement rate, whereas thymoglobulin dose (<2.0, 2.0-3.9 and ⩾4.0 mg/kg) was NS. Factors associated with significantly higher nonrelapse mortality included higher patient age (⩾50 years), grade IV acute GvHD, no improvement of GvHD and higher dose of thymoglobulin (hazard ratio, 2.55; 95% confidence interval, 1.34-4.85; P=0.004 for 2.0-3.9 mg/kg group and 1.79; 0.91-3.55; P=0.093 for ⩾4.0 mg/kg group). Higher dose of thymoglobulin was associated with a higher incidence of bacterial infections, CMV antigenemia and any additional infection. Taken together, low-dose thymoglobulin at a median total dose of 2.5 mg/kg provides a comparable response rate to standard-dose thymoglobulin reported previously, and <2.0 mg/kg thymoglobulin is recommended in terms of the balance between efficacy and adverse effects.