The cerebellum: a neural system for the study of reinforcement learning

Neuroanatomical predictors of real‐time fMRI ‐based anterior insula regulation. A supervised machine learning study

Increasing evidence showed that learned control of metabolic activity in selected brain regions can support emotion regulation. Notably, a number of studies demonstrated that neurofeedback-based regulation of fMRI activity in several emotion-related areas leads to modifications of emotional behavior along with changes of neural activity in local and distributed networks, in both healthy individuals and individuals with emotional disorders. However, the current understanding of the neural mechanisms underlying self-regulation of the emotional brain, as well as their relationship with other emotion regulation strategies, is still limited. In this study, we attempted to delineate neuroanatomical regions mediating real-time fMRI-based emotion regulation by exploring whole brain GM and WM features predictive of self-regulation of anterior insula (AI) activity, a neuromodulation procedure that can successfully support emotional brain regulation in healthy individuals and patients. To this aim, we employed a multivariate kernel ridge regression model to assess brain volumetric features, at regional and network level, predictive of real-time fMRI-based AI regulation. Our results showed that several GM regions including fronto-occipital and medial temporal areas and the basal ganglia as well as WM regions including the fronto-occipital fasciculus, tapetum and fornix significantly predicted learned AI regulation. Remarkably, we observed a substantial contribution of the cerebellum in relation to both the most effective regulation run and average neurofeedback performance. Overall, our findings highlighted specific neurostructural features contributing to individual differences of AI-guided emotion regulation. Notably, such neuroanatomical topography partially overlaps with the neurofunctional network associated with cognitive emotion regulation strategies, suggesting common neural mechanisms.

To explore the mechanism of tobacco addiction using structural and functional MRI: a preliminary study of the role of the cerebellum-striatum circuit

Previous studies have found that the striatum and the cerebellum played important roles in nicotine dependence, respectively. In heavy smokers, however, the effect of resting-state functional connectivity of cerebellum-striatum circuits in nicotine dependence remained unknown. This study aimed to explore the role of the circuit between the striatum and the cerebellum in addiction in heavy smokers using structural and functional magnetic resonance imaging. The grey matter volume differences and the resting-state functional connectivity differences in cerebellum-striatum circuits were investigated between 23 heavy smokers and 23 healthy controls. The cigarette dependence in heavy smokers and healthy controls were evaluated by using Fagerström Test. Then, we applied mediation analysis to test whether the resting-state functional connectivity between the striatum and the cerebellum mediates the relationship between the striatum morphometry and the nicotine dependence in heavy smokers. Compared with healthy controls, the heavy smokers' grey matter volumes decreased significantly in the cerebrum (bilateral), and increased significantly in the caudate (bilateral). Seed-based resting-state functional connectivity analysis showed significantly higher resting-state functional connectivity among the bilateral caudate, the left cerebellum, and the right middle temporal gyrus in heavy smokers. The cerebellum-striatum resting-state functional connectivity fully mediated the relationship between the striatum morphometry and the nicotine dependence in heavy smokers. Heavy smokers showed abnormal interactions and functional connectivity between the striatum and the cerebellum, which were associated with the striatum morphometry and nicotine dependence. Such findings could provide new insights into the neural correlates of nicotine dependence in heavy smokers.

Visuomotor control in mice and primates

We conduct a comparative evaluation of the visual systems from the retina to the muscles of the mouse and the macaque monkey noting the differences and similarities between these two species. The topics covered include (1) visual-field overlap, (2) visual spatial resolution, (3) V1 cortical point-image [i.e., V1 tissue dedicated to analyzing a unit receptive field], (4) object versus motion encoding, (5) oculomotor range, (6) eye, head, and body movement coordination, and (7) neocortical and cerebellar function. We also discuss blindsight in rodents and primates which provides insights on how the neocortex mediates conscious vision in these species. This review is timely because the field of visuomotor neurophysiology is expanding beyond the macaque monkey to include the mouse; there is therefore a need for a comparative analysis between these two species on how the brain generates visuomotor responses.

The Implementation of Predictions During Sequencing

Optimal control mechanisms require prediction capabilities. If one cannot predict the consequences of a motor act or behavior, one will continually collide with walls or become a social pariah. "Looking into the future" is thus one of the most important prerequisites for smooth movements and social interactions. To achieve this goal, the brain must constantly predict future events. This principle applies to all domains of information processing, including motor and cognitive control, as well as the development of decision-making skills, theory of mind, and virtually all cognitive processes. Sequencing is suggested to support the predictive capacity of the brain. To recognize that events are related, the brain must discover links among them in the spatiotemporal domain. To achieve this, the brain must often hold one event in working memory and compare it to a second one, and the characteristics of the two must be compared and correctly placed in space and time. Among the different brain structures involved in sequencing, the cerebellum has been proposed to have a central function. We have suggested that the operational mode of the cerebellum is based on "sequence detection" and that this process is crucial for prediction. Patterns of temporally or spatially structured events are conveyed to the cerebellum via the pontine nuclei and compared with actual ones conveyed through the climbing fibers olivary inputs. Through this interaction, data on previously encountered sequences can be obtained and used to generate internal models from which predictions can be made. This mechanism would allow the cerebellum not only to recognize sequences but also to detect sequence violations. Cerebellar pattern detection and prediction would thus be a means to allow feedforward control based on anticipation. We will argue that cerebellar sequencing allows implementation of prediction by setting the correct excitatory levels in defined brain areas to implement the adaptive response for a given pattern of stimuli that embeds sufficient information to be recognized as a previously encountered template. Here, we will discuss results from human and animal studies and correlate them with the present understanding of cerebellar function in cognition and behavior.

Discriminating schizophrenia using recurrent neural network applied on time courses of multi-site FMRI data

BACKGROUND

Current fMRI-based classification approaches mostly use functional connectivity or spatial maps as input, instead of exploring the dynamic time courses directly, which does not leverage the full temporal information.

METHODS

Motivated by the ability of recurrent neural networks (RNN) in capturing dynamic information of time sequences, we propose a multi-scale RNN model, which enables classification between 558 schizophrenia and 542 healthy controls by using time courses of fMRI independent components (ICs) directly. To increase interpretability, we also propose a leave-one-IC-out looping strategy for estimating the top contributing ICs.

FINDINGS

Accuracies of 83·2% and 80·2% were obtained respectively for the multi-site pooling and leave-one-site-out transfer classification. Subsequently, dorsal striatum and cerebellum components contribute the top two group-discriminative time courses, which is true even when adopting different brain atlases to extract time series.

INTERPRETATION

This is the first attempt to apply a multi-scale RNN model directly on fMRI time courses for classification of mental disorders, and shows the potential for multi-scale RNN-based neuroimaging classifications. FUND: Natural Science Foundation of China, the Strategic Priority Research Program of the Chinese Academy of Sciences, National Institutes of Health Grants, National Science Foundation.

Sex-Specific Relationship of Childhood Adversity With Gray Matter Volume and Temperament

Background: To date, many studies have attempted to show a relationship between potentially harmful experiences in childhood and gray matter volume (GMV) in specific brain areas. These studies managed to identify several affected regions, yet most of them neglected the influence of sex or the occurrence of mental health problems. Furthermore, little is known about mechanisms linking childhood adversity (CA) and temperamental traits as plausible endophenotypes of psychopathology. Objective: The present study addresses these two issues by trying to identify sex-specific relationships between CA and brain volumes as well as to show the role of the latter in predicting temperament scores. Method: Forty-eight people (23 women) without anxiety or affective disorders participated in this study. CA was measured using the Childhood Questionnaire (CQ) and temperament was measured with the use of the behavioral inhibition system-behavioral activation system (BIS-BAS) Scales. Whole-brain MR imaging was performed to identify GMV differences. Results: In women, we identified negative relationships between CA and GMV in the left inferior parietal lobule (IPL), right cerebellum, and right precentral gyrus. In men, we found a negative correlation between CA and GMV in the right fusiform gyrus. We also identified sex-specific relationships between CA and temperament traits. Conclusions: The results of our study suggest a sex-specific pattern in the relationship between early adverse experiences and brain structure. The results can also help explain the role that temperament plays in the relationship between CA and the risk of psychopathology.

Revisiting a theory of cerebellar cortex

Long-term depression at parallel fiber-Purkinje cell synapses plays a principal role in learning in the cerebellum, which acts as a supervised learning machine. Recent experiments demonstrate various forms of synaptic plasticity at different sites within the cerebellum. In this article, we take into consideration synaptic plasticity at parallel fiber-molecular layer interneuron synapses as well as at parallel fiber-Purkinje cell synapses, and propose that the cerebellar cortex performs reinforcement learning, another form of learning that is more capable than supervised learning. We posit that through the use of reinforcement learning, the need for explicit teacher signals for learning in the cerebellum is eliminated; instead, learning can occur via responses from evaluative feedback. We demonstrate the learning capacity of cerebellar reinforcement learning using simple computer simulations of delay eyeblink conditioning and the cart-pole balancing task.

Cognitive and psychopathology correlates of brain white/grey matter structure in severely psychotic schizophrenic inpatients

The brain structural correlates of cognitive and psychopathological symptoms within the active phase in severely psychotic schizophrenic inpatients have been rarely investigated. Twenty-eight inpatients with a DSM-5 diagnosis of Schizophrenia (SZ), admitted for acute psychotic decompensation, were assessed through a comprehensive neuropsychological and psychopathological battery. All patients underwent a high-resolution T1-weighted magnetic resonance imaging investigation. Increased psychotic severity was related to reduced grey matter volumes in the medial portion of the right superior frontal cortex, the superior orbitofrontal cortex bilaterally and to white matter volume reduction in the medial portion of the left superior frontal area. Immediate verbal memory performance was related to left insula and inferior parietal cortex volume, while long-term visuo-spatial memory was related to grey matter volume of the right middle temporal cortex, and the right (lobule VII, CRUS1) and left (lobule VI) cerebellum. Moreover, psychotic severity correlated with cognitive inflexibility and negative symptom severity was related to visuo-spatial processing and reasoning disturbances. These findings indicate that a disruption of the cortical-subcortical-cerebellar circuit, and distorted memory function contribute to the development and maintenance of psychotic exacerbation.

Cognition and resting-state functional connectivity in schizophrenia

Individuals with schizophrenia consistently display deficits in a multitude of cognitive domains, but the neurobiological source of these cognitive impairments remains unclear. By analyzing the functional connectivity of resting-state functional magnetic resonance imaging (rs-fcMRI) data in clinical populations like schizophrenia, research groups have begun elucidating abnormalities in the intrinsic communication between specific brain regions, and assessing relationships between these abnormalities and cognitive performance in schizophrenia. Here we review studies that have reported analysis of these brain-behavior relationships. Through this systematic review we found that patients with schizophrenia display abnormalities within and between regions comprising (1) the cortico-cerebellar-striatal-thalamic loop and (2) task-positive and task-negative cortical networks. Importantly, we did not observe unique relationships between specific functional connectivity abnormalities and distinct cognitive domains, suggesting that the observed functional systems may underlie mechanisms that are shared across cognitive abilities, the disturbance of which could contribute to the "generalized" cognitive deficit found in schizophrenia. We also note several areas of methodological change that we believe will strengthen this literature.

Avoidance prone individuals self reporting behavioral inhibition exhibit facilitated acquisition and altered extinction of conditioned eyeblinks with partial reinforcement schedules

Avoidance in the face of novel situations or uncertainty is a prime feature of behavioral inhibition which has been put forth as a risk factor for the development of anxiety disorders. Recent work has found that behaviorally inhibited (BI) individuals acquire conditioned eyeblinks faster than non-inhibited (NI) individuals in omission and yoked paradigms in which the predictive relationship between the conditioned stimulus (CS) and unconditional stimulus (US) is less than optimal as compared to standard training with CS-US paired trials (Holloway et al., 2014). In the current study, we tested explicitly partial schedules in which half the trials were CS alone or US alone trials in addition to the standard CS-US paired trials. One hundred and forty nine college-aged undergraduates participated in the study. All participants completed the Adult Measure of Behavioral Inhibition (i.e., AMBI) which was used to group participants as BI and NI. Eyeblink conditioning consisted of three US alone trials, 60 acquisition trials, and 20 CS-alone extinction trials presented in one session. Conditioning stimuli were a 500 ms tone CS and a 50-ms air puff US. Behaviorally inhibited individuals receiving 50% partial reinforcement with CS alone or US alone trials produced facilitated acquisition as compared to NI individuals. A partial reinforcement extinction effect (PREE) was evident with CS alone trials in BI but not NI individuals. These current findings indicate that avoidance prone individuals self-reporting behavioral inhibition over-learn an association and are slow to extinguish conditioned responses (CRs) when there is some level of uncertainty between paired trials and CS or US alone presentations.

Fast convergence of learning requires plasticity between inferior olive and deep cerebellar nuclei in a manipulation task: a closed-loop robotic simulation

The cerebellum is known to play a critical role in learning relevant patterns of activity for adaptive motor control, but the underlying network mechanisms are only partly understood. The classical long-term synaptic plasticity between parallel fibers (PFs) and Purkinje cells (PCs), which is driven by the inferior olive (IO), can only account for limited aspects of learning. Recently, the role of additional forms of plasticity in the granular layer, molecular layer and deep cerebellar nuclei (DCN) has been considered. In particular, learning at DCN synapses allows for generalization, but convergence to a stable state requires hundreds of repetitions. In this paper we have explored the putative role of the IO-DCN connection by endowing it with adaptable weights and exploring its implications in a closed-loop robotic manipulation task. Our results show that IO-DCN plasticity accelerates convergence of learning by up to two orders of magnitude without conflicting with the generalization properties conferred by DCN plasticity. Thus, this model suggests that multiple distributed learning mechanisms provide a key for explaining the complex properties of procedural learning and open up new experimental questions for synaptic plasticity in the cerebellar network.

Dopaminergic modulation of resting-state functional connectivity in de novo patients with Parkinson's disease

Parkinson's disease (PD) is characterized by degenerative changes of nigral dopamine neurons, resulting in the dopaminergic denervation of the striatum. Resting state networks studies have demonstrated that dopamine modulates distinct network connectivity patterns in both a linear and a nonlinear fashion, but quantitative analyses of dopamine-dependent functional connectivity secondary to PD pathology were less informative. In the present study, we performed a correlation analysis between striatal dopamine levels assessed quantitatively by FP-CIT positron emission tomography imaging and resting-state functional connectivity in 23 drug naïve de novo patients with PD to elucidate dopamine-dependent functional networks. The major finding is that the patterns of dopamine-dependent positive functional connectivity varied depending on the location of striatal seeds. Dopamine-dependent functional connectivity with the caudate predominantly overlay pericentral cortical areas, whereas dopamine-dependent structures functionally connected with the posterior putamen predominantly involved cerebellar areas. The dorsolateral frontal area overlapped as a dopamine-dependent cortical region that was positively connected with the anterior and posterior putamen. On the other hand, cortical areas where functional connectivity from the posterior cingulate was negatively correlated with dopaminergic status in the posterior putamen were localized in the left anterior prefrontal area and the parietal area. Additionally, functional connectivity between the anterior putamen and mesiofrontal areas was negatively coupled with striatal dopamine levels. The present study demonstrated that dopamine-dependent functional network connectivity secondary to PD pathology mainly exhibits a consistent pattern, albeit with some variation. These patterns may reflect the diverse effects of dopaminergic medication on parkinsonian-related motor and cognitive performance.

The cerebellum and addiction: insights gained from neuroimaging research

Although cerebellar alterations have been consistently noted in the addiction literature, the pathophysiology of this link remains unclear. The cerebellum is commonly classified as a motor structure, but human functional neuroimaging along with clinical observations in cerebellar stroke patients and anatomical tract tracing in non-human primates suggests its involvement in cognitive and affective processing. A comprehensive literature search on the role of the cerebellum in addiction was performed. This review article (1) considers the potential role of the cerebellum in addiction; (2) summarizes the cerebellar structural alterations linked to addiction; (3) presents the functional neuroimaging evidence linking the cerebellum with addiction; and (4) proposes a model for addiction that underscores the role of the cerebellum. The data implicate the cerebellum as an intermediary between motor and reward, motivation and cognitive control systems, as all are relevant etiologic factors in addiction. Furthermore, consideration of these findings could contribute to deeper and more sophisticated insights into normal reward and motivational function. The goal of this review is to spread awareness of cerebellar involvement in addictive processes, and to suggest a preliminary model for its potential role.

Distributed cerebellar plasticity implements adaptable gain control in a manipulation task: a closed-loop robotic simulation

Adaptable gain regulation is at the core of the forward controller operation performed by the cerebro-cerebellar loops and it allows the intensity of motor acts to be finely tuned in a predictive manner. In order to learn and store information about body-object dynamics and to generate an internal model of movement, the cerebellum is thought to employ long-term synaptic plasticity. LTD at the PF-PC synapse has classically been assumed to subserve this function (Marr, 1969). However, this plasticity alone cannot account for the broad dynamic ranges and time scales of cerebellar adaptation. We therefore tested the role of plasticity distributed over multiple synaptic sites (Hansel et al., 2001; Gao et al., 2012) by generating an analog cerebellar model embedded into a control loop connected to a robotic simulator. The robot used a three-joint arm and performed repetitive fast manipulations with different masses along an 8-shape trajectory. In accordance with biological evidence, the cerebellum model was endowed with both LTD and LTP at the PF-PC, MF-DCN and PC-DCN synapses. This resulted in a network scheme whose effectiveness was extended considerably compared to one including just PF-PC synaptic plasticity. Indeed, the system including distributed plasticity reliably self-adapted to manipulate different masses and to learn the arm-object dynamics over a time course that included fast learning and consolidation, along the lines of what has been observed in behavioral tests. In particular, PF-PC plasticity operated as a time correlator between the actual input state and the system error, while MF-DCN and PC-DCN plasticity played a key role in generating the gain controller. This model suggests that distributed synaptic plasticity allows generation of the complex learning properties of the cerebellum. The incorporation of further plasticity mechanisms and of spiking signal processing will allow this concept to be extended in a more realistic computational scenario.

Transient dynamics and rhythm coordination of inferior olive spatio-temporal patterns

The inferior olive (IO) is a neural network belonging to the olivo-cerebellar system whose neurons are coupled with electrical synapses and display subthreshold oscillations and spiking activity. The IO is frequently proposed as the generator of timing signals to the cerebellum. Electrophysiological and imaging recordings show that the IO network generates complex spatio-temporal patterns. The generation and modulation of coherent spiking activity in the IO is one key issue in cerebellar research. In this work, we build a large scale IO network model of electrically coupled conductance-based neurons to study the emerging spatio-temporal patterns of its transient neuronal activity. Our modeling reproduces and helps to understand important phenomena observed in IO in vitro and in vivo experiments, and draws new predictions regarding the computational properties of this network and the associated cerebellar circuits. The main factors studied governing the collective dynamics of the IO network were: the degree of electrical coupling, the extent of the electrotonic connections, the presence of stimuli or regions with different excitability levels and the modulatory effect of an inhibitory loop (IL). The spatio-temporal patterns were analyzed using a discrete wavelet transform to provide a quantitative characterization. Our results show that the electrotonic coupling produces quasi-synchronized subthreshold oscillations over a wide dynamical range. The synchronized oscillatory activity plays the role of a timer for a coordinated representation of spiking rhythms with different frequencies. The encoding and coexistence of several coordinated rhythms is related to the different clusterization and coherence of transient spatio-temporal patterns in the network, where the spiking activity is commensurate with the quasi-synchronized subthreshold oscillations. In the presence of stimuli, different rhythms are encoded in the spiking activity of the IO neurons that nevertheless remains constrained to a commensurate value of the subthreshold frequency. The stimuli induced spatio-temporal patterns can reverberate for long periods, which contributes to the computational properties of the IO. We also show that the presence of regions with different excitability levels creates sinks and sources of coordinated activity which shape the propagation of spike wave fronts. These results can be generalized beyond IO studies, as the control of wave pattern propagation is a highly relevant problem in the context of normal and pathological states in neural systems (e.g., related to tremor, migraine, epilepsy) where the study of the modulation of activity sinks and sources can have a potential large impact.

Cerebellar networks with the cerebral cortex and basal ganglia

The dominant view of cerebellar function has been that it is exclusively concerned with motor control and coordination. Recent findings from neuroanatomical, behavioral, and imaging studies have profoundly changed this view. Neuroanatomical studies using virus transneuronal tracers have demonstrated that cerebellar output reaches vast areas of the neocortex, including regions of prefrontal and posterior parietal cortex. Furthermore, it has recently become clear that the cerebellum is reciprocally connected with the basal ganglia, which suggests that the two subcortical structures are part of a densely interconnected network. Taken together, these findings elucidate the neuroanatomical substrate for cerebellar involvement in non-motor functions mediated by the prefrontal and posterior parietal cortex, as well as in processes traditionally associated with the basal ganglia.

Enhanced conditioned eyeblink response acquisition and proactive interference in anxiety vulnerable individuals

In classical conditioning, proactive interference may arise from experience with the conditioned stimulus (CS), the unconditional stimulus (US), or both, prior to their paired presentations. Interest in the application of proactive interference has extended to clinical populations as either a risk factor for disorders or as a secondary sign. Although the current literature is dense with comparisons of stimulus pre-exposure effects in animals, such comparisons are lacking in human subjects. As such, interpretation of proactive interference over studies as well as its generalization and utility in clinical research is limited. The present study was designed to assess eyeblink response acquisition after equal numbers of CS, US, and explicitly unpaired CS and US pre-exposures, as well as to evaluate how anxiety vulnerability might modulate proactive interference. In the current study, anxiety vulnerability was assessed using the State/Trait Anxiety Inventories as well as the adult and retrospective measures of behavioral inhibition (AMBI and RMBI, respectively). Participants were exposed to 1 of 4 possible pre-exposure contingencies: 30 CS, 30 US, 30 CS, and 30 US explicitly unpaired pre-exposures, or Context pre-exposure, immediately prior to standard delay training. Robust proactive interference was evident in all pre-exposure groups relative to Context pre-exposure, independent of anxiety classification, with CR acquisition attenuated at similar rates. In addition, trait anxious individuals were found to have enhanced overall acquisition as well as greater proactive interference relative to non-vulnerable individuals. The findings suggest that anxiety vulnerable individuals learn implicit associations faster, an effect which persists after the introduction of new stimulus contingencies. This effect is not due to enhanced sensitivity to the US. Such differences would have implications for the development of anxiety psychopathology within a learning framework.

The generation of phase differences and frequency changes in a network model of inferior olive subthreshold oscillations

It is commonly accepted that the Inferior Olive (IO) provides a timing signal to the cerebellum. Stable subthreshold oscillations in the IO can facilitate accurate timing by phase-locking spikes to the peaks of the oscillation. Several theoretical models accounting for the synchronized subthreshold oscillations have been proposed, however, two experimental observations remain an enigma. The first is the observation of frequent alterations in the frequency of the oscillations. The second is the observation of constant phase differences between simultaneously recorded neurons. In order to account for these two observations we constructed a canonical network model based on anatomical and physiological data from the IO. The constructed network is characterized by clustering of neurons with similar conductance densities, and by electrical coupling between neurons. Neurons inside a cluster are densely connected with weak strengths, while neurons belonging to different clusters are sparsely connected with stronger connections. We found that this type of network can robustly display stable subthreshold oscillations. The overall frequency of the network changes with the strength of the inter-cluster connections, and phase differences occur between neurons of different clusters. Moreover, the phase differences provide a mechanistic explanation for the experimentally observed propagating waves of activity in the IO. We conclude that the architecture of the network of electrically coupled neurons in combination with modulation of the inter-cluster coupling strengths can account for the experimentally observed frequency changes and the phase differences.

There is a profound interest in the dynamics of neuronal networks and the simulation of network models is a prevalent approach to study these dynamics. Generally, network models contain neurons that are connected mostly through chemical synapses to form either a completely regular topology (such as nearest neighbor connections), a completely random topology, small-world networks or scale-free networks. We investigate the dynamics of an atypical network, inspired by the Inferior Olive (IO) network, a brain structure located at the end of the brainstem that is responsible for timely execution of motor commands. This network is atypical in the sense that it has neurons in a clustered topology, which are connected solely by electrical synapses. The dynamics in the IO are enigmatic as the membrane voltage of some neurons can oscillate at the same frequency while maintaining phase difference with other neurons. It has also been demonstrated that propagating waves of activity occur spontaneously in this network. Using computer simulations we unraveled the mechanism underlying these previously enigmatic experimental observations. In so doing, we stress the importance of investigating more realistic network topologies to explore complex brain dynamics.