Dynamic organization of cerebellar climbing fiber response and synchrony in multiple functional components reduces dimensions for reinforcement learning

Predictive reward-prediction errors of climbing fiber inputs integrate modular reinforcement learning with supervised learning

Although the cerebellum is typically associated with supervised learning algorithms, it also exhibits extensive involvement in reward processing. In this study, we investigated the cerebellum's role in executing reinforcement learning algorithms, with a particular emphasis on essential reward-prediction errors. We employed the Q-learning model to accurately reproduce the licking responses of mice in a Go/No-go auditory-discrimination task. This method enabled the calculation of reinforcement learning variables, such as reward, predicted reward, and reward-prediction errors in each learning trial. Through tensor component analysis of two-photon Ca2+ imaging data from more than 6,000 Purkinje cells, we found that climbing fiber inputs of the two distinct components, which were specifically activated during Go and No-go cues in the learning process, showed an inverse relationship with predictive reward-prediction errors. Assuming bidirectional parallel-fiber Purkinje-cell synaptic plasticity, we constructed a cerebellar neural-network model with 5,000 spiking neurons of granule cells, Purkinje cells, cerebellar nuclei neurons, and inferior olive neurons. The network model qualitatively reproduced distinct changes in licking behaviors, climbing-fiber firing rates, and their synchronization during discrimination learning separately for Go/No-go conditions. We found that Purkinje cells in the two components could develop specific motor commands for their respective auditory cues, guided by the predictive reward-prediction errors from their climbing fiber inputs. These results indicate a possible role of context-specific actors in modular reinforcement learning, integrating with cerebellar supervised learning capabilities.

Cerebellar control of targeted tongue movements

The cerebellum is critical for coordinating movements related to eating, drinking and swallowing, all of which require proper control of the tongue. Cerebellar Purkinje cells can encode tongue movements, but it is unclear how their simple spikes and complex spikes induce changes in the shape of the tongue that contribute to goal-directed movements. To study these relations, we recorded and stimulated Purkinje cells in the vermis and hemispheres of mice during spontaneous licking from a stationary or moving water spout. We found that Purkinje cells can encode rhythmic licking with both their simple spikes and complex spikes. Increased simple spike firing during tongue protrusion induces ipsiversive bending of the tongue. Unexpected changes in the target location trigger complex spikes that alter simple spike firing during subsequent licks, adjusting the tongue trajectory. Furthermore, we observed increased complex spike firing during behavioural state changes at both the start and the end of licking bouts. Using machine learning, we confirmed that alterations in Purkinje cell activity accompany licking, with different Purkinje cells often exerting heterogeneous encoding schemes. Our data highlight that directional movement control is paramount in cerebellar function and that modulation of the complex spikes and that of the simple spikes are complementary during acquisition and execution of sensorimotor coordination. These results bring us closer to understanding the clinical implications of cerebellar disorders during eating, drinking and swallowing. KEY POINTS: When drinking, mice make rhythmic tongue movements directed towards the water source. Cerebellar Purkinje cells can fire rhythmically in tune with the tongue movements. Purkinje cells encode changes in the position of the water source with complex spikes. Purkinje cell simple spike firing affects the direction of tongue movements. Purkinje cells that report changes in the position of the target can also adjust movements in the right direction.

The cognitive reality monitoring network and theories of consciousness

Theories of consciousness abound. However, it is difficult to arbitrate reliably among competing theories because they target different levels of neural and cognitive processing or anatomical loci, and only some were developed with computational models in mind. In particular, theories of consciousness need to fully address the three levels of understanding of the brain proposed by David Marr: computational theory, algorithms and hardware. Most major theories refer to only one or two levels, often indirectly. The cognitive reality monitoring network (CRMN) model is derived from computational theories of mixture-of-experts architecture, hierarchical reinforcement learning and generative/inference computing modules, addressing all three levels of understanding. A central feature of the CRMN is the mapping of a gating network onto the prefrontal cortex, making it a prime coding circuit involved in monitoring the accuracy of one's mental states and distinguishing them from external reality. Because the CRMN builds on the hierarchical and layer structure of the cerebral cortex, it may connect research and findings across species, further enabling concrete computational models of consciousness with new, explicitly testable hypotheses. In sum, we discuss how the CRMN model can help further our understanding of the nature and function of consciousness.