The "proactive" model of learning: Integrative framework for model-free and model-based reinforcement learning utilizing the associative learning-based proactive brain concept

Irisin/BDNF signaling in the muscle-brain axis and circadian system: A review

In mammals, the timing of physiological, biochemical and behavioral processes over a 24-h period is controlled by circadian rhythms. To entrain the master clock located in the suprachiasmatic nucleus of the hypothalamus to a precise 24-h rhythm, environmental zeitgebers are used by the circadian system. This is done primarily by signals from the retina via the retinohypothalamic tract, but other cues like exercise, feeding, temperature, anxiety, and social events have also been shown to act as non-photic zeitgebers. The recently identified myokine irisin is proposed to serve as an entraining non-photic signal of exercise. Irisin is a product of cleavage and modification from its precursor membrane fibronectin type Ⅲ domain-containing protein 5 (FNDC5) in response to exercise. Apart from well-known peripheral effects, such as inducing the "browning" of white adipocytes, irisin can penetrate the blood-brain barrier and display the effects on the brain. Experimental data suggest that FNDC5/irisin mediates the positive effects of physical activity on brain functions. In several brain areas, irisin induces the production of brain-derived neurotrophic factor (BDNF). In the master clock, a significant role in gating photic stimuli in the retinohypothalamic synapse for BDNF is suggested. However, the brain receptor for irisin remains unknown. In the current review, the interactions of physical activity and the irisin/BDNF axis with the circadian system are reconceptualized.

Mechanism of CNS regulation by irisin, a multifunctional protein

Exercise not only builds up our body but also improves cognitive function. Skeletal muscle secretes myokine during exercise as a large reservoir of signaling molecules, which can be considered as a medium between exercise and brain health. Irisin is a circulating myokine derived from the Fibronectin type III domain-containing protein 5 (FNDC5). Irisin regulates energy metabolism because it can stimulate the "Browning" of white adipose tissue. It has been reported that irisin can cross the blood-brain barrier and increase the expression of a brain-derived neurotrophic factor (BDNF) in the hippocampus, which improves learning and memory. In addition, the neuroprotective effect of irisin has been verified in various disease models. Therefore, this review summarizes how irisin plays a neuroprotective role, including its signal pathway and mechanism. In addition, we will briefly discuss the therapeutic potential of irisin for neurological diseases.

The Changes of Irisin and Inflammatory Cytokines in the Age-Related Macular Degeneration and Retinal Vein Occlusion

PURPOSE

Age-related macular degeneration (AMD) and retinal vein occlusion (RVO) are irreversible chorioretinal diseases, which might induce severe damage in visual function. The metabolic factor and inflammatory factors might play important roles in the pathogenesis of AMD and RVO. The levels of irisin and 14 cytokines were analyzed in aqueous humor of AMD and RVO eyes to evaluate the roles of irisin and inflammatory factors.

METHODS

We collected aqueous humor samples from patients with AMD (n = 27), RVO (n = 30), and cataract (as control, n = 23) eyes. Samples were assayed using ELISA kit for irisin and a multiplex immunoassay kit for 14 cytokines. The macular thickness (MT) was measured with OCT in all included eyes.

RESULTS

MT in the RVO group is significantly higher than that in the AMD or control group. Irisin levels in the aqueous samples of AMD and RVO eyes were both significantly lower than that in the control. Furthermore, a positive correlation was found between irisin and MT in the RVO. Compared with the controls, AMD eyes had significantly higher levels of BDNF, VEGF-A, VEGF-R1, VEGF-R2, IL-10, TNF-α, VCAM-1, IP-10, and MCP-1. Similarly, RVO eyes had significantly higher levels of BDNF, VEGF-A, VEGF-R1, VEGF-R2, IL-6, IL-8, IL-10, TNF-α, ICAM-1, VCAM-1, IP-10, and MCP-1. However, there was no significant difference between the levels of PDGF-BB or TNF-β in these three groups. A negative correlation was found between VEGF-A and MT in AMD, as well as in control. Furthermore, a positive correlation was found between IL-6 and MT in the 80 included eyes, as well as in RVO. A positive correlation was found between ICAM-1 and MT in the 80 included eyes, as well as in RVO.

CONCLUSIONS

The metabolic factor, irisin levels in the aqueous humor are decreased in AMD and RVO eyes and show a positive correlation between irisin and MT in RVO eyes, prompting researchers to explore the relationship between irisin and macular edema. We also identified the higher expression of vascular growth factors (VEGF-A, VEGF-R1, and PDGF-BB), inflammatory cytokines (IL-6, IL-8, IL-10, and TNF-α), and chemokines (ICAM-1, VCAM-1, IP-10, and MCP-1) in AMD and RVO eyes.

Irisin Acts via the PGC-1α and BDNF Pathway to Improve Depression-like Behavior

Depression is the most prevalent psychiatric disorder experienced by the world's population. Mechanisms associated with depression-like behavior have not been fully investigated. Among the therapeutic solution for depression, exercise is considered an important regulator attenuating depressive neuropathology. Exercise has been reported to boost the secretion of myokines such as irisin and myostatin in skeletal muscles. Myokines secreted during exercise are involved in various cellular responses including the endocrine and autocrine systems. Especially, irisin as a cleaved version of fibronectin domain-containing protein 5 has multiple functions such as white fat-browning, energy expenditure increase, anti-inflammatory effects, and mitochondrial function improvement in both systemic circulation and central nervous system. Furthermore, irisin activates energy metabolism-related signaling peroxisome proliferator-activated receptor-gamma coactivator-1 alpha and memory formation-related signaling brain-derived neurotrophic factor involved in depression. However, the role and mechanism of irisin in depression disorder is not obvious until now. Here, we review recent evidences regarding the therapeutic effect of irisin in depression disorder. We suggest that irisin is a key molecule that suppresses several neuropathological mechanisms involved in depression.

Vestibular Stimulation May Drive Multisensory Processing: Principles for Targeted Sensorimotor Therapy (TSMT)

At birth, the vestibular system is fully mature, whilst higher order sensory processing is yet to develop in the full-term neonate. The current paper lays out a theoretical framework to account for the role vestibular stimulation may have driving multisensory and sensorimotor integration. Accordingly, vestibular stimulation, by activating the parieto-insular vestibular cortex, and/or the posterior parietal cortex may provide the cortical input for multisensory neurons in the superior colliculus that is needed for multisensory processing. Furthermore, we propose that motor development, by inducing change of reference frames, may shape the receptive field of multisensory neurons. This, by leading to lack of spatial contingency between formally contingent stimuli, may cause degradation of prior motor responses. Additionally, we offer a testable hypothesis explaining the beneficial effect of sensory integration therapies regarding attentional processes. Key concepts of a sensorimotor integration therapy (e.g., targeted sensorimotor therapy (TSMT)) are also put into a neurological context. TSMT utilizes specific tools and instruments. It is administered in 8-weeks long successive treatment regimens, each gradually increasing vestibular and postural stimulation, so sensory-motor integration is facilitated, and muscle strength is increased. Empirically TSMT is indicated for various diseases. Theoretical foundations of this sensorimotor therapy are discussed.

The Role of Irisin in Alzheimer's Disease

Alzheimer’s disease (AD) is characterized by progressive memory dysfunction, oxidative stress, and presence of senile plaques formed by amyloid beta (Aβ) accumulation in the brain. AD is one of the most important causes of morbidity and mortality worldwide. AD has a variety of risk factors, including environmental factors, metabolic dysfunction, and genetic background. Recent research has highlighted the relationship between AD and systemic metabolic changes such as glucose and lipid imbalance and insulin resistance. Irisin, a myokine closely linked to exercise, has been associated with glucose metabolism, insulin sensitivity, and fat browning. Recent studies have suggested that irisin is involved in the process in central nervous system (CNS) such as neurogenesis and has reported the effects of irisin on AD as one of the neurodegenerative disease. Here, we review the roles of irisin with respect to AD and suggest that irisin highlight therapeutic important roles in AD. Thus, we propose that irisin could be a potential future target for ameliorating AD pathology and preventing AD onset.

Blind Spot for Sedentarism: Redefining the Diseasome of Physical Inactivity in View of Circadian System and the Irisin/BDNF Axis

Introduction: The term "diseasome of physical inactivity" was coined by Pedersen to explain clustering of chronic diseases linked to physical inactivity. Accordingly, physical inactivity per se contributes to the accumulation of visceral fat, which, generates chronic low-grade systemic inflammation, contributes to emergence of chronic, non-communicable diseases. Diversity of these disorders posits the possible involvement of a supraphysiological system. Methods: Hypothesis driven literature search and deductive reasoning was used to review relevant literature and formulate a novel theory. Results: We have identified the circadian system, omnipresent in virtually every cell, as a possible vehicle for brain muscle crosstalk, explaining some aspects of the diseasome of physical inactivity This system is hierarchically organized, with the suprachiasmatic nucleus (SCN) being the master clock that entrains to the dark/light cycle and synchronizes subsidiary molecular clocks in the periphery. Insufficient photic entrainment also causes chronic disease evolution. The recently identified irisin, was shown to induce brain-derived neurotrophic factor (BDNF) production in several brain areas. BDNF assumes significant role in gating light's influence in the retinohypothalamic synapse, by having a permissive effect on glutamate signal transduction underlying photic entrainment. Conclusions: Here we provide theoretical evidence to support the hypothesis that irisin may facilitate photic entrainment of the SCN, via BDNF. By this irisin opens up possible pathways for peripheral non-photic entrainment signals to exert influence on the master clock that is otherwise resistant to these. Furthermore, we suggest that intertwining processes of circadian, redox, inflammatory, and myokine systems lay underneath the diseasome of physical inactivity.

The Alteration of Irisin-Brain-Derived Neurotrophic Factor Axis Parallels Severity of Distress Disorder in Bronchial Asthma Patients

Distress disorder (a collective term for generalized anxiety disorder and major depressive disorder) is a well-known co-morbidity of bronchial asthma. The irisin-brain-derived neurotrophic factor (BDNF) axis is a pathway that influences several neurobehavioral mechanisms involved in the pathogenesis of distress disorder. Thus, the aim of the present study was to quantify the serum irisin and BDNF concentrations in order to investigate the possible link between the irisin/BDNF axis and distress disorder in an asthma patient cohort. Data of 167 therapy-controlled asthma patients were analyzed. Demographic, anthropometric, and anamnestic data were collected, routine laboratory parameters supplemented with serum irisin and BDNF levels were determined, pulmonary function test was performed using whole-body plethysmography, and quality of life was quantified by means of the St. George's Respiratory Questionnaire (SGRQ). Correlation analysis as well as simple and multiple linear regression were used to assess the relationship between the irisin level and the Impacts score of SGRQ, which latter is indicative of the presence and severity of distress disorder. We have found a significant, positive linear relationship between the Impacts score and the reciprocal of irisin level. This association was stronger in patients whose BDNF level was higher, and it was weaker (and statistically non-significant) in patients whose BDNF level was lower. Our results indicate that higher serum irisin level together with higher serum BDNF level are associated with milder (or no) distress disorder. This finding suggests that alteration of the irisin/BDNF axis influences the presence and severity of distress disorder in asthma patients.

Alteration of the irisin-brain-derived neurotrophic factor axis contributes to disturbance of mood in COPD patients

COPD is accompanied by limited physical activity, worse quality of life, and increased prevalence of depression. A possible link between COPD and depression may be irisin, a myokine, expression of which in the skeletal muscle and brain positively correlates with physical activity. Irisin enhances the synthesis of brain-derived neurotrophic factor (BDNF), a neurotrophin involved in reward-related processes. Thus, we hypothesized that mood disturbances accompanying COPD are reflected by the changes in the irisin–BDNF axis. Case history, routine laboratory parameters, serum irisin and BDNF levels, pulmonary function, and disease-specific quality of life, measured by St George’s Respiratory Questionnaire (SGRQ), were determined in a cohort of COPD patients (n=74). Simple and then multiple linear regression were used to evaluate the data. We found that mood disturbances are associated with lower serum irisin levels (SGRQ’s Impacts score and reciprocal of irisin showed a strong positive association; β: 419.97; 95% confidence interval [CI]: 204.31, 635.63; P<0.001). This association was even stronger among patients in the lower 50% of BDNF levels (β: 434.11; 95% CI: 166.17, 702.05; P=0.002), while it became weaker for patients in the higher 50% of BDNF concentrations (β: 373.49; 95% CI: −74.91, 821.88; P=0.1). These results suggest that irisin exerts beneficial effect on mood in COPD patients, possibly by inducing the expression of BDNF in brain areas associated with reward-related processes involved in by depression. Future interventional studies targeting the irisin–BDNF axis (eg, endurance training) are needed to further support this notion.

'Proactive' use of cue-context congruence for building reinforcement learning's reward function

BACKGROUND

Reinforcement learning is a fundamental form of learning that may be formalized using the Bellman equation. Accordingly an agent determines the state value as the sum of immediate reward and of the discounted value of future states. Thus the value of state is determined by agent related attributes (action set, policy, discount factor) and the agent's knowledge of the environment embodied by the reward function and hidden environmental factors given by the transition probability. The central objective of reinforcement learning is to solve these two functions outside the agent's control either using, or not using a model.

RESULTS

In the present paper, using the proactive model of reinforcement learning we offer insight on how the brain creates simplified representations of the environment, and how these representations are organized to support the identification of relevant stimuli and action. Furthermore, we identify neurobiological correlates of our model by suggesting that the reward and policy functions, attributes of the Bellman equitation, are built by the orbitofrontal cortex (OFC) and the anterior cingulate cortex (ACC), respectively.

CONCLUSIONS

Based on this we propose that the OFC assesses cue-context congruence to activate the most context frame. Furthermore given the bidirectional neuroanatomical link between the OFC and model-free structures, we suggest that model-based input is incorporated into the reward prediction error (RPE) signal, and conversely RPE signal may be used to update the reward-related information of context frames and the policy underlying action selection in the OFC and ACC, respectively. Furthermore clinical implications for cognitive behavioral interventions are discussed.

FNDC5/irisin, a molecular target for boosting reward-related learning and motivation

Interventions focusing on the prevention and treatment of chronic non-communicable diseases are on rise. In the current article, we propose that dysfunction of the mesocortico-limbic reward system contributes to the emergence of the WHO-identified risk behaviors (tobacco use, unhealthy diet, physical inactivity and harmful use of alcohol), behaviors that underlie the evolution of major non-communicable diseases (e.g. cardiovascular diseases, cancer, diabetes and chronic respiratory diseases). Given that dopaminergic neurons of the mesocortico-limbic system are tightly associated with reward-related processes and motivation, their dysfunction may fundamentally influence behavior. While nicotine and alcohol alter dopamine neuron function by influencing some receptors, mesocortico-limbic system dysfunction was associated with elevation of metabolic set-point leading to hedonic over-eating. Although there is some empirical evidence, precise molecular mechanism for linking physical inactivity and mesocortico-limbic dysfunction per se seems to be missing; identification of which may contribute to higher success rates for interventions targeting lifestyle changes pertaining to physical activity. In the current article, we compile evidence in support of a link between exercise and the mesocortico-limbic system by elucidating interactions on the axis of muscle - irisin - brain derived neurotrophic factor (BDNF) - and dopaminergic function of the midbrain. Irisin is a contraction-regulated myokine formed primarily in skeletal muscle but also in the brain. Irisin stirred considerable interest, when its ability to induce browning of white adipose tissue parallel to increasing thermogenesis was discovered. Furthermore, it may also play a role in the regulation of behavior given it readily enters the central nervous system, where it induces BDNF expression in several brain areas linked to reward processing, e.g. the ventral tegmental area and the hippocampus. BDNF is a neurotropic factor that increases neuronal dopamine content, modulates dopamine release relevant for neuronal plasticity and increased neuronal survival as well as learning and memory. Further linking BDNF to dopaminergic function is BDNF's ability to activate tropomyosin-related kinase B receptor that shares signalization with presynaptic dopamine-3 receptors in the ventral tegmental area. Summarizing, we propose that the skeletal muscle derived irisin may be the link between physical activity and reward-related processes and motivation. Moreover alteration of this axis may contribute to sedentary lifestyle and subsequent non-communicable diseases. Preclinical and clinical experimental models to test this hypothesis are also proposed.