Value signals guide abstraction during learning

Type of feedback affects formation of prototype or exemplar representations

Category learning theories have typically focused on how the underlying category structure affects the category representations acquired by learners. However, other factors such as type of feedback may also affect what representations are learned and utilized, but have received little attention. We used a novel "5/5" categorization task developed from the well-studied 5/4 task and held category structure constant while varying type of feedback: verbal correct/incorrect feedback in Experiment 1, and rewarded differential point-valued feedback in Experiment 2. We used behavioral measures and computational modeling to identify whether participants learned to categorize using exemplar or prototype categorization representations. Correct/incorrect feedback resulted in greater use of exemplar representations and better performance by participants who used exemplar representations, whereas in comparison, rewarded point-valued feedback resulted in relatively greater use of prototype representations. These results indicate that differential reward increased abstraction during category learning. The importance of feedback type in guiding what is learned during category learning should be incorporated into future experimental work and theoretical development.

Emergence of Categorical Representations in Parietal and Ventromedial Prefrontal Cortex across Extended Training

How do the neural representations underlying category learning change as skill develops? We examined perceptual category learning using a prototype learning task known to recruit a corticostriatal system including the posterior striatum, motor cortex, visual cortex, and intraparietal sulcus (IPS). Male and female human participants practiced categorizing stimuli as category members or nonmembers (A vs not-A) across 3 d, with fMRI data collected at the beginning and end. Univariate analyses found that corticostriatal activity in regions associated with habitual instrumental learning was recruited across both sessions, but activity in regions associated with goal-directed instrumental learning decreased from Day 1 to Day 3. Multivoxel pattern analysis (MVPA) indicated that after training, the trained category could be more easily decoded from the IPS when compared with a novel category. Representational similarity analysis (RSA) showed development of category representations in the IPS and motor cortex. In addition, RSA revealed evidence for category-related representations including prototype representation in the ventromedial prefrontal cortex which may reflect parallel development of schematic memory for the category structure. Overall, the results converge to show how performance of category decisions and representations of the category structure emerge after extensive training across the corticostriatal system underlying perceptual category learning.

Contributions of Attention to Learning in Multidimensional Reward Environments

Real-world choice options have many features or attributes, whereas the reward outcome from those options only depends on a few features or attributes. It has been shown that humans learn and combine feature-based with more complex conjunction-based learning to tackle challenges of learning in naturalistic reward environments. However, it remains unclear how different learning strategies interact to determine what features or conjunctions should be attended to and control choice behavior, and how subsequent attentional modulations influence future learning and choice. To address these questions, we examined the behavior of male and female human participants during a three-dimensional learning task in which reward outcomes for different stimuli could be predicted based on a combination of an informative feature and conjunction. Using multiple approaches, we found that both choice behavior and reward probabilities estimated by participants were most accurately described by attention-modulated models that learned the predictive values of both the informative feature and the informative conjunction. Specifically, in the reinforcement learning model that best fit choice data, attention was controlled by the difference in the integrated feature and conjunction values. The resulting attention weights modulated learning by increasing the learning rate on attended features and conjunctions. Critically, modulating decision-making by attention weights did not improve the fit of data, providing little evidence for direct attentional effects on choice. These results suggest that in multidimensional environments, humans direct their attention not only to selectively process reward-predictive attributes but also to find parsimonious representations of the reward contingencies for more efficient learning.

Distinct neural bases of visual art- and music-induced aesthetic experiences

Aesthetic experiences are characterized by a conscious, emotionally and hedonically rewarding perceptions of a stimulus's aesthetic qualities and are thought to arise from a unique combination of cognitive and affective processes. To pinpoint neural correlates of aesthetic experiences, in the present study, we performed a series of meta-analyses based on the existing functional Magnetic Resonance Imaging (fMRI) studies of art appreciation in visual art (34 experiments, 692 participants) and music (34 experiments, 718 participants). The Activation Likelihood Estimation (ALE) analyses showed that the frontal pole (FP), ventromedial prefrontal cortex (vmPFC), and inferior frontal gyrus (IFG) were commonly activated in visual-art-induced aesthetic experiences, whilst bilateral superior temporal gyrus (STG) and striatal areas were commonly activated in music appreciation. Additionally, task-independent Resting-state Functional Connectivity (RSFC), task-dependent Meta-analytical Connectivity Modelling (MACM) analyses, as well as Activation Network Modeling (ANM) further showed that visual art and music engaged quite distinct brain networks. Our findings support the domain-specific view of aesthetic appreciation and challenge the notion that there is a general "common neural currency" for aesthetic experiences across domains.

Autoencoder-based phenotyping of ophthalmic images highlights genetic loci influencing retinal morphology and provides informative biomarkers

MOTIVATION

Genome-wide association studies (GWAS) have been remarkably successful in identifying associations between genetic variants and imaging-derived phenotypes. To date, the main focus of these analyses has been on established, clinically-used imaging features. We sought to investigate if deep learning approaches can detect more nuanced patterns of image variability.

RESULTS

We used an autoencoder to represent retinal optical coherence tomography (OCT) images from 31 135 UK Biobank participants. For each subject, we obtained a 64-dimensional vector representing features of retinal structure. GWAS of these autoencoder-derived imaging parameters identified 118 statistically significant loci; 41 of these associations were also significant in a replication study. These loci encompassed variants previously linked with retinal thickness measurements, ophthalmic disorders, and/or neurodegenerative conditions. Notably, the generated retinal phenotypes were found to contribute to predictive models for glaucoma and cardiovascular disorders. Overall, we demonstrate that self-supervised phenotyping of OCT images enhances the discoverability of genetic factors influencing retinal morphology and provides epidemiologically informative biomarkers.

AVAILABILITY AND IMPLEMENTATION

Code and data links available at https://github.com/tf2/autoencoder-oct.

The neuroscience of active learning and direct instruction

Throughout the educational system, students experiencing active learning pedagogy perform better and fail less than those taught through direct instruction. Can this be ascribed to differences in learning from a neuroscientific perspective? This review examines mechanistic, neuroscientific evidence that might explain differences in cognitive engagement contributing to learning outcomes between these instructional approaches. In classrooms, direct instruction comprehensively describes academic content, while active learning provides structured opportunities for learners to explore, apply, and manipulate content. Synaptic plasticity and its modulation by arousal or novelty are central to all learning and both approaches. As a form of social learning, direct instruction relies upon working memory. The reinforcement learning circuit, associated agency, curiosity, and peer-to-peer social interactions combine to enhance motivation, improve retention, and build higher-order-thinking skills in active learning environments. When working memory becomes overwhelmed, additionally engaging the reinforcement learning circuit improves retention, providing an explanation for the benefits of active learning. This analysis provides a mechanistic examination of how emerging neuroscience principles might inform pedagogical choices at all educational levels.

Functional specialization of medial and lateral orbitofrontal cortex in inferential decision-making

Inferring prospective outcomes and updating behavior are prerequisites for making flexible decisions in the changing world. These abilities are highly associated with the functions of the orbitofrontal cortex (OFC) in humans and animals. The functional specialization of OFC subregions in decision-making has been established in animals. However, the understanding of how human OFC contributes to decision-making remains limited. Therefore, we studied this issue by examining the information representation and functional interactions of human OFC subregions during inference-based decision-making. We found that the medial OFC (mOFC) and lateral OFC (lOFC) collectively represented the inferred outcomes which, however, were context-general coding in the mOFC and context-specific in the lOFC. Furthermore, the mOFC-motor and lOFC-frontoparietal functional connectivity may indicate the motor execution of mOFC and the cognitive control of lOFC during behavioral updating. In conclusion, our findings support the dissociable functional roles of OFC subregions in decision-making.

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

From learned value to sustained bias: how reward conditioning changes attentional priority

Introduction

Attentional bias to reward-associated stimuli can occur even when it interferes with goal-driven behavior. One theory posits that dopaminergic signaling in the striatum during reward conditioning leads to changes in visual cortical and parietal representations of the stimulus used, and this, in turn, sustains attentional bias even when reward is discontinued. However, only a few studies have examined neural activity during both rewarded and unrewarded task phases.

Methods

In the current study, participants first completed a reward-conditioning phase, during which responses to certain stimuli were associated with monetary reward. These stimuli were then included as non-predictive cues in a spatial cueing task. Participants underwent functional brain imaging during both task phases.

Results

The results show that striatal activity during the learning phase predicted increased visual cortical and parietal activity and decreased ventro-medial prefrontal cortex activity in response to conditioned stimuli during the test. Striatal activity was also associated with anterior cingulate cortex activation when the reward-conditioned stimulus directed attention away from the target.

Discussion

Our findings suggest that striatal activity during reward conditioning predicts the degree to which reward history biases attention through learning-induced changes in visual and parietal activities.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Computationally Informed Interventions for Targeting Compulsive Behaviors

Compulsive behaviors are central to addiction and obsessive-compulsive disorder and can be understood as a failure of adaptive decision making. Particularly, they can be conceptualized as an imbalance in behavioral control, such that behavior is guided predominantly by learned rather than inferred outcome expectations. Inference is a computational process required for adaptive behavior, and recent work across species has identified the neural circuitry that supports inference-based decision making. This includes the orbitofrontal cortex, which has long been implicated in disorders of compulsive behavior. Inspired by evidence that modulating orbitofrontal cortex activity can alter inference-based behaviors, here we discuss noninvasive approaches to target these circuits in humans. Specifically, we discuss the potential of network-targeted transcranial magnetic stimulation and real-time neurofeedback to modulate the neural underpinnings of inference. Both interventions leverage recent advances in our understanding of the neurocomputational mechanisms of inference-based behavior and may be used to complement current treatment approaches for behavioral disorders.

Goals, usefulness and abstraction in value-based choice

Colombian drug lord Pablo Escobar, while on the run, purportedly burned two million dollars in banknotes to keep his daughter warm. A stark reminder that, in life, circumstances and goals can quickly change, forcing us to reassess and modify our values on-the-fly. Studies in decision-making and neuroeconomics have often implicitly equated value to reward, emphasising the hedonic and automatic aspect of the value computation, while overlooking its functional (concept-like) nature. Here we outline the computational and biological principles that enable the brain to compute the usefulness of an option or action by creating abstractions that flexibly adapt to changing goals. We present different algorithmic architectures, comparing ideas from artificial intelligence (AI) and cognitive neuroscience with psychological theories and, when possible, drawing parallels.

Humans combine value learning and hypothesis testing strategically in multi-dimensional probabilistic reward learning

Realistic and complex decision tasks often allow for many possible solutions. How do we find the correct one? Introspection suggests a process of trying out solutions one after the other until success. However, such methodical serial testing may be too slow, especially in environments with noisy feedback. Alternatively, the underlying learning process may involve implicit reinforcement learning that learns about many possibilities in parallel. Here we designed a multi-dimensional probabilistic active-learning task tailored to study how people learn to solve such complex problems. Participants configured three-dimensional stimuli by selecting features for each dimension and received probabilistic reward feedback. We manipulated task complexity by changing how many feature dimensions were relevant to maximizing reward, as well as whether this information was provided to the participants. To investigate how participants learn the task, we examined models of serial hypothesis testing, feature-based reinforcement learning, and combinations of the two strategies. Model comparison revealed evidence for hypothesis testing that relies on reinforcement-learning when selecting what hypothesis to test. The extent to which participants engaged in hypothesis testing depended on the instructed task complexity: people tended to serially test hypotheses when instructed that there were fewer relevant dimensions, and relied more on gradual and parallel learning of feature values when the task was more complex. This demonstrates a strategic use of task information to balance the costs and benefits of the two methods of learning.

From internal models toward metacognitive AI

In several papers published in Biological Cybernetics in the 1980s and 1990s, Kawato and colleagues proposed computational models explaining how internal models are acquired in the cerebellum. These models were later supported by neurophysiological experiments using monkeys and neuroimaging experiments involving humans. These early studies influenced neuroscience from basic, sensory-motor control to higher cognitive functions. One of the most perplexing enigmas related to internal models is to understand the neural mechanisms that enable animals to learn large-dimensional problems with so few trials. Consciousness and metacognition-the ability to monitor one's own thoughts, may be part of the solution to this enigma. Based on literature reviews of the past 20 years, here we propose a computational neuroscience model of metacognition. The model comprises a modular hierarchical reinforcement-learning architecture of parallel and layered, generative-inverse model pairs. In the prefrontal cortex, a distributed executive network called the "cognitive reality monitoring network" (CRMN) orchestrates conscious involvement of generative-inverse model pairs in perception and action. Based on mismatches between computations by generative and inverse models, as well as reward prediction errors, CRMN computes a "responsibility signal" that gates selection and learning of pairs in perception, action, and reinforcement learning. A high responsibility signal is given to the pairs that best capture the external world, that are competent in movements (small mismatch), and that are capable of reinforcement learning (small reward-prediction error). CRMN selects pairs with higher responsibility signals as objects of metacognition, and consciousness is determined by the entropy of responsibility signals across all pairs. This model could lead to new-generation AI, which exhibits metacognition, consciousness, dimension reduction, selection of modules and corresponding representations, and learning from small samples. It may also lead to the development of a new scientific paradigm that enables the causal study of consciousness by combining CRMN and decoded neurofeedback.

Metacognitive resources for adaptive learning⋆

Biological organisms display remarkably flexible behaviours. This is an area of active investigation, in particular in the fields of artificial intelligence, computational and cognitive neuroscience. While inductive biases and broader cognitive functions are undoubtedly important, the ability to monitor and evaluate one's performance or oneself -- metacognition -- strikes as a powerful resource for efficient learning. Often measured as decision confidence in neuroscience and psychology experiments, metacognition appears to reflect a broad range of abstraction levels and downstream behavioural effects. Within this context, the formal investigation of how metacognition interacts with learning processes is a recent endeavour. Of special interest are the neural and computational underpinnings of confidence and reinforcement learning modules. This review discusses a general hierarchy of confidence functions and their neuro-computational relevance for adaptive behaviours. It then introduces novel ways to study the formation and use of meta-representations and nonconscious mental representations related to learning and confidence, and concludes with a discussion on outstanding questions and wider perspectives.