Visual novelty, curiosity, and intrinsic reward in machine learning and the brain

Does repetition enhance curiosity to learn trivia question answers? Implications for memory and motivated learning

Curiosity, an intrinsic desire to seek new information, benefits motivation and learning. While curiosity is associated with novelty, less is known about how the repetition of a question without its answer affects curiosity and memory. In two experiments, participants viewed 60 trivia questions, half of which were repeated, and rated their curiosity to learn the answers. Repeated questions had their answers revealed during the second presentation, and participants were given a cued-recall test after 24 h. We found that curiosity ratings remained constant across presentations, but when repeated and non-repeated questions were intermixed, participants were more curious about non-repeated questions, which were relatively more novel (Experiment 1). However, when participants guessed answers before studying them (pretesting), they were more curious about repeated questions (Experiment 2). Curiosity ratings also increased across presentations, perhaps reflecting greater cognitive agency motivated by an eagerness to verify one's guess. Overall, the subjective experience of curiosity appears to be influenced by both relative novelty, as manipulated through repetition, and task demands, specifically whether individuals engage in pretesting, indicating that curiosity-based learning is shaped by various cognitive operations.

To know or not to know? Curiosity and the value of prospective information in animals

Humans and other animals often seek instrumental information to strategically improve their decisions in the present. Our curiosity also leads us to acquire non-instrumental information that is not immediately useful but can be encoded in memory and stored for use in the future by means of episodic recall. Despite its adaptive benefits and central role in human cognition, questions remain about the cognitive mechanisms and evolutionary origins that underpin curiosity. Here, we comparatively review recent empirical studies that some authors have suggested reflects curiosity in nonhuman animals. We focus on findings from laboratory tasks in which individuals can choose to gain advanced information about uncertain future outcomes, even though the information cannot be used to increase future rewards and is often costly. We explore the prevalence of preferences in these tasks across animals, discuss the theoretical advances that they have promoted, and outline some limitations in contemporary research. We also discuss several features of human curiosity that can guide future empirical research aimed at characterising and understanding curiosity in animals. Though the prevalence of curiosity in animals is actively debated, we surmise that investigating behavioural candidates for curiosity-motivated behaviour in a broader range of species and contexts, should help promote theoretical advances in our understanding of cognitive principles and evolutionary pressures that support curiosity-driven behaviour.

Adaptive Decision-Making "Fast" and "Slow": A Model of Creative Thinking

The late Daniel Kahneman introduced the concept of fast and slow thinking, representing two distinct cognitive systems involved in decision-making (DM). Fast thinking (System 1) operates intuitively and spontaneously. In contrast, slow thinking (System 2) is characterized by deliberation and analytical reasoning. Following Kahneman's view, called the biases view, we suggest a framework involving the interplay between two systems, the bottom-up and top-down approaches. These two approaches involve various modalities, including learning skills, perception, cognition, attention, and emotion. Accordingly, we incorporate temporal modulation, which varies based on individual differences and accounts for adaptive DM. Our overarching framework elucidates how the brain dynamically allocates resources for adaptive DM and how creative mental processes could drive it. We highlight the immense value of interdisciplinary research collaboration in progressing the empirical research of our proposed framework.

A spiking neural network for active efficient coding

Biological vision systems simultaneously learn to efficiently encode their visual inputs and to control the movements of their eyes based on the visual input they sample. This autonomous joint learning of visual representations and actions has previously been modeled in the Active Efficient Coding (AEC) framework and implemented using traditional frame-based cameras. However, modern event-based cameras are inspired by the retina and offer advantages in terms of acquisition rate, dynamic range, and power consumption. Here, we propose a first AEC system that is fully implemented as a Spiking Neural Network (SNN) driven by inputs from an event-based camera. This input is efficiently encoded by a two-layer SNN, which in turn feeds into a spiking reinforcement learner that learns motor commands to maximize an intrinsic reward signal. This reward signal is computed directly from the activity levels of the first two layers. We test our approach on two different behaviors: visual tracking of a translating target and stabilizing the orientation of a rotating target. To the best of our knowledge, our work represents the first ever fully spiking AEC model.

Simple synaptic modulations implement diverse novelty computations

Detecting novelty is ethologically useful for an organism's survival. Recent experiments characterize how different types of novelty over timescales from seconds to weeks are reflected in the activity of excitatory and inhibitory neuron types. Here, we introduce a learning mechanism, familiarity-modulated synapses (FMSs), consisting of multiplicative modulations dependent on presynaptic or pre/postsynaptic neuron activity. With FMSs, network responses that encode novelty emerge under unsupervised continual learning and minimal connectivity constraints. Implementing FMSs within an experimentally constrained model of a visual cortical circuit, we demonstrate the generalizability of FMSs by simultaneously fitting absolute, contextual, and omission novelty effects. Our model also reproduces functional diversity within cell subpopulations, leading to experimentally testable predictions about connectivity and synaptic dynamics that can produce both population-level novelty responses and heterogeneous individual neuron signals. Altogether, our findings demonstrate how simple plasticity mechanisms within a cortical circuit structure can produce qualitatively distinct and complex novelty responses.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.

Novelty and uncertainty differentially drive exploration across development

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.

Systemic kappa opioid receptor antagonism accelerates reinforcement learning via augmentation of novelty processing in male mice

Selective inhibition of kappa opioid receptors (KORs) is highly anticipated as a pharmacotherapeutic intervention for substance use disorders and depression. The accepted explanation for KOR antagonist-induced amelioration of aberrant behaviors posits that KORs globally function as a negative valence system; antagonism thereby blunts the behavioral influence of negative internal states such as anhedonia and negative affect. While effects of systemic KOR manipulations have been widely reproduced, explicit evaluation of negative valence as an explanatory construct is lacking. Here, we tested a series of falsifiable hypotheses generated a priori based on the negative valence model by pairing reinforcement learning tasks with systemic pharmacological KOR blockade in male C57BL/6J mice. The negative valence model failed to predict multiple experimental outcomes: KOR blockade accelerated contingency learning during both positive and negative reinforcement without altering innate responses to appetitive or aversive stimuli. We next proposed novelty processing, which influences learning independent of valence, as an alternative explanatory construct. Hypotheses based on novelty processing predicted subsequent observations: KOR blockade increased exploration of a novel, but not habituated, environment and augmented the reinforcing efficacy of novel visual stimuli in a sensory reinforcement task. Together, these results revise and extend long-standing theories of KOR system function.

Face familiarity detection with complex synapses

Synaptic plasticity is a complex phenomenon involving multiple biochemical processes that operate on different timescales. Complexity can greatly increase memory capacity when the variables characterizing the synaptic dynamics have limited precision, as shown in simple memory retrieval problems involving random patterns. Here we turn to a real-world problem, face familiarity detection, and we show that synaptic complexity can be harnessed to store in memory a large number of faces that can be recognized at a later time. The number of recognizable faces grows almost linearly with the number of synapses and quadratically with the number of neurons. Complex synapses outperform simple ones characterized by a single variable, even when the total number of dynamical variables is matched. Complex and simple synapses have distinct signatures that are testable in experiments. Our results indicate that a system with complex synapses can be used in real-world tasks such as face familiarity detection.

The zona incerta in control of novelty seeking and investigation across species

Many organisms rely on a capacity to rapidly replicate, disperse, and evolve when faced with uncertainty and novelty. But mammals do not evolve and replicate quickly. They rely on a sophisticated nervous system to generate predictions and select responses when confronted with these challenges. An important component of their behavioral repertoire is the adaptive context-dependent seeking or avoiding of perceptually novel objects, even when their values have not yet been learned. Here, we outline recent cross-species breakthroughs that shed light on how the zona incerta (ZI), a relatively evolutionarily conserved brain area, supports novelty-seeking and novelty-related investigations. We then conjecture how the architecture of the ZI's anatomical connectivity - the wide-ranging top-down cortical inputs to the ZI, and its specifically strong outputs to both the brainstem action controllers and to brain areas involved in action value learning - place the ZI in a unique role at the intersection of cognitive control and learning.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Salience memories formed by value, novelty and aversiveness jointly shape object responses in the prefrontal cortex and basal ganglia

Ecological fitness depends on maintaining object histories to guide future interactions. Recent evidence shows that value memory changes passive visual responses to objects in ventrolateral prefrontal cortex (vlPFC) and substantia nigra reticulata (SNr). However, it is not known whether this effect is limited to reward history and if not how cross-domain representations are organized within the same or different neural populations in this corticobasal circuitry. To address this issue, visual responses of the same neurons across appetitive, aversive and novelty domains were recorded in vlPFC and SNr. Results showed that changes in visual responses across domains happened in the same rather than separate populations and were related to salience rather than valence of objects. Furthermore, while SNr preferentially encoded outcome related salience memory, vlPFC encoded salience memory across all domains in a correlated fashion, consistent with its role as an information hub to guide behavior.

A neural theory for counting memories

Keeping track of the number of times different stimuli have been experienced is a critical computation for behavior. Here, we propose a theoretical two-layer neural circuit that stores counts of stimulus occurrence frequencies. This circuit implements a data structure, called a count sketch, that is commonly used in computer science to maintain item frequencies in streaming data. Our first model implements a count sketch using Hebbian synapses and outputs stimulus-specific frequencies. Our second model uses anti-Hebbian plasticity and only tracks frequencies within four count categories ("1-2-3-many"), which trades-off the number of categories that need to be distinguished with the potential ethological value of those categories. We show how both models can robustly track stimulus occurrence frequencies, thus expanding the traditional novelty-familiarity memory axis from binary to discrete with more than two possible values. Finally, we show that an implementation of the "1-2-3-many" count sketch exists in the insect mushroom body.

The effect of smartphone use on mental effort, learning, and creativity

We argue that scientific studies have not directly assessed the key cognitive processes affected by smartphone use. We propose that smartphone use can be disruptively habitual, with the main detrimental consequence being an inability to exert prolonged mental effort. This inability might negatively affect real-life creativity and domain-specific knowledge acquisition.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

How the value of the environment controls persistence in visual search

Classic foraging theory predicts that humans and animals aim to gain maximum reward per unit time. However, in standard instrumental conditioning tasks individuals adopt an apparently suboptimal strategy: they respond slowly when the expected value is low. This reward-related bias is often explained as reduced motivation in response to low rewards. Here we present evidence this behavior is associated with a complementary increased motivation to search the environment for alternatives. We trained monkeys to search for reward-related visual targets in environments with different values. We found that the reward-related bias scaled with environment value, was consistent with persistent searching after the target was already found, and was associated with increased exploratory gaze to objects in the environment. A novel computational model of foraging suggests that this search strategy could be adaptive in naturalistic settings where both environments and the objects within them provide partial information about hidden, uncertain rewards.

The population doctrine in cognitive neuroscience

A major shift is happening within neurophysiology: a population doctrine is drawing level with the single-neuron doctrine that has long dominated the field. Population-level ideas have so far had their greatest impact in motor neuroscience, but they hold great promise for resolving open questions in cognition as well. Here, we codify the population doctrine and survey recent work that leverages this view to specifically probe cognition. Our discussion is organized around five core concepts that provide a foundation for population-level thinking: (1) state spaces, (2) manifolds, (3) coding dimensions, (4) subspaces, and (5) dynamics. The work we review illustrates the progress and promise that population-level thinking holds for cognitive neuroscience-for delivering new insight into attention, working memory, decision-making, executive function, learning, and reward processing.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

Acute, but not longer-term, exposure to environmental enrichment attenuates Pavlovian cue-evoked conditioned approach and Fos expression in the prefrontal cortex in mice

Exposure to environmental enrichment can modify the impact of motivationally relevant stimuli. For instance, previous studies in rats have found that even a brief, acute (~1 day), but not chronic, exposure to environmentally enriched (EE) housing attenuates instrumental lever pressing for sucrose-associated cues in a conditioned reinforcement setup. Moreover, acute EE reduces corticoaccumbens activity, as measured by decreases in expression of the neuronal activity marker "Fos." Currently, it is not known whether acute EE also reduces sucrose seeking and corticoaccumbens activity elicited by non-contingent or "forced" exposure to sucrose cues, which more closely resembles cue exposure encountered in daily life. We therefore measured the effects of acute/intermittent (1 day or 6 day of EE prior to test day) versus chronic (EE throughout conditioning lasting until test day) EE on the ability of a Pavlovian sucrose cue to elicit sucrose seeking (conditioned approach) and Fos expression in the medial prefrontal cortex (mPFC), orbitofrontal cortex (OFC), and nucleus accumbens (NAc) in mice. One day, but not 6 day or chronic EE , reduced sucrose seeking and Fos in the deep layers of the dorsal mPFC. By contrast, 1 day, 6 day, and chronic EE all reduced Fos in the shallow layers of the OFC. None of the EE manipulations modulated NAc Fos expression. We reveal how EE reduces behavioral reactivity to sucrose cues by reducing activity in select prefrontal cortical brain areas. Our work further demonstrates the robustness of EE in its ability to modulate various forms of reward-seeking across species.

The growth and form of knowledge networks by kinesthetic curiosity

Throughout life, we might seek a calling, companions, skills, entertainment, truth, self-knowledge, beauty, and edification. The practice of curiosity can be viewed as an extended and open-ended search for valuable information with hidden identity and location in a complex space of interconnected information. Despite its importance, curiosity has been challenging to computationally model because the practice of curiosity often flourishes without specific goals, external reward, or immediate feedback. Here, we show how network science, statistical physics, and philosophy can be integrated into an approach that coheres with and expands the psychological taxonomies of specific-diversive and perceptual-epistemic curiosity. Using this interdisciplinary approach, we distill functional modes of curious information seeking as searching movements in information space. The kinesthetic model of curiosity offers a vibrant counterpart to the deliberative predictions of model-based reinforcement learning. In doing so, this model unearths new computational opportunities for identifying what makes curiosity curious.

Brain Networks Sensitive to Object Novelty, Value, and Their Combination

Novel and valuable objects are motivationally attractive for animals including primates. However, little is known about how novelty and value processing is organized across the brain. We used fMRI in macaques to map brain responses to visual fractal patterns varying in either novelty or value dimensions and compared the results with the structure of functionally connected brain networks determined at rest. The results show that different brain networks possess unique combinations of novelty and value coding. One network identified in the ventral temporal cortex preferentially encoded object novelty, whereas another in the parietal cortex encoded the learned value. A third network, broadly composed of temporal and prefrontal areas (TP network), along with functionally connected portions of the striatum, amygdala, and claustrum, encoded both dimensions with similar activation dynamics. Our results support the emergence of a common currency signal in the TP network that may underlie the common attitudes toward novel and valuable objects.

Discovery of a Role for Rab3b in Habituation and Cocaine Induced Locomotor Activation in Mice Using Heterogeneous Functional Genomic Analysis

Substance use disorders are prevalent and present a tremendous societal cost but the mechanisms underlying addiction behavior are poorly understood and few biological treatments exist. One strategy to identify novel molecular mechanisms of addiction is through functional genomic experimentation. However, results from individual experiments are often noisy. To address this problem, the convergent analysis of multiple genomic experiments can discern signal from these studies. In the present study, we examine genetic loci that modulate the locomotor response to cocaine identified in the recombinant inbred (BXD RI) genetic reference population. We then applied the GeneWeaver software system for heterogeneous functional genomic analysis to integrate and aggregate multiple studies of addiction genomics, resulting in the identification of Rab3b as a functional correlate of the locomotor response to cocaine in rodents. This gene encodes a member of the RAB family of Ras-like GTPases known to be involved in trafficking of secretory and endocytic vesicles in eukaryotic cells. The convergent evidence for a role of Rab3b includes co-occurrence in previously published genetic mapping studies of cocaine related behaviors; methamphetamine response and cocaine- and amphetamine-regulated transcript prepropeptide (Cartpt) transcript abundance; evidence related to other addictive substances; density of polymorphisms; and its expression pattern in reward pathways. To evaluate this finding, we examined the effect of RAB3 complex perturbation in cocaine response. B6;129-Rab3btm1Sud Rab3ctm1sud Rab3dtm1sud triple null mice (Rab3bcd -/-) exhibited significant deficits in habituation, and increased acute and repeated cocaine responses. This previously unidentified mechanism of the behavioral predisposition and response to cocaine is an example of many that can be identified and validated using aggregate genomic studies.

Attention in Psychology, Neuroscience, and Machine Learning

Attention is the important ability to flexibly control limited computational resources. It has been studied in conjunction with many other topics in neuroscience and psychology including awareness, vigilance, saliency, executive control, and learning. It has also recently been applied in several domains in machine learning. The relationship between the study of biological attention and its use as a tool to enhance artificial neural networks is not always clear. This review starts by providing an overview of how attention is conceptualized in the neuroscience and psychology literature. It then covers several use cases of attention in machine learning, indicating their biological counterparts where they exist. Finally, the ways in which artificial attention can be further inspired by biology for the production of complex and integrative systems is explored.