Control of movement vigor and decision making during foraging

Slowing of Movements in Healthy Aging as a Rational Economic Response to an Elevated Effort Landscape

Why do we move slower as we grow older? The reward circuits of the brain, which tend to invigorate movements, decline with aging, raising the possibility that reduced vigor is due to the diminishing value that our brain assigns to movements. However, as we grow older, it also becomes more effortful to make movements. Is age-related slowing principally a consequence of increased effort costs from the muscles, or reduced valuation of reward by the brain? Here, we first quantified the cost of reaching via metabolic energy expenditure in human participants (male and female), and found that older adults consumed more energy than the young at a given speed. Thus, movements are objectively more costly for older adults. Next, we observed that when reward increased, older adults, like the young, responded by initiating their movements earlier. Yet, unlike the young, they were unwilling to increase their movement speed. Was their reluctance to reach quicker for rewards due to the increased effort costs, or because they ascribed less value to the movement? Motivated by a mathematical model, we next made the young experience a component of aging by making their movements more effortful. Now the young responded to reward by reacting faster but chose not to increase their movement speed. This suggests that slower movements in older adults are partly driven by an adaptive response to an elevated effort landscape. Moving slower may be a rational economic response the brain is making to mitigate the elevated effort costs that accompany aging.

Effects of reward and effort history on decision making and movement vigor during foraging

During foraging, animals explore a site and harvest reward and then abandon that site and travel to the next opportunity. One aspect of this behavior involves decision making, and the other involves movement control. These two aspects of behavior may be linked via an underlying desire to maximize a single normative utility: the sum of all rewards acquired, minus all efforts expended, divided by time. According to this theory, the history of rewards, and not just its immediate availability, should dictate how long one should stay and harvest reward and how vigorously one should travel to the next opportunity. We tested this theory in a series of experiments in which humans used their hand to harvest tokens at a reward patch and then used their arm to reach toward another patch. After a history of high rewards, the subjects not only shortened their harvest duration but also moved more vigorously toward the next reward opportunity. In contrast, after a history of high effort they lengthened their harvest duration but reduced their movement vigor, reaching more slowly to the next reward site. Thus, a history of high reward or low effort biased decisions by promoting early abandonment of the reward site and biased movements by promoting vigor.NEW & NOTEWORTHY Much of life is spent foraging. Whereas previous work has focused on the decision regarding time spent harvesting from a reward patch, here we test the idea that both decision making and movement control are tuned to optimize the net rate of reward in an environment. Our results show that movement patterns reflect not just immediate expectations but also past experiences in the environment, providing fundamental insight into the factors governing volitional control of arm movements.

Using virtual reality-based neurocognitive testing and eye tracking to study naturalistic cognitive-motor performance

Natural human behavior arises from continuous interactions between the cognitive and motor domains. However, assessments of cognitive abilities are typically conducted using pen and paper tests, i.e., in isolation from "real life" cognitive-motor behavior and in artificial contexts. In the current study, we aimed to assess cognitive-motor task performance in a more naturalistic setting while recording multiple motor and eye tracking signals. Specifically, we aimed to (i) delineate the contribution of cognitive and motor components to overall task performance and (ii) probe for a link between cognitive-motor performance and pupil size. To that end, we used a virtual reality (VR) adaptation of a well-established neurocognitive test for executive functions, the 'Color Trails Test' (CTT). The VR-CTT involves performing 3D reaching movements to follow a trail of numbered targets. To tease apart the cognitive and motor components of task performance, we included two additional conditions: a condition where participants only used their eyes to perform the CTT task (using an eye tracking device), incurring reduced motor demands, and a condition where participants manually tracked visually-cued targets without numbers on them, incurring reduced cognitive demands. Our results from a group of 30 older adults (>65) showed that reducing cognitive demands shortened completion times more extensively than reducing motor demands. Conditions with higher cognitive demands had longer target search time, as well as decreased movement execution velocity and head-hand coordination. We found larger pupil sizes in the more cognitively demanding conditions, and an inverse correlation between pupil size and completion times across individuals in all task conditions. Lastly, we found a possible link between VR-CTT performance measures and clinical signatures of participants (fallers versus non-fallers). In summary, performance and pupil parameters were mainly dependent on task cognitive load, while maintaining systematic interindividual differences. We suggest that this paradigm opens the possibility for more detailed profiling of individual cognitive-motor performance capabilities in older adults and other at-risk populations.

A software tool for at-home measurement of sensorimotor adaptation

Sensorimotor adaptation is traditionally studied in well-controlled laboratory settings with specialized equipment. However, recent public health concerns such as the COVID-19 pandemic, as well as a desire to recruit a more diverse study population, have led the motor control community to consider at-home study designs. At-home motor control experiments are still rare because of the requirement to write software that can be easily used by anyone on any platform. To this end, we developed software that runs locally on a personal computer. The software provides audiovisual instructions and measures the ability of the subject to control the cursor in the context of visuomotor perturbations. We tested the software on a group of at-home participants and asked whether the adaptation principles inferred from in-lab measurements were reproducible in the at-home setting. For example, we manipulated the perturbations to test whether there were changes in adaptation rates (savings and interference), whether adaptation was associated with multiple timescales of memory (spontaneous recovery), and whether we could selectively suppress subconscious learning (delayed feedback, perturbation variability) or explicit strategies (limited reaction time). We found remarkable similarity between in-lab and at-home behaviors across these experimental conditions. Thus, we developed a software tool that can be used by research teams with little or no programming experience to study mechanisms of adaptation in an at-home setting.

Effort cost of harvest affects decisions and movement vigor of marmosets during foraging

Our decisions are guided by how we perceive the value of an option, but this evaluation also affects how we move to acquire that option. Why should economic variables such as reward and effort alter the vigor of our movements? In theory, both the option that we choose and the vigor with which we move contribute to a measure of fitness in which the objective is to maximize rewards minus efforts, divided by time. To explore this idea, we engaged marmosets in a foraging task in which on each trial they decided whether to work by making saccades to visual targets, thus accumulating food, or to harvest by licking what they had earned. We varied the effort cost of harvest by moving the food tube with respect to the mouth. Theory predicted that the subjects should respond to the increased effort costs by choosing to work longer, stockpiling food before commencing harvest, but reduce their movement vigor to conserve energy. Indeed, in response to an increased effort cost of harvest, marmosets extended their work duration, but slowed their movements. These changes in decisions and movements coincided with changes in pupil size. As the effort cost of harvest declined, work duration decreased, the pupils dilated, and the vigor of licks and saccades increased. Thus, when acquisition of reward became effortful, the pupils constricted, the decisions exhibited delayed gratification, and the movements displayed reduced vigor.

From neural noise to co-adaptability: Rethinking the multifaceted architecture of motor variability

In the last decade, the source and the functional meaning of motor variability have attracted considerable attention in behavioral and brain sciences. This construct classically combined different levels of description, variable internal robustness or coherence, and multifaceted operational meanings. We provide here a comprehensive review of the literature with the primary aim of building a precise lexicon that goes beyond the generic and monolithic use of motor variability. In the pars destruens of the work, we model three domains of motor variability related to peculiar computational elements that influence fluctuations in motor outputs. Each domain is in turn characterized by multiple sub-domains. We begin with the domains of noise and differentiation. However, the main contribution of our model concerns the domain of adaptability, which refers to variation within the same exact motor representation. In particular, we use the terms learning and (social)fitting to specify the portions of motor variability that depend on our propensity to learn and on our largely constitutive propensity to be influenced by external factors. A particular focus is on motor variability in the context of the sub-domain named co-adaptability. Further groundbreaking challenges arise in the modeling of motor variability. Therefore, in a separate pars construens, we attempt to characterize these challenges, addressing both theoretical and experimental aspects as well as potential clinical implications for neurorehabilitation. All in all, our work suggests that motor variability is neither simply detrimental nor beneficial, and that studying its fluctuations can provide meaningful insights for future research.

Common Mechanisms of Learning in Motor and Cognitive Systems

Rapid progress in our understanding of the brain's learning mechanisms has been accomplished over the past decade, particularly with conceptual advances, including representing behavior as a dynamical system, large-scale neural population recordings, and new methods of analysis of neuronal populations. However, motor and cognitive systems have been traditionally studied with different methods and paradigms. Recently, some common principles, evident in both behavior and neural activity, that underlie these different types of learning have become to emerge. Here we review results from motor and cognitive learning, relying on different techniques and studying different systems to understand the mechanisms of learning. Movement is intertwined with cognitive operations, and its dynamics reflect cognitive variables. Training, in either motor or cognitive tasks, involves recruitment of previously unresponsive neurons and reorganization of neural activity in a low dimensional manifold. Mapping of new variables in neural activity can be very rapid, instantiating flexible learning of new tasks. Communication between areas is just as critical a part of learning as are patterns of activity within an area emerging with learning. Common principles across systems provide a map for future research.

Understanding patch foraging strategies across development

Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.

The value of time in the invigoration of human movements when interacting with a robotic exoskeleton

Time and effort are thought to be subjectively balanced during the planning of goal-directed actions, thereby setting the vigor of volitional movements. Theoretical models predicted that the value of time should then amount to high levels of effort. However, the time-effort trade-off has so far only been studied for a narrow range of efforts. To investigate the extent to which humans can invest in a time-saving effort, we used a robotic exoskeleton to substantially vary the energetic cost associated with a certain vigor during reaching movements. In this situation, minimizing the time-effort trade-off should lead to high and low human efforts for upward and downward movements, respectively. Consistently, all participants expended substantial amounts of energy upward and remained essentially inactive by harnessing the work of gravity downward, while saving time in both cases. A common time-effort trade-off may therefore determine the vigor of reaching movements for a wide range of efforts.

Moving, fast and slow: behavioural insights into bradykinesia in Parkinson's disease

The debilitating symptoms of Parkinson's disease, including the hallmark slowness of movement, termed bradykinesia, were described more than 100 years ago. Despite significant advances in elucidating the genetic, molecular and neurobiological changes in Parkinson's disease, it remains conceptually unclear exactly why patients with Parkinson's disease move slowly. To address this, we summarize behavioural observations of movement slowness in Parkinson's disease and discuss these findings in a behavioural framework of optimal control. In this framework, agents optimize the time it takes to gather and harvest rewards by adapting their movement vigour according to the reward that is at stake and the effort that needs to be expended. Thus, slow movements can be favourable when the reward is deemed unappealing or the movement very costly. While reduced reward sensitivity, which makes patients less inclined to work for reward, has been reported in Parkinson's disease, this appears to be related mainly to motivational deficits (apathy) rather than bradykinesia. Increased effort sensitivity has been proposed to underlie movement slowness in Parkinson's disease. However, careful behavioural observations of bradykinesia are inconsistent with abnormal computations of effort costs due to accuracy constraints or movement energetic expenditure. These inconsistencies can be resolved when considering that a general disability to switch between stable and dynamic movement states can contribute to an abnormal composite effort cost related to movement in Parkinson's disease. This can account for paradoxical observations such as the abnormally slow relaxation of isometric contractions or difficulties in halting a movement in Parkinson's disease, both of which increase movement energy expenditure. A sound understanding of the abnormal behavioural computations mediating motor impairment in Parkinson's disease will be vital for linking them to their underlying neural dynamics in distributed brain networks and for grounding future experimental studies in well-defined behavioural frameworks.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Effort cost of reaching prompts vigor reduction in older adults

As people age, they move slower. Is age-related reduction in vigor a reflection of a reduced valuation of reward by the brain, or a consequence of increased effort costs by the muscles? Here, we quantified cost of movements objectively via the metabolic energy that young and old participants consumed during reaching and found that in order reach at a given speed, older adults expended more energy than the young. We next quantified how reward modulated movements in the same populations and found that like the young, older adults responded to increased reward by initiating their movements earlier. Yet, their movements were less sensitive to increased reward, resulting in little or no modulation of reach speed. Lastly, we quantified the effect of increased effort on how reward modulated movements in young adults. Like the effects of aging, when faced with increased effort the young adults responded to reward primarily by reacting faster, with little change in movement speed. Therefore, reaching required greater energetic expenditure in the elderly, suggesting that the slower movements and reactions exhibited in aging are partly driven by an adaptive response to an elevation in the energetic landscape of effort. That is, moving slower appears to be a rational economic consequence of aging.

Significance statement

Healthy aging coincides with a reduction in speed, or vigor, of walking, reaching, and eye movements. Here we focused on disentangling two opposing sources of aging-related movement slowing: reduced reward sensitivity due to loss of dopaminergic tone, or increased energy expenditure movements related to mitochondrial or muscular inefficiencies. Through a series of three experiments and construction of a computational model, here we demonstrate that transient changes in reaction time and movement speed together offer a quantifiable metric to differentiate between reward- and effort-based alterations in movement vigor. Further, we suggest that objective increases in the metabolic cost of moving, not reductions in reward valuation, are driving much of the movement slowing occurring alongside healthy aging.

Effort cost of harvest affects decisions and movement vigor of marmosets during foraging

Our decisions are guided by how we perceive the value of an option, but this evaluation also affects how we move to acquire that option. Why should economic variables such as reward and effort alter the vigor of our movements? In theory, both the option that we choose and the vigor with which we move contribute to a measure of fitness in which the objective is to maximize rewards minus efforts, divided by time. To explore this idea, we engaged marmosets in a foraging task in which on each trial they decided whether to work by making saccades to visual targets, thus accumulating food, or to harvest by licking what they had earned. We varied the effort cost of harvest by moving the food tube with respect to the mouth. Theory predicted that the subjects should respond to the increased effort costs by choosing to work longer, stockpiling food before commencing harvest, but reduce their movement vigor to conserve energy. Indeed, in response to an increased effort cost of harvest, marmosets extended their work duration, but slowed their movements. These changes in decisions and movements coincided with changes in pupil size. As the effort cost of harvest declined, work duration decreased, the pupils dilated, and the vigor of licks and saccades increased. Thus, when acquisition of reward became effortful, the pupils constricted, the decisions exhibited delayed gratification, and the movements displayed reduced vigor.

Significance statement

Our results suggest that as the brainstem neuromodulatory circuits that control pupil size respond to effort costs, they alter computations in the brain regions that control decisions, encouraging work and delaying gratification, and the brain regions that control movements, reducing vigor and suppressing energy expenditure. This coordinated response suggests that decisions and actions are part of a single control policy that aims to maximize a variable relevant to fitness: the capture rate.

Integrated control of non-motor and motor efforts during perceptual decision-making and action execution: a pilot study

Humans daily life is characterized by a succession of voluntary actions. Since energy resources are limited, the ability to invest the appropriate amount of effort for selecting and executing these actions is a hallmark of adapted behavior. Recent studies indicate that decisions and actions share important principles, including the optimization of their duration when the context requires it. In the present pilot study, we test the hypothesis that the management of effort-related energy resources is shared between decision and action too. Healthy human subjects performed a perceptual decision task where they had to choose between two levels of effort to invest in making the decision (i.e. two levels of perceptual difficulty), and report it with a reaching movement. Crucially, the movement accuracy requirement gradually increased from trial to trial depending on participants' decision performance. Results indicate an overall moderate and non-significant impact of the increasing motor difficulty on the choice of the non-motor (decision) effort to invest in each trial and on decision performance. By contrast, motor performance strongly decreased depending on both the motor and decisional difficulties. Together, the results support the hypothesis of an integrated management of the effort-related energy resources between decision and action. They also suggest that in the present task, the mutualized resources are primarily allocated to the decision-making process to the detriment of movements.

Cognitive effort and active inference

This paper aims to integrate some key constructs in the cognitive neuroscience of cognitive control and executive function by formalising the notion of cognitive (or mental) effort in terms of active inference. To do so, we call upon a task used in neuropsychology to assess impulse inhibition-a Stroop task. In this task, participants must suppress the impulse to read a colour word and instead report the colour of the text of the word. The Stroop task is characteristically effortful, and we unpack a theory of mental effort in which, to perform this task accurately, participants must overcome prior beliefs about how they would normally act. However, our interest here is not in overt action, but in covert (mental) action. Mental actions change our beliefs but have no (direct) effect on the outside world-much like deploying covert attention. This account of effort as mental action lets us generate multimodal (choice, reaction time, and electrophysiological) data of the sort we might expect from a human participant engaging in this task. We analyse how parameters determining cognitive effort influence simulated responses and demonstrate that-when provided only with performance data-these parameters can be recovered, provided they are within a certain range.

Unlocking a new dimension in the speed-accuracy trade-off

Why do we sometimes spend too much time on seemingly impossible-to-solve tasks instead of just moving on? Masís et al. provide a new perspective on the speed-accuracy trade-off (SAT), showing that, although prolonging deliberation looks suboptimal in the short run, it is a long-term investment that helps organisms reach proficient performance more rapidly.

Movement characteristics impact decision-making and vice versa

Previous studies suggest that humans are capable of coregulating the speed of decisions and movements if promoted by task incentives. It is unclear however whether such behavior is inherent to the process of translating decisional information into movements, beyond posing a valid strategy in some task contexts. Therefore, in a behavioral online study we imposed time constraints to either decision- or movement phases of a sensorimotor task, ensuring that coregulating decisions and movements was not promoted by task incentives. We found that participants indeed moved faster when fast decisions were promoted and decided faster when subsequent finger tapping movements had to be executed swiftly. These results were further supported by drift diffusion modelling and inspection of psychophysical kernels: Sensorimotor delays related to initiating the finger tapping sequence were shorter in fast-decision as compared to slow-decision blocks. Likewise, the decisional speed-accuracy tradeoff shifted in favor of faster decisions in fast-tapping as compared to slow-tapping blocks. These findings suggest that decisions not only impact movement characteristics, but that properties of movement impact the time taken to decide. We interpret these behavioral results in the context of embodied decision-making, whereby shared neural mechanisms may modulate decisions and movements in a joint fashion.

The dopamine circuit as a reward-taxis navigation system

Studying the brain circuits that control behavior is challenging, since in addition to their structural complexity there are continuous feedback interactions between actions and sensed inputs from the environment. It is therefore important to identify mathematical principles that can be used to develop testable hypotheses. In this study, we use ideas and concepts from systems biology to study the dopamine system, which controls learning, motivation, and movement. Using data from neuronal recordings in behavioral experiments, we developed a mathematical model for dopamine responses and the effect of dopamine on movement. We show that the dopamine system shares core functional analogies with bacterial chemotaxis. Just as chemotaxis robustly climbs chemical attractant gradients, the dopamine circuit performs ‘reward-taxis’ where the attractant is the expected value of reward. The reward-taxis mechanism provides a simple explanation for scale-invariant dopaminergic responses and for matching in free operant settings, and makes testable quantitative predictions. We propose that reward-taxis is a simple and robust navigation strategy that complements other, more goal-directed navigation mechanisms.

Rat Anterior Cingulate Cortex Continuously Signals Decision Variables in a Patch Foraging Task

In patch foraging tasks, animals must decide whether to remain with a depleting resource or to leave it in search of a potentially better source of reward. In such tasks, animals consistently follow the general predictions of optimal foraging theory (the marginal value theorem; MVT): to leave a patch when the reward rate in the current patch depletes to the average reward rate across patches. Prior studies implicate an important role for the anterior cingulate cortex (ACC) in foraging decisions based on MVT: within single trials, ACC activity increases immediately preceding foraging decisions, and across trials, these dynamics are modulated as the value of staying in the patch depletes to the average reward rate. Here, we test whether these activity patterns reflect dynamic encoding of decision-variables and whether these signals are directly involved in decision-making. We developed a leaky accumulator model based on the MVT that generates estimates of decision variables within and across trials, and tested model predictions against ACC activity recorded from male rats performing a patch foraging task. Model predicted changes in MVT decision variables closely matched rat ACC activity. Next, we pharmacologically inactivated ACC in male rats to test the contribution of these signals to decision-making. ACC inactivation had a profound effect on rats' foraging decisions and response times (RTs) yet rats still followed the MVT decision rule. These findings indicate that the ACC encodes foraging-related variables for reasons unrelated to patch-leaving decisions.SIGNIFICANCE STATEMENT The ability to make adaptive patch-foraging decisions, to remain with a depleting resource or search for better alternatives, is critical to animal well-being. Previous studies have found that anterior cingulate cortex (ACC) activity is modulated at different points in the foraging decision process, raising questions about whether the ACC guides ongoing decisions or serves a more general purpose of regulating cognitive control. To investigate the function of the ACC in foraging, the present study developed a dynamic model of behavior and neural activity, and tested model predictions using recordings and inactivation of ACC. Findings revealed that ACC continuously signals decision variables but that these signals are more likely used to monitor and regulate ongoing processes than to guide foraging decisions.

A reference-based theory of motivation and effort allocation

Motivation is key for performance in domains such as work, sport, and learning. Research has established that motivation and the willingness to invest effort generally increase as a function of reward. However, this view struggles to explain some empirical observations-for example, in the domain of sport, athletes sometimes appear to lose motivation when playing against weak opponents-this despite objective rewards being high. This and similar evidence highlight the role of subjective value in motivation and effort allocation. To capture this, here, we advance a novel theory and computational model where motivation and effort allocation arise from reference-based evaluation processes. Our proposal argues that motivation (and the ensuing willingness to exert effort) stems from subjective value, which in turns depends on one's standards about performance and on the confidence about these standards. In a series of simulations, we show that the model explains puzzling motivational dynamics and associated feelings. Crucially, the model identifies realistic standards (i.e., those matching one's own actual ability) as those more beneficial for motivation and performance. On this basis, the model establishes a normative solution to the problem of optimal allocation of effort, analogous to the optimal allocation of neural and computational resources as in efficient coding.

Eye Movements as a Window into Decision-Making

For over 100 years, eye movements have been studied and used as indicators of human sensory and cognitive functions. This review evaluates how eye movements contribute to our understanding of the processes that underlie decision-making. Eye movement metrics signify the visual and task contexts in which information is accumulated and weighed. They indicate the efficiency with which we evaluate the instructions for decision tasks, the timing and duration of decision formation, the expected reward associated with a decision, the accuracy of the decision outcome, and our ability to predict and feel confident about a decision. Because of their continuous nature, eye movements provide an exciting opportunity to probe decision processes noninvasively in real time. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Mortality Awareness: New Directions

Thinking about our own death and its salience in relation to decision making has become a fruitful area of multidisciplinary research across the breadth of psychological science. By bringing together experts from philosophy, cognitive and affective neuroscience, clinical and computational psychiatry we have attempted to set out the current state of the art and point to areas of further enquiry. One stimulus for doing this is the need to engage with policy makers who are now having to consider guidelines on suicide and assisted suicide so that they may be aware of their own as well as the wider populations' cognitive processes when confronted with the ultimate truth of mortality.

Synchronous spiking of cerebellar Purkinje cells during control of movements

SignificanceThe information that one region of the brain transmits to another is usually viewed through the lens of firing rates. However, if the output neurons could vary the timing of their spikes, then, through synchronization, they would spotlight information that may be critical for control of behavior. Here we report that, in the cerebellum, Purkinje cell populations that share a preference for error convey, to the nucleus, when to decelerate the movement, by reducing their firing rates and temporally synchronizing the remaining spikes.

Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making

Nervous systems evolved to effectively navigate the dynamics of the environment to achieve their goals. One framework used to study this fundamental problem arose in the study of learning and decision-making. In this framework, the demands of effective behavior require slow dynamics-on the scale of seconds to minutes-of networks of neurons. Here, we review the phenomena and mechanisms involved. Using vignettes from a few species and areas of the nervous system, we view neuromodulators as key substrates for temporal scaling of neuronal dynamics. Expected final online publication date for the Annual Review of Neuroscience, Volume 45 is July 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Filling the gaps: Cognitive control as a critical lens for understanding mechanisms of value-based decision-making

While often seeming to investigate rather different problems, research into value-based decision making and cognitive control have historically offered parallel insights into how people select thoughts and actions. While the former studies how people weigh costs and benefits to make a decision, the latter studies how they adjust information processing to achieve their goals. Recent work has highlighted ways in which decision-making research can inform our understanding of cognitive control. Here, we provide the complementary perspective: how cognitive control research has informed understanding of decision-making. We highlight three particular areas of research where this critical interchange has occurred: (1) how different types of goals shape the evaluation of choice options, (2) how people use control to adjust the ways they make their decisions, and (3) how people monitor decisions to inform adjustments to control at multiple levels and timescales. We show how adopting this alternate viewpoint offers new insight into the determinants of both decisions and control; provides alternative interpretations for common neuroeconomic findings; and generates fruitful directions for future research.

Reward-Dependent Selection of Feedback Gains Impacts Rapid Motor Decisions

Target reward influences motor planning strategies through modulation of movement vigor. Considering current theories of sensorimotor control suggesting that movement planning consists in selecting a goal-directed control strategy, we sought to investigate the influence of reward on feedback control. Here, we explored this question in three human reaching experiments. First, we altered the explicit reward associated with the goal target and found an overall increase in feedback gains for higher target rewards, highlighted by larger velocities, feedback responses to external loads, and background muscle activity. Then, we investigated whether the differences in target rewards across multiple goals impacted rapid motor decisions during movement. We observed idiosyncratic switching strategies dependent on both target rewards and, surprisingly, the feedback gains at perturbation onset: the more vigorous movements were less likely to switch to a new goal following perturbations. To gain further insight into a causal influence of the feedback gains on rapid motor decisions, we demonstrated that biasing the baseline activity and reflex gains by means of a background load evoked a larger proportion of target switches in the direction opposite to the background load associated with lower muscle activity. Together, our results demonstrate an impact of target reward on feedback control and highlight the competition between movement vigor and flexibility.

Evidence for a cost of time in the invigoration of isometric reaching movements

How the brain determines the vigor of goal-directed movements is a fundamental question in neuroscience. Recent evidence has suggested that vigor results from a trade-off between a cost related to movement production (cost of movement) and a cost related to our brain's tendency to temporally discount the value of future reward (cost of time). However, whether it is critical to hypothesize a cost of time to explain the vigor of basic reaching movements with intangible reward is unclear because the cost of movement may be theoretically sufficient for this purpose. Here we directly address this issue by designing an isometric reaching task whose completion can be accurate and effortless in prefixed durations. The cost of time hypothesis predicts that participants should be prone to spend energy to save time even if the task can be accomplished at virtually no motor cost. Accordingly, we found that all participants generated substantial amounts of force to invigorate task accomplishment, especially when the prefixed duration was long enough. Remarkably, the time saved by each participant was linked to their original vigor in the task and predicted by an optimal control model balancing out movement and time costs. Taken together, these results supports the existence of an idiosyncratic, cognitive cost of time that underlies the invigoration of basic isometric reaching movements.

How the value of the environment controls persistence in visual search

Classic foraging theory predicts that humans and animals aim to gain maximum reward per unit time. However, in standard instrumental conditioning tasks individuals adopt an apparently suboptimal strategy: they respond slowly when the expected value is low. This reward-related bias is often explained as reduced motivation in response to low rewards. Here we present evidence this behavior is associated with a complementary increased motivation to search the environment for alternatives. We trained monkeys to search for reward-related visual targets in environments with different values. We found that the reward-related bias scaled with environment value, was consistent with persistent searching after the target was already found, and was associated with increased exploratory gaze to objects in the environment. A novel computational model of foraging suggests that this search strategy could be adaptive in naturalistic settings where both environments and the objects within them provide partial information about hidden, uncertain rewards.

Dissociating the Impact of Movement Time and Energy Costs on Decision-Making and Action Initiation in Humans

Recent theories and data suggest that adapted behavior involves economic computations during which multiple trade-offs between reward value, accuracy requirement, energy expenditure, and elapsing time are solved so as to obtain rewards as soon as possible while spending the least possible amount of energy. However, the relative impact of movement energy and duration costs on perceptual decision-making and movement initiation is poorly understood. Here, we tested 31 healthy subjects on a perceptual decision-making task in which they executed reaching movements to report probabilistic choices. In distinct blocks of trials, the reaching duration (“Time” condition) and energy (“Effort” condition) costs were independently varied compared to a “Reference” block, while decision difficulty was maintained similar at the block level. Participants also performed a simple delayed-reaching (DR) task aimed at estimating movement initiation duration in each motor condition. Results in that DR task show that long duration movements extended reaction times (RTs) in most subjects, whereas energy-consuming movements led to mixed effects on RTs. In the decision task, about half of the subjects decreased their decision durations (DDs) in the Time condition, while the impact of energy on DDs were again mixed across subjects. Decision accuracy was overall similar across motor conditions. These results indicate that movement duration and, to a lesser extent, energy expenditure, idiosyncratically affect perceptual decision-making and action initiation. We propose that subjects who shortened their choices in the time-consuming condition of the decision task did so to limit a drop of reward rate.

How movements shape the perception of time

In order to keep up with a changing environment, mobile organisms must be capable of deciding both where and when to move. This precision necessitates a strong sense of time, as otherwise we would fail in many of our movement goals. Yet, despite this intrinsic link, only recently have researchers begun to understand how these two features interact. Primarily, two effects have been observed: movements can bias time estimates, but they can also make them more precise. Here we review this literature and propose that both effects can be explained by a Bayesian cue combination framework, in which movement itself affords the most precise representation of time, which can influence perception in either feedforward or active sensing modes.

The cost of correcting for error during sensorimotor adaptation

Learning from error is often a slow process. In machine learning, the learning rate depends on a loss function that specifies a cost for error. Here, we hypothesized that during motor learning, error carries an implicit cost for the brain because the act of correcting for error consumes time and energy. Thus, if this implicit cost could be increased, it may robustly alter how the brain learns from error. To vary the implicit cost of error, we designed a task that combined saccade adaptation with motion discrimination: movement errors resulted in corrective saccades, but those corrections took time away from acquiring information in the discrimination task. We then modulated error cost using coherence of the discrimination task and found that when error cost was large, pupil diameter increased and the brain learned more from error. However, when error cost was small, the pupil constricted and the brain learned less from the same error. Thus, during sensorimotor adaptation, the act of correcting for error carries an implicit cost for the brain. Modulating this cost affects how much the brain learns from error.

Context-sensitive valuation and learning

A variety of behavioral and neural phenomena suggest that organisms evaluate outcomes not on an absolute utility scale, but relative to some dynamic and context-sensitive reference or scale. Sometimes, as in foraging tasks, this results in sensible choices; in other situations, like choosing between options learned in different contexts, irrational choices can result. We argue that what unites and demystifies these various phenomena is that the brain's goal is not assessing utility as an end in itself, but rather comparing different options to choose the better one. In the presence of uncertainty, noise, or costly computation, adjusting options to the context can produce more accurate choices.

Movement control, decision-making, and the building of Roman roads to link them

In science, as in life, one can only hope to both inform others, and be informed by them. The commentaries associated with our book Vigor have highlighted the many ways in which the theory that we proposed can be improved. For example, there are a myriad of factors that need to be considered in a fully encompassing objective function. The neural mechanisms underlying the links between movement and decision-making have yet to be unraveled. The implications of a two-way interaction between movement and decisions at both the individual and social levels remain to be understood. The commentaries outline future questions, and encouragingly highlight the diversity of science communities that may be linked via the concept of vigor.

A role of serotonin and the insula in vigor: Tracking environmental and physiological resources

We describe a neural monitor of environmental and physiological resources that informs effort expenditure. Depending on resources and environmental stability, serotonergic and dopaminergic neuromodulations favor different behavioral controls that are organized in corticostriatal loops. This broader perspective produces some suggestions and questions that may not be covered by the foraging approach to vigor of Shadmehr and Ahmed (2020).

Control over patch encounters changes foraging behavior

Foraging is a common decision problem in natural environments. When new exploitable sites are always available, a simple optimal strategy is to leave a current site when its return falls below a single average reward rate. Here, we examined foraging in a more structured environment, with a limited number of sites that replenished at different rates and had to be revisited. When participants could choose sites, they visited fast-replenishing sites more often, left sites at higher levels of reward, and achieved a higher net reward rate. Decisions to exploit-or-leave a site were best explained with a computational model that included both the average reward rate for the environment and reward information about the unattended sites. This suggests that unattended sites influence leave decisions, in foraging environments where sites can be revisited.

•

Being able to select sites during foraging increased visits to high value sites

•

This visitation pattern was efficient, producing higher average reward rates

•

Decisions to leave a site were influenced by information about alternative sites

Walking: How visual exploration informs step choice

We rely on gaze to guide subsequent steps during walking, more so when the terrain ahead is more uncertain. New research shows that the increased visual exploration during walking as the terrain becomes more uncertain reflects our preference for accuracy over effort in step choice.

Dopamine and the interdependency of time perception and reward

Time is a fundamental dimension of our perception of the world and is therefore of critical importance to the organization of human behavior. A corpus of work - including recent optogenetic evidence - implicates striatal dopamine as a crucial factor influencing the perception of time. Another stream of literature implicates dopamine in reward and motivation processes. However, these two domains of research have remained largely separated, despite neurobiological overlap and the apothegmatic notion that "time flies when you're having fun". This article constitutes a review of the literature linking time perception and reward, including neurobiological and behavioral studies. Together, these provide compelling support for the idea that time perception and reward processing interact via a common dopaminergic mechanism.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Perceived effort affects choice of limb and reaction time of movements

The decision regarding which arm to use to perform a task reflects a complex process that can be influenced by many factors, including effort requirements of acquiring the goal. In this study, we considered a virtual reality environment in which people reached to a visual target in three-dimensional space. To vary the cost of reaching, we altered the visual feedback associated with motion of one arm but not the other. This altered the extent of motion that was required to reach, thus changing the effort required to acquire the goal. We then measured how that change in effort affected the decision regarding which arm to use, as well as the preparation time for the movement that ensued. As expected, with increased visual amplification of one arm (reduced effort to reach the goal), subjects increased the probability of choosing that arm. Surprisingly, however, the reaction times to start these movements were also reduced: despite constancy of the visual representation of the target, reaction times were shorter for movements with less effort. Thus, as the perceived effort associated with accomplishing a goal was reduced for a given limb, the decision-making process was biased toward use of that limb. Furthermore, movements that were perceived to be less effortful were performed with shorter reaction times. These results suggest that visual amplification can alter the perceived effort associated with using a limb, thus increasing frequency of use. This may provide a useful method to increase use of a limb during rehabilitation.NEW & NOTEWORTHY We report that visual amplification may serve as an effective means to alter the perceived effort associated with use of a limb. This method may provide an effective tool with which use of the affected limb can be encouraged noninvasively after neurological injury.

Precis of Vigor: Neuroeconomics of movement control

Why do we run toward people we love, but only walk toward others? Why do people in New York seem to walk faster than other cities? Why do our eyes linger longer on things we value more? There is a link between how the brain assigns value to things, and how it controls our movements. This link is an ancient one, developed through shared neural circuits that on one hand teach us how to value things, and on the other hand control the vigor with which we move. As a result, when there is damage to systems that signal reward, like dopamine and serotonin, that damage not only affects our mood and patterns of decision making, but how we move. In this book, we first ask why in principle evolution should have developed a shared system of control between valuation and vigor. We then focus on the neural basis of vigor, synthesizing results from experiments that have measured activity in various brain structures and neuromodulators, during tasks in which animals decide how patiently they should wait for reward, and how vigorously they should move to acquire it. Thus, the way we move unmasks one of our well-guarded secrets: how much we value the thing we are moving toward.

A mobile approach-avoidance task

Approach and avoidance tendencies have helped explain phenomena as diverse as addiction (Mogg, Field, & Bradley, 2005), phobia (Rinck & Becker, 2007), and intergroup discrimination (Bianchi, Carnaghi, & Shamloo, 2018; Degner, Essien, & Reichardt, 2016). When the original approach-avoidance task (AAT; Solarz, 1960) that measures these tendencies was redesigned to run on regular desktop computers, it made the task much more flexible but also sacrificed some important behavioral properties of the original task-most notably its reliance on physical distance change (Chen & Bargh, 1999). Here, we present a new, mobile version of the AAT that runs entirely on smartphones and combines the flexibility of modern tasks with the behavioral properties of the original AAT. In addition, it can easily be deployed in the field and, next to traditional reaction time measurements, includes the novel measurement of response force. In two studies, we demonstrate that the mobile AAT can reliably measure known approach-avoidance tendencies toward happy and angry faces both in the laboratory and in the field.

Neural mechanisms underlying the effects of physical fatigue on effort-based choice

Physical fatigue crucially influences our decisions to partake in effortful action. However, there is a limited understanding of how fatigue impacts effort-based decision-making at the level of brain and behavior. We use functional magnetic resonance imaging to record markers of brain activity while human participants engage in uncertain choices for prospective physical effort, before and after bouts of exertion. Using computational modeling of choice behavior we find that fatiguing exertions cause participants to increase their subjective cost of effort, compared to a baseline/rested state. We describe a mechanism by which signals related to motor cortical state in premotor cortex influence effort value computations, instantiated by insula, thereby increasing an individual's subjective valuation of prospective physical effort while fatigued. Our findings provide a neurobiological account of how information about bodily state modulates decisions to engage in physical activity.

Humans sacrifice decision-making for action execution when a demanding control of movement is required

A growing body of evidence suggests that decision-making and action execution are governed by partly overlapping operating principles. Especially, previous work proposed that a shared decision urgency/movement vigor signal, possibly computed in the basal ganglia, coordinates both deliberation and movement durations in a way that maximizes the reward rate. Recent data support one aspect of this hypothesis, indicating that the urgency level at which a decision is made influences the vigor of the movement produced to express this choice. Here we investigated whether, conversely, the motor context in which a movement is executed determines decision speed and accuracy. Twenty human subjects performed a probabilistic decision task in which perceptual choices were expressed by reaching movements toward targets whose size and distance from a starting position varied in distinct blocks of trials. We found strong evidence for an influence of the motor context on most of the subjects' decision policy, but contrary to the predictions of the "shared regulation" hypothesis, we observed that slow movements executed in the most demanding motor blocks in terms of accuracy were often preceded by faster and less accurate decisions compared with blocks of trials in which big targets allowed expression of choices with fast and inaccurate movements. These results suggest that decision-making and motor control are not regulated by one unique "invigoration" signal determining both decision urgency and action vigor, but more likely by independent, yet interacting, decision urgency and movement vigor signals.NEW & NOTEWORTHY Recent hypotheses propose that choices and movements share optimization principles derived from economy, possibly implemented by one unique context-dependent regulation signal determining both processes' speed. In the present behavioral study conducted on human subjects, we demonstrate that action properties indeed influence perceptual decision-making, but that decision duration and action vigor are actually independently set depending on the difficulty of the movement executed to report a choice.

Saccade vigor and the subjective economic value of visual stimuli

Decisions are made based on the subjective value that the brain assigns to options. However, subjective value is a mathematical construct that cannot be measured directly, but rather is inferred from choices. Recent results have demonstrated that reaction time, amplitude, and velocity of movements are modulated by reward, raising the possibility that there is a link between how the brain evaluates an option and how it controls movements toward that option. Here, we asked people to choose among risky options represented by abstract stimuli, some associated with gain (points in a game), and others with loss. From their choices we estimated the subjective value that they assigned to each stimulus. In probe trials, a single stimulus appeared at center, instructing subjects to make a saccade to a peripheral target. We found that the reaction time, peak velocity, and amplitude of the peripherally directed saccade varied roughly linearly with the subjective value that the participant had assigned to the central stimulus: reaction time was shorter, velocity was higher, and amplitude was larger for stimuli that the participant valued more. Naturally, participants differed in how much they valued a given stimulus. Remarkably, those who valued a stimulus more, as evidenced by their choices in decision trials, tended to move with shorter reaction time and greater velocity in response to that stimulus in probe trials. Overall, the reaction time of the saccade in response to a stimulus partly predicted the subjective value that the brain assigned to that stimulus.NEW & NOTEWORTHY Behavioral economics relies on subjective evaluation, an abstract quantity that cannot be measured directly but must be inferred by fitting decision models to the choice patterns. Here, we present a new approach to estimate subjective value: with nothing to fit, we show that it is possible to estimate subjective value based on movement kinematics, providing a modest ability to predict a participant's preferences without prior measurement of their choice patterns.

Foraging optimally in social neuroscience: computations and methodological considerations

Research in social neuroscience has increasingly begun to use the tools of computational neuroscience to better understand behaviour. Such approaches have proven fruitful for probing underlying neural mechanisms. However, little attention has been paid to how the structure of experimental tasks relates to real-world decisions, and the problems that brains have evolved to solve. To go significantly beyond current understanding, we must begin to use paradigms and mathematical models from behavioural ecology, which offer insights into the decisions animals must make successfully in order to survive. One highly influential theory-marginal value theorem (MVT)-precisely characterises and provides an optimal solution to a vital foraging decision that most species must make: the patch-leaving problem. Animals must decide when to leave collecting rewards in a current patch (location) and travel somewhere else. We propose that many questions posed in social neuroscience can be approached as patch-leaving problems. A richer understanding of the neural mechanisms underlying social behaviour will be obtained by using MVT. In this 'tools of the trade' article, we outline the patch-leaving problem, the computations of MVT and discuss the application to social neuroscience. Furthermore, we consider the practical challenges and offer solutions for designing paradigms probing patch leaving, both behaviourally and when using neuroimaging techniques.

Eye movements as a readout of sensorimotor decision processes

Real-world tasks, such as avoiding obstacles, require a sequence of interdependent choices to reach accurate motor actions. Yet, most studies on primate decision making involve simple one-step choices. Here we analyze motor actions to investigate how sensorimotor decisions develop over time. In a go/no-go interception task human observers (n = 42) judged whether a briefly presented moving target would pass (interceptive hand movement required) or miss (no hand movement required) a strike box while their eye and hand movements were recorded. Go/no-go decision formation had to occur within the first few hundred milliseconds to allow time-critical interception. We found that the earliest time point at which eye movements started to differentiate actions (go versus no-go) preceded hand movement onset. Moreover, eye movements were related to different stages of decision making. Whereas higher eye velocity during smooth pursuit initiation was related to more accurate interception decisions (whether or not to act), faster pursuit maintenance was associated with more accurate timing decisions (when to act). These results indicate that pursuit initiation and maintenance are continuously linked to ongoing sensorimotor decision formation.NEW & NOTEWORTHY Here we show that eye movements are a continuous indicator of decision processes underlying go/no-go actions. We link different stages of decision formation to distinct oculomotor events during open- and closed-loop smooth pursuit. Critically, the earliest time point at which eye movements differentiate actions preceded hand movement onset, suggesting shared sensorimotor processing for eye and hand movements. These results emphasize the potential of studying eye movements as a readout of cognitive processes.

Reaching decisions during ongoing movements

Neurophysiological studies suggest that when decisions are made between concrete actions, the selection process involves a competition between potential action representations in the same sensorimotor structures involved in executing those actions. However, it is unclear how such models can explain situations, often encountered during natural behavior, in which we make decisions while were are already engaged in performing an action. Does the process of deliberation characterized in classical studies of decision-making proceed the same way when subjects are deciding while already acting? In the present study, human subjects continuously tracked a target moving in the horizontal plane and were occasionally presented with a new target to which they could freely choose to switch at any time, whereupon it became the new tracked target. We found that the probability of choosing to switch increased with decreasing distance to the new target and increasing size of the new target relative to the tracked target, as well as when the direction to the new target was aligned (either toward or opposite) to the current tracking direction. However, contrary to our expectations, subjects did not choose targets that minimized the energetic costs of execution, as calculated by a biomechanical model of the arm. When the constraints of continuous tracking were removed in variants of the task involving point-to-point movements, the expected preference for lower cost choices was seen. These results are discussed in the context of current theories of nested feedback control, internal models of forward dynamics, and high-dimensional neural spaces.NEW & NOTEWORTHY Current theories of decision-making primarily address how subjects make decisions before executing selected actions. However, in our daily lives we often make decisions while already performing some action (e.g., while playing a sport or navigating through a crowd). To gain insight into how current theories can be extended to such "decide-while-acting" scenarios, we examined human decisions during continuous manual tracking and found some intriguing departures from how decisions are made in classical "decide-then-act" paradigms.