A Sensorimotor Circuit in Mouse Cortex for Visual Flow Predictions

Whole-brain mapping of basal forebrain cholinergic neurons reveals a long-range reciprocal input-output loop between distinct subtypes

Basal forebrain cholinergic neurons (BFCNs) influence cognition and emotion through specific acetylcholine release in various brain regions, including the prefrontal cortices and basolateral amygdala (BLA). Acetylcholine release is controlled by distinct BFCN subtypes, modulated by excitatory and inhibitory inputs. However, the organization of the whole-brain input-output networks of these subtypes remains unclear. Here, we identified two distinct BFCN subtypes-BFCN→ACA and BFCN→BLA-innervating the anterior cingulate cortex (ACA) and BLA, each with unique distributions, electrophysiological properties, and projection patterns. Combining rabies-virus-assisted mapping and triple-plex RNAscope hybridization, we characterized their whole-brain input networks, identifying unique excitatory and shared inhibitory inputs for these subtypes. Moreover, our results reveal a long-range reciprocal input-output loop: BFCN→ACA neurons target the isocortex, their shared excitatory-input source, whereas BFCN→BLA neurons target shared inhibitory-input sources such as the striatum and pallidum, thus enabling dynamic interactions among these BFCN subtypes. Our study deepens understanding of cholinergic modulation in cognition and emotion and provides insights into their functional interactions.

Quantification of the effect of hemodynamic occlusion in two-photon imaging of mouse cortex

The last few years have seen an explosion in the number of tools available to measure neuronal activity using fluorescence imaging (Chen et al., 2013; Feng et al., 2019; Jing et al., 2019; Sun et al., 2018; Wan et al., 2021). When performed in vivo, these measurements are invariably contaminated by hemodynamic occlusion artifacts. In widefield calcium imaging, this problem is well recognized. For two-photon imaging, however, the effects of hemodynamic occlusion have only been sparsely characterized. Here, we perform a quantification of hemodynamic occlusion effects using measurements of fluorescence changes observed with GFP expression using both widefield and two-photon imaging in mouse cortex. We find that in many instances the magnitude of signal changes attributable to hemodynamic occlusion is comparable to that observed with activity sensors. Moreover, we find that hemodynamic occlusion effects were spatially heterogeneous, both over cortical regions and across cortical depth, and exhibited a complex relationship with behavior. Thus, hemodynamic occlusion is an important caveat to consider when analyzing and interpreting not just widefield but also two-photon imaging data.

Reliable sensory processing of superficial cortical interneurons is modulated by behavioral state

GABAergic interneurons in cortical layer 1 (L1) integrate sensory and top-down inputs to modulate network activity and support learning-related plasticity. However, little is known about how sensory inputs drive L1 interneuron activity. We used two-photon calcium imaging to measure sound-evoked responses in two L1 interneuron populations expressing vasoactive intestinal peptide (VIP) or neuron-derived neurotrophic factor (NDNF) in mouse auditory cortex. We found that L1 interneurons respond to both simple and complex sounds, but their responses are highly variable across trials. Despite this variability, these interneurons respond reliably to a narrow range of stimuli, reflecting selectivity for specific spectrotemporal sound features. Response reliability was modulated by behavioral state and predicted by the activity of neighboring interneurons. These findings reveal that L1 interneurons exhibit sensory tuning and identify the modulation of response reliability as a potential mechanism by which L1 relays state-dependent cues to shape sensory representations.

Top-down modulation of sensory processing and mismatch in the mouse posterior parietal cortex

An important function of the neocortex is to compare sensory feedback stimuli with internal predictions of the outside world and evoke mismatch responses to deviations, thus allowing expectations to be updated. The mechanisms behind sensory feedback mismatch and prediction formation however remain unclear. Here we created a learned association of an auditory-tactile stimulus sequence in awake head-fixed mice, where a sound predicted an up-coming whisker stimulus and introduced mismatches by omitting or altering the whisker stimulus intensity. We showed that layer 2/3 posterior parietal cortex (PPC) neurons could report stimulus sequence mismatches, as well as display neural correlates of expectation. Inhibition of PPC-projecting secondary motor cortex (M2) neurons suppressed these correlates, along with population mismatch responses. Hence, M2 can influence sensory processing in the PPC and potentially contribute to the prediction of sensory feedback from learned relationships within sequences of sensory stimuli.

Motor and vestibular signals in the visual cortex permit the separation of self versus externally generated visual motion

Knowing whether we are moving or something in the world is moving around us is possibly the most critical sensory discrimination we need to perform. How the brain and, in particular, the visual system solves this motion-source separation problem is not known. Here, we find that motor, vestibular, and visual motion signals are used by the mouse primary visual cortex (VISp) to differentially represent the same visual flow information according to whether the head is stationary or experiencing passive versus active translation. During locomotion, we find that running suppresses running-congruent translation input and that translation signals dominate VISp activity when running and translation speed become incongruent. This cross-modal interaction between the motor and vestibular systems was found throughout the cortex, indicating that running and translation signals provide a brain-wide egocentric reference frame for computing the internally generated and actual speed of self when moving through and sensing the external world.

Layer 6 corticocortical neurons are a major route for intra- and interhemispheric feedback

The neocortex comprises anatomically discrete yet interconnected areas that are symmetrically located across the two hemispheres. Determining the logic of these macrocircuits is necessary for understanding high level brain function. Here in mice, we have mapped the areal and laminar organization of the ipsi- and contralateral cortical projection onto the primary visual, somatosensory, and motor cortices. We find that although the ipsilateral hemisphere is the primary source of cortical input, there is substantial contralateral symmetry regarding the relative contribution and areal identity of input. Laminar analysis of these input areas show that excitatory Layer 6 corticocortical cells (L6 CCs) are a major projection pathway within and between the two hemispheres. Analysis of the relative contribution of inputs from supra- (feedforward) and infragranular (feedback) layers reveals that contra-hemispheric projections reflect a dominant feedback organization compared to their ipsi-cortical counterpart. The magnitude of the interhemispheric difference in hierarchy was largest for sensory and motor projection areas compared to frontal, medial, or lateral brain areas due to a proportional increase in input from L6 neurons. L6 CCs therefore not only mediate long-range cortical communication but also reflect its inherent feedback organization.

Multimodal mismatch responses in mouse auditory cortex

Our movements result in predictable sensory feedback that is often multimodal. Based on deviations between predictions and actual sensory input, primary sensory areas of cortex have been shown to compute sensorimotor prediction errors. How prediction errors in one sensory modality influence the computation of prediction errors in another modality is still unclear. To investigate multimodal prediction errors in mouse auditory cortex, we used a virtual environment to experimentally couple running to both self-generated auditory and visual feedback. Using two-photon microscopy, we first characterized responses of layer 2/3 (L2/3) neurons to sounds, visual stimuli, and running onsets and found responses to all three stimuli. Probing responses evoked by audiomotor (AM) mismatches, we found that they closely resemble visuomotor (VM) mismatch responses in visual cortex (V1). Finally, testing for cross modal influence on AM mismatch responses by coupling both sound amplitude and visual flow speed to the speed of running, we found that AM mismatch responses were amplified when paired with concurrent VM mismatches. Our results demonstrate that multimodal and non-hierarchical interactions shape prediction error responses in cortical L2/3.

Learning to segment self-generated from externally caused optic flow through sensorimotor mismatch circuits

Efficient sensory detection requires the capacity to ignore task-irrelevant information, for example when optic flow patterns created by egomotion need to be disentangled from object perception. To investigate how this is achieved in the visual system, predictive coding with sensorimotor mismatch detection is an attractive starting point. Indeed, experimental evidence for sensorimotor mismatch signals in early visual areas exists, but it is not understood how they are integrated into cortical networks that perform input segmentation and categorization. Our model advances a biologically plausible solution by extending predictive coding models with the ability to distinguish self-generated from externally caused optic flow. We first show that a simple three neuron circuit produces experience-dependent sensorimotor mismatch responses, in agreement with calcium imaging data from mice. This microcircuit is then integrated into a neural network with two generative streams. The motor-to-visual stream consists of parallel microcircuits between motor and visual areas and learns to spatially predict optic flow resulting from self-motion. The second stream bidirectionally connects a motion-selective higher visual area (mHVA) to V1, assigning a crucial role to the abundant feedback connections to V1: the maintenance of a generative model of externally caused optic flow. In the model, area mHVA learns to segment moving objects from the background, and facilitates object categorization. Based on shared neurocomputational principles across species, the model also maps onto primate visual cortex. Our work extends Hebbian predictive coding to sensorimotor settings, in which the agent actively moves - and learns to predict the consequences of its own movements.

Is the impact of spontaneous movements on early visual cortex species specific?

Recent studies in non-human primates do not find pronounced signals related to the animal's own body movements in the responses of neurons in the visual cortex. This is notable because such pronounced signals have been widely observed in the visual cortex of mice. Here, we discuss factors that may contribute to the differences observed between species, such as state, slow neural drift, eccentricity, and changes in retinal input. The interpretation of movement-related signals in the visual cortex also exemplifies the challenge of identifying the sources of correlated variables. Dissecting these sources is central for understanding the functional roles of movement-related signals. We suggest a functional classification of the possible sources, aimed at facilitating cross-species comparative approaches to studying the neural mechanisms of vision during natural behavior.

Adolescent-like Processing of Behaviorally Salient Cues in Sensory and Prefrontal Cortices of Adult Preterm-Born Mice

Preterm birth is a leading risk factor for atypicalities in cognitive and sensory processing, but it is unclear how prematurity impacts circuits that support these functions. To address this, we trained adult mice born a day early (preterm mice) on a visual discrimination task and found that they commit more errors and fail to achieve high levels of performance. Using in vivo electrophysiology, we found that the neurons in the primary visual cortex (V1) and the V1-projecting prefrontal anterior cingulate cortex (ACC) are hyper-responsive to the reward, reminiscent of cue processing in adolescence. Moreover, the non-rewarded cue fails to robustly activate the V1 and V1-projecting ACC neurons during error trials, in contrast to prefrontal fast-spiking (FS) interneurons which show elevated error-related activity, suggesting that preterm birth impairs the function of prefrontal circuits for error monitoring. Finally, environmental enrichment, a well-established paradigm that promotes sensory maturation, failed to improve the performance of preterm mice, suggesting limited capacity of early interventions for reducing the risk of cognitive deficits after preterm birth. Altogether, our study for the first time identifies potential circuit mechanisms of cognitive atypicalities in the preterm population and highlights the vulnerability of prefrontal circuits to advanced onset of extrauterine experience.

Individualized functional magnetic resonance imaging neuromodulation enhances visuospatial perception: a proof-of-concept study

This proof-of-concept study uses individualized functional magnetic resonance imaging neuromodulation (iNM) to explore the mechanisms that enhance BOLD signals in visuospatial perception (VP) networks that are crucial for navigation. Healthy participants (n = 8) performed a VP up- and down-direction discrimination task at full and subthreshold coherence through peripheral vision, and superimposed direction through visual imagery (VI) at central space under iNM and control conditions. iNM targets individualized anatomical and functional middle- and medial-superior temporal (MST) networks that control VP. We found that iNM engaged selective exteroceptive and interoceptive attention (SEIA) and motor planning (MP) networks. Specifically, iNM increased overall: (i) area under the curve of the BOLD magnitude: 100% in VP (but decreased for weak coherences), 21-47% in VI, 26-59% in MP and 48-76% in SEIA through encoding; and (ii) classification performance for each direction, coherence and network through decoding, predicting stimuli from brain maps. Our findings, derived from encoding and decoding models, suggest that mechanisms induced by iNM are causally linked in enhancing visuospatial networks and demonstrate iNM as a feasibility treatment for low-vision patients with cortical blindness or visuospatial impairments that precede cognitive decline.This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

In vivo dual-plane 3-photon microscopy: spanning the depth of the mouse neocortex

Cortical computations arise from patterns of neuronal activity that span across all cortical layers and cell types. Three-photon excitation has extended the depth limit of in vivo imaging within the mouse brain to encompass all cortical layers. However, simultaneous three-photon imaging throughout cortical layers has yet to be demonstrated. Here, we combine non-unity magnification remote focusing with adaptive optics to achieve single-cell resolution imaging from two temporally multiplexed planes separated by up to 600 µm. This approach enables the simultaneous acquisition of neuronal activity from genetically defined cell types in any pair of cortical layers across the mouse neocortical column.

Adolescent-like Processing of Behaviorally Salient Cues in Sensory and Prefrontal Cortices of Adult Preterm-Born Mice

Preterm birth is a leading risk factor for atypicalities in cognitive and sensory processing, but it is unclear how prematurity impacts circuits that support these functions. To address this, we trained adult mice born a day early (preterm mice) on a visual discrimination task and found that they commit more errors and fail to achieve high levels of performance. Using in vivo electrophysiology , we found that the neurons in the primary visual cortex (V1) and the V1-projecting prefrontal anterior cingulate cortex (ACC) are hyper-responsive to the reward, reminiscent of cue processing in adolescence. Moreover, the non-rewarded cue fails to robustly activate the V1 and V1-projecting ACC neurons during error trials, in contrast to prefrontal fast-spiking (FS) interneurons which show elevated error-related activity, suggesting that preterm birth impairs the function of prefrontal circuits for error monitoring. Finally, environmental enrichment, a well-established paradigm that promotes sensory maturation, failed to improve the performance of preterm mice, suggesting limited capacity of early interventions for reducing the risk of cognitive deficits after preterm birth. Altogether, our study for the first time identifies potential circuit mechanisms of cognitive atypicalities in the preterm population and highlights the vulnerability of prefrontal circuits to advanced onset of extrauterine experience.

Computational model of layer 2/3 in mouse primary visual cortex explains observed visuomotor mismatch response

Activity in layer 2/3 of the mouse primary visual cortex has been shown to depend both on visual input and the mouse's locomotion. Moreover, this activity is altered by a mismatch between the observed visual flow and the predicted visual flow from locomotion. Here, I present a simple computational model that explains previously reported recordings from layer 2/3 neurons in mice. In my model, layer 2/3 encodes the velocity difference between the estimate from visual flow and the prediction from locomotion using a neural population code. Moreover, I describe a hypothesized mechanism for how the brain may carry out computations of variables encoded in population codes. This mechanism may point to a general principle for computing any mathematical function in the brain.

Spatiotemporal resonance in mouse primary visual cortex

Human primary visual cortex (V1) responds more strongly, or resonates, when exposed to ∼10, ∼15-20, and ∼40-50 Hz rhythmic flickering light. Full-field flicker also evokes the perception of hallucinatory geometric patterns, which mathematical models explain as standing-wave formations emerging from periodic forcing at resonant frequencies of the simulated neural network. However, empirical evidence for such flicker-induced standing waves in the visual cortex was missing. We recorded cortical responses to flicker in awake mice using high-spatial-resolution widefield imaging in combination with high-temporal-resolution glutamate-sensing fluorescent reporter (iGluSnFR). The temporal frequency tuning curves in the mouse V1 were similar to those observed in humans, showing a banded structure with multiple resonance peaks (8, 15, and 33 Hz). Spatially, all flicker frequencies evoked responses in V1 corresponding to retinotopic stimulus location, but some evoked additional peaks. These flicker-induced cortical patterns displayed standing-wave characteristics and matched linear wave equation solutions in an area restricted to the visual cortex. Taken together, the interaction of periodic traveling waves with cortical area boundaries leads to spatiotemporal activity patterns that may affect perception.

Wide-field calcium imaging of cortical activation and functional connectivity in externally- and internally-driven locomotion

The role of the cerebral cortex in self-initiated versus sensory-driven movements is central to understanding volitional action. Whether the differences in these two movement classes are due to specific cortical areas versus more cortex-wide engagement is debated. Using wide-field Ca2+ imaging, we compared neural dynamics during spontaneous and motorized treadmill locomotion, determining the similarities and differences in cortex-wide activation and functional connectivity (FC). During motorized locomotion, the cortex exhibits greater activation globally prior to and during locomotion starting compared to spontaneous and less during steady-state walking, during stopping, and after termination. Both conditions are characterized by FC increases in anterior secondary motor cortex (M2) nodes and decreases in all other regions. There are also cortex-wide differences; most notably, M2 decreases in FC with all other nodes during motorized stopping and after termination. Therefore, both internally- and externally-generated movements widely engage the cortex, with differences represented in cortex-wide activation and FC patterns.

Learning to use landmarks for navigation amplifies their representation in retrosplenial cortex

Visual landmarks provide powerful reference signals for efficient navigation by altering the activity of spatially tuned neurons, such as place cells, head direction cells, and grid cells. To understand the neural mechanism by which landmarks exert such strong influence, it is necessary to identify how these visual features gain spatial meaning. In this study, we characterized visual landmark representations in mouse retrosplenial cortex (RSC) using chronic two-photon imaging of the same neuronal ensembles over the course of spatial learning. We found a pronounced increase in landmark-referenced activity in RSC neurons that, once established, remained stable across days. Changing behavioral context by uncoupling treadmill motion from visual feedback systematically altered neuronal responses associated with the coherence between visual scene flow speed and self-motion. To explore potential underlying mechanisms, we modeled how burst firing, mediated by supralinear somatodendritic interactions, could efficiently mediate context- and coherence-dependent integration of landmark information. Our results show that visual encoding shifts to landmark-referenced and context-dependent codes as these cues take on spatial meaning during learning.

Prediction-error signals in anterior cingulate cortex drive task-switching

Task-switching is a fundamental cognitive ability that allows animals to update their knowledge of current rules or contexts. Detecting discrepancies between predicted and observed events is essential for this process. However, little is known about how the brain computes cognitive prediction-errors and whether neural prediction-error signals are causally related to task-switching behaviours. Here we trained mice to use a prediction-error to switch, in a single trial, between responding to the same stimuli using two distinct rules. Optogenetic silencing and un-silencing, together with widefield and two-photon calcium imaging revealed that the anterior cingulate cortex (ACC) was specifically required for this rapid task-switching, but only when it exhibited neural prediction-error signals. These prediction-error signals were projection-target dependent and were larger preceding successful behavioural transitions. An all-optical approach revealed a disinhibitory interneuron circuit required for successful prediction-error computation. These results reveal a circuit mechanism for computing prediction-errors and transitioning between distinct cognitive states.

Prefrontal Cortex subregions provide distinct visual and behavioral feedback modulation to the Primary Visual Cortex

The mammalian Prefrontal Cortex (PFC) has been suggested to modulate sensory information processing across multiple cortical regions via long-range axonal projections. These axonal projections arise from PFC subregions with unique brain-wide connectivity and functional repertoires, which may provide the architecture for modular feedback intended to shape sensory processing. Here, we used axonal tracing, axonal and somatic 2-photon calcium imaging, and chemogenetic manipulations in mice to delineate how projections from the Anterior Cingulate Cortex (ACA) and ventrolateral Orbitofrontal Cortex (ORB) of the PFC modulate sensory processing in the primary Visual Cortex (VISp) across behavioral states. Structurally, we found that ACA and ORB have distinct patterning of projections across both cortical regions and layers. ACA axons in VISp had a stronger representation of visual stimulus information than ORB axons, but both projections showed non-visual, behavior-dependent activity. ACA input to VISp enhanced the encoding of visual stimuli by VISp neurons, and modulation of visual responses scaled with arousal. On the other hand, ORB input shaped movement and arousal related modulation of VISp visual responses, but specifically reduced the encoding of high-contrast visual stimuli. Thus, ACA and ORB feedback have separable projection patterns and encode distinct visual and behavioral information, putatively providing the substrate for their unique effects on visual representations and behavioral modulation in VISp. Our results offer a refined model of cortical hierarchy and its impact on sensory information processing, whereby distinct as opposed to generalized properties of PFC projections contribute to VISp activity during discrete behavioral states.

Desegregation of neuronal predictive processing

Neural circuits construct internal 'world-models' to guide behavior. The predictive processing framework posits that neural activity signaling sensory predictions and concurrently computing prediction-errors is a signature of those internal models. Here, to understand how the brain generates predictions for complex sensorimotor signals, we investigate the emergence of high-dimensional, multi-modal predictive representations in recurrent networks. We find that robust predictive processing arises in a network with loose excitatory/inhibitory balance. Contrary to previous proposals of functionally specialized cell-types, the network exhibits desegregation of stimulus and prediction-error representations. We confirmed these model predictions by experimentally probing predictive-coding circuits using a rich stimulus-set to violate learned expectations. When constrained by data, our model further reveals and makes concrete testable experimental predictions for the distinct functional roles of excitatory and inhibitory neurons, and of neurons in different layers along a laminar hierarchy, in computing multi-modal predictions. These results together imply that in natural conditions, neural representations of internal models are highly distributed, yet structured to allow flexible readout of behaviorally-relevant information. The generality of our model advances the understanding of computation of internal models across species, by incorporating different types of predictive computations into a unified framework.

Visual hallucinations originating in the retinofugal pathway under clinical and psychedelic conditions

Psychedelics like LSD (Lysergic acid diethylamide) and psilocybin are known to modulate perceptual modalities due to the activation of mostly serotonin receptors in specific cortical (e.g., visual cortex) and subcortical (e.g., thalamus) regions of the brain. In the visual domain, these psychedelic modulations often result in peculiar disturbances of viewed objects and light and sometimes even in hallucinations of non-existent environments, objects, and creatures. Although the underlying processes are poorly understood, research conducted over the past twenty years on the subjective experience of psychedelics details theories that attempt to explain these perceptual alterations due to a disruption of communication between cortical and subcortical regions. However, rare medical conditions in the visual system like Charles Bonnet syndrome that cause perceptual distortions may shed new light on the additional importance of the retinofugal pathway in psychedelic subjective experiences. Interneurons in the retina called amacrine cells could be the first site of visual psychedelic modulation and aid in disrupting the hierarchical structure of how humans perceive visual information. This paper presents an understanding of how the retinofugal pathway communicates and modulates visual information in psychedelic and clinical conditions. Therefore, we elucidate a new theory of psychedelic modulation in the retinofugal pathway.

Predictive Processing: A Circuit Approach to Psychosis

Predictive processing is a computational framework that aims to explain how the brain processes sensory information by making predictions about the environment and minimizing prediction errors. It can also be used to explain some of the key symptoms of psychotic disorders such as schizophrenia. In recent years, substantial advances have been made in our understanding of the neuronal circuitry that underlies predictive processing in cortex. In this review, we summarize these findings and how they might relate to psychosis and to observed cell type-specific effects of antipsychotic drugs. We argue that quantifying the effects of antipsychotic drugs on specific neuronal circuit elements is a promising approach to understanding not only the mechanism of action of antipsychotic drugs but also psychosis. Finally, we outline some of the key experiments that should be done. The aims of this review are to provide an overview of the current circuit-based approaches to psychosis and to encourage further research in this direction.

Running into differences

Body movement does not significantly increase neuronal activity in the primary visual cortex of marmosets, in contrast to the effects observed in mice.

Flexible neural population dynamics govern the speed and stability of sensory encoding in mouse visual cortex

Time courses of neural responses underlie real-time sensory processing and perception. How these temporal dynamics change may be fundamental to how sensory systems adapt to different perceptual demands. By simultaneously recording from hundreds of neurons in mouse primary visual cortex, we examined neural population responses to visual stimuli at sub-second timescales, during different behavioural states. We discovered that during active behavioural states characterised by locomotion, single-neurons shift from transient to sustained response modes, facilitating rapid emergence of visual stimulus tuning. Differences in single-neuron response dynamics were associated with changes in temporal dynamics of neural correlations, including faster stabilisation of stimulus-evoked changes in the structure of correlations during locomotion. Using Factor Analysis, we examined temporal dynamics of latent population responses and discovered that trajectories of population activity make more direct transitions between baseline and stimulus-encoding neural states during locomotion. This could be partly explained by dampening of oscillatory dynamics present during stationary behavioural states. Functionally, changes in temporal response dynamics collectively enabled faster, more stable and more efficient encoding of new visual information during locomotion. These findings reveal a principle of how sensory systems adapt to perceptual demands, where flexible neural population dynamics govern the speed and stability of sensory encoding.

Cholinergic input to mouse visual cortex signals a movement state and acutely enhances layer 5 responsiveness

Acetylcholine is released in visual cortex by axonal projections from the basal forebrain. The signals conveyed by these projections and their computational significance are still unclear. Using two-photon calcium imaging in behaving mice, we show that basal forebrain cholinergic axons in the mouse visual cortex provide a binary locomotion state signal. In these axons, we found no evidence of responses to visual stimuli or visuomotor prediction errors. While optogenetic activation of cholinergic axons in visual cortex in isolation did not drive local neuronal activity, when paired with visuomotor stimuli, it resulted in layer-specific increases of neuronal activity. Responses in layer 5 neurons to both top-down and bottom-up inputs were increased in amplitude and decreased in latency, whereas those in layer 2/3 neurons remained unchanged. Using opto- and chemogenetic manipulations of cholinergic activity, we found acetylcholine to underlie the locomotion-associated decorrelation of activity between neurons in both layer 2/3 and layer 5. Our results suggest that acetylcholine augments the responsiveness of layer 5 neurons to inputs from outside of the local network, possibly enabling faster switching between internal representations during locomotion.

Sensation and expectation are embedded in mouse motor cortical activity

During behavior, the motor cortex sends copies of motor-related signals to sensory cortices. Here, we combine closed-loop behavior with large-scale physiology, projection-pattern-specific recordings, and circuit perturbations to show that neurons in mouse secondary motor cortex (M2) encode sensation and are influenced by expectation. When a movement unexpectedly produces a sound, M2 becomes dominated by sound-evoked activity. Sound responses in M2 are inherited partially from the auditory cortex and are routed back to the auditory cortex, providing a path for the reciprocal exchange of sensory-motor information during behavior. When the acoustic consequences of a movement become predictable, M2 responses to self-generated sounds are selectively gated off. These changes in single-cell responses are reflected in population dynamics, which are influenced by both sensation and expectation. Together, these findings reveal the embedding of sensory and expectation signals in motor cortical activity.

Primary auditory thalamus relays directly to cortical layer 1 interneurons

Inhibitory interneurons within cortical layer 1 (L1-INs) integrate inputs from diverse brain regions to modulate sensory processing and plasticity, but the sensory inputs that recruit these interneurons have not been identified. Here we used monosynaptic retrograde tracing and whole-cell electrophysiology to characterize the thalamic inputs onto two major subpopulations of L1-INs in the mouse auditory cortex. We find that the vast majority of auditory thalamic inputs to these L1-INs unexpectedly arise from the ventral subdivision of the medial geniculate body (MGBv), the tonotopically-organized primary auditory thalamus. Moreover, these interneurons receive robust functional monosynaptic MGBv inputs that are comparable to those recorded in the L4 excitatory pyramidal neurons. Our findings identify a direct pathway from the primary auditory thalamus to the L1-INs, suggesting that these interneurons are uniquely positioned to integrate thalamic inputs conveying precise sensory information with top-down inputs carrying information about brain states and learned associations.

A sensory-motor theory of the neocortex

Recent neurophysiological and neuroanatomical studies suggest a close interaction between sensory and motor processes across the neocortex. Here, I propose that the neocortex implements active predictive coding (APC): each cortical area estimates both latent sensory states and actions (including potentially abstract actions internal to the cortex), and the cortex as a whole predicts the consequences of actions at multiple hierarchical levels. Feedback from higher areas modulates the dynamics of state and action networks in lower areas. I show how the same APC architecture can explain (1) how we recognize an object and its parts using eye movements, (2) why perception seems stable despite eye movements, (3) how we learn compositional representations, for example, part-whole hierarchies, (4) how complex actions can be planned using simpler actions, and (5) how we form episodic memories of sensory-motor experiences and learn abstract concepts such as a family tree. I postulate a mapping of the APC model to the laminar architecture of the cortex and suggest possible roles for cortico-cortical and cortico-subcortical pathways.

Sample collection protocol for single-cell RNA sequencing of functionally identified neuronal populations in vivo

Here, we present a sample collection protocol for single-cell RNA sequencing of functionally identified neuronal populations in vivo with a virally delivered activity-dependent labeling tool (CaMPARI2). We describe steps for photoconversion in mice during the onset of computationally relevant events in a virtual reality environment, followed by removal and dissociation of the photo-labeled tissue, and separation of differentially labeled groups with fluorescence-activated cell sorting (FACS). We then detail procedures for characterizing and examining functionally relevant groups using standard bioinformatic techniques. For complete details on the use and execution of this protocol, please refer to O'Toole et al.1.

Behavioral strategy shapes activation of the Vip-Sst disinhibitory circuit in visual cortex

In complex environments, animals can adopt diverse strategies to find rewards. How distinct strategies differentially engage brain circuits is not well understood. Here, we investigate this question, focusing on the cortical Vip-Sst disinhibitory circuit between vasoactive intestinal peptide-postive (Vip) interneurons and somatostatin-positive (Sst) interneurons. We characterize the behavioral strategies used by mice during a visual change detection task. Using a dynamic logistic regression model, we find that individual mice use mixtures of a visual comparison strategy and a statistical timing strategy. Separately, mice also have periods of task engagement and disengagement. Two-photon calcium imaging shows large strategy-dependent differences in neural activity in excitatory, Sst inhibitory, and Vip inhibitory cells in response to both image changes and image omissions. In contrast, task engagement has limited effects on neural population activity. We find that the diversity of neural correlates of strategy can be understood parsimoniously as the increased activation of the Vip-Sst disinhibitory circuit during the visual comparison strategy, which facilitates task-appropriate responses.

Toward a neuroscience of natural behavior

One of the most exciting new developments in systems neuroscience is the progress being made toward neurophysiological experiments that move beyond simplified laboratory settings and address the richness of natural behavior. This is enabled by technological advances such as wireless recording in freely moving animals, automated quantification of behavior, and new methods for analyzing large data sets. Beyond new empirical methods and data, however, there is also a need for new theories and concepts to interpret that data. Such theories need to address the particular challenges of natural behavior, which often differ significantly from the scenarios studied in traditional laboratory settings. Here, we discuss some strategies for developing such novel theories and concepts and some example hypotheses being proposed.

Behind mouse eyes: The function and control of eye movements in mice

The mouse visual system has become the most popular model to study the cellular and circuit mechanisms of sensory processing. However, the importance of eye movements only started to be appreciated recently. Eye movements provide a basis for predictive sensing and deliver insights into various brain functions and dysfunctions. A plethora of knowledge on the central control of eye movements and their role in perception and behaviour arose from work on primates. However, an overview of various eye movements in mice and a comparison to primates is missing. Here, we review the eye movement types described to date in mice and compare them to those observed in primates. We discuss the central neuronal mechanisms for their generation and control. Furthermore, we review the mounting literature on eye movements in mice during head-fixed and freely moving behaviours. Finally, we highlight gaps in our understanding and suggest future directions for research.

Organization of corticocortical and thalamocortical top-down inputs in the primary visual cortex

Unified visual perception requires integration of bottom-up and top-down inputs in the primary visual cortex (V1), yet the organization of top-down inputs in V1 remains unclear. Here, we used optogenetics-assisted circuit mapping to identify how multiple top-down inputs from higher-order cortical and thalamic areas engage V1 excitatory and inhibitory neurons. Top-down inputs overlap in superficial layers yet segregate in deep layers. Inputs from the medial secondary visual cortex (V2M) and anterior cingulate cortex (ACA) converge on L6 Pyrs, whereas ventrolateral orbitofrontal cortex (ORBvl) and lateral posterior thalamic nucleus (LP) inputs are processed in parallel in Pyr-type-specific subnetworks (Pyr←ORBvl and Pyr←LP) and drive mutual inhibition between them via local interneurons. Our study deepens understanding of the top-down modulation mechanisms of visual processing and establishes that V2M and ACA inputs in L6 employ integrated processing distinct from the parallel processing of LP and ORBvl inputs in L5.

Auditory cortex conveys non-topographic sound localization signals to visual cortex

Spatiotemporally congruent sensory stimuli are fused into a unified percept. The auditory cortex (AC) sends projections to the primary visual cortex (V1), which could provide signals for binding spatially corresponding audio-visual stimuli. However, whether AC inputs in V1 encode sound location remains unknown. Using two-photon axonal calcium imaging and a speaker array, we measured the auditory spatial information transmitted from AC to layer 1 of V1. AC conveys information about the location of ipsilateral and contralateral sound sources to V1. Sound location could be accurately decoded by sampling AC axons in V1, providing a substrate for making location-specific audiovisual associations. However, AC inputs were not retinotopically arranged in V1, and audio-visual modulations of V1 neurons did not depend on the spatial congruency of the sound and light stimuli. The non-topographic sound localization signals provided by AC might allow the association of specific audiovisual spatial patterns in V1 neurons.

Antipsychotic drugs selectively decorrelate long-range interactions in deep cortical layers

Psychosis is characterized by a diminished ability of the brain to distinguish externally driven activity patterns from self-generated activity patterns. Antipsychotic drugs are a class of small molecules with relatively broad binding affinity for a variety of neuromodulator receptors that, in humans, can prevent or ameliorate psychosis. How these drugs influence the function of cortical circuits, and in particular their ability to distinguish between externally and self-generated activity patterns, is still largely unclear. To have experimental control over self-generated sensory feedback, we used a virtual reality environment in which the coupling between movement and visual feedback can be altered. We then used widefield calcium imaging to determine the cell type-specific functional effects of antipsychotic drugs in mouse dorsal cortex under different conditions of visuomotor coupling. By comparing cell type-specific activation patterns between locomotion onsets that were experimentally coupled to self-generated visual feedback and locomotion onsets that were not coupled, we show that deep cortical layers were differentially activated in these two conditions. We then show that the antipsychotic drug clozapine disrupted visuomotor integration at locomotion onsets also primarily in deep cortical layers. Given that one of the key components of visuomotor integration in cortex is long-range cortico-cortical connections, we tested whether the effect of clozapine was detectable in the correlation structure of activity patterns across dorsal cortex. We found that clozapine as well as two other antipsychotic drugs, aripiprazole and haloperidol, resulted in a strong reduction in correlations of layer 5 activity between cortical areas and impaired the spread of visuomotor prediction errors generated in visual cortex. Our results are consistent with the interpretation that a major functional effect of antipsychotic drugs is a selective alteration of long-range layer 5-mediated communication.

Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex

Primary sensory cortices respond to crossmodal stimuli-for example, auditory responses are found in primary visual cortex (V1). However, it remains unclear whether these responses reflect sensory inputs or behavioral modulation through sound-evoked body movement. We address this controversy by showing that sound-evoked activity in V1 of awake mice can be dissociated into auditory and behavioral components with distinct spatiotemporal profiles. The auditory component began at approximately 27 ms, was found in superficial and deep layers and originated from auditory cortex. Sound-evoked orofacial movements correlated with V1 neural activity starting at approximately 80-100 ms and explained auditory frequency tuning. Visual, auditory and motor activity were expressed by different laminar profiles and largely segregated subsets of neuronal populations. During simultaneous audiovisual stimulation, visual representations remained dissociable from auditory-related and motor-related activity. This three-fold dissociability of auditory, motor and visual processing is central to understanding how distinct inputs to visual cortex interact to support vision.

Responses to Pattern-Violating Visual Stimuli Evolve Differently Over Days in Somata and Distal Apical Dendrites

Scientists have long conjectured that the neocortex learns patterns in sensory data to generate top-down predictions of upcoming stimuli. In line with this conjecture, different responses to pattern-matching vs pattern-violating visual stimuli have been observed in both spiking and somatic calcium imaging data. However, it remains unknown whether these pattern-violation signals are different between the distal apical dendrites, which are heavily targeted by top-down signals, and the somata, where bottom-up information is primarily integrated. Furthermore, it is unknown how responses to pattern-violating stimuli evolve over time as an animal gains more experience with them. Here, we address these unanswered questions by analyzing responses of individual somata and dendritic branches of layer 2/3 and layer 5 pyramidal neurons tracked over multiple days in primary visual cortex of awake, behaving female and male mice. We use sequences of Gabor patches with patterns in their orientations to create pattern-matching and pattern-violating stimuli, and two-photon calcium imaging to record neuronal responses. Many neurons in both layers show large differences between their responses to pattern-matching and pattern-violating stimuli. Interestingly, these responses evolve in opposite directions in the somata and distal apical dendrites, with somata becoming less sensitive to pattern-violating stimuli and distal apical dendrites more sensitive. These differences between the somata and distal apical dendrites may be important for hierarchical computation of sensory predictions and learning, since these two compartments tend to receive bottom-up and top-down information, respectively.

Layer 1 neocortex: Gating and integrating multidimensional signals

Layer 1 (L1) of the neocortex acts as a nexus for the collection and processing of widespread information. By integrating ascending inputs with extensive top-down activity, this layer likely provides critical information regulating how the perception of sensory inputs is reconciled with expectation. This is accomplished by sorting, directing, and integrating the complex network of excitatory inputs that converge onto L1. These signals are combined with neuromodulatory afferents and gated by the wealth of inhibitory interneurons that either are embedded within L1 or send axons from other cortical layers. Together, these interactions dynamically calibrate information flow throughout the neocortex. This review will primarily focus on L1 within the primary sensory cortex and will use these insights to understand L1 in other cortical areas.

Computational components of visual predictive coding circuitry

If a full visual percept can be said to be a 'hypothesis', so too can a neural 'prediction' - although the latter addresses one particular component of image content (such as 3-dimensional organisation, the interplay between lighting and surface colour, the future trajectory of moving objects, and so on). And, because processing is hierarchical, predictions generated at one level are conveyed in a backward direction to a lower level, seeking to predict, in fact, the neural activity at that prior stage of processing, and learning from errors signalled in the opposite direction. This is the essence of 'predictive coding', at once an algorithm for information processing and a theoretical basis for the nature of operations performed by the cerebral cortex. Neural models for the implementation of predictive coding invoke specific functional classes of neuron for generating, transmitting and receiving predictions, and for producing reciprocal error signals. Also a third general class, 'precision' neurons, tasked with regulating the magnitude of error signals contingent upon the confidence placed upon the prediction, i.e., the reliability and behavioural utility of the sensory data that it predicts. So, what is the ultimate source of a 'prediction'? The answer is multifactorial: knowledge of the current environmental context and the immediate past, allied to memory and lifetime experience of the way of the world, doubtless fine-tuned by evolutionary history too. There are, in consequence, numerous potential avenues for experimenters seeking to manipulate subjects' expectation, and examine the neural signals elicited by surprising, and less surprising visual stimuli. This review focuses upon the predictive physiology of mouse and monkey visual cortex, summarising and commenting on evidence to date, and placing it in the context of the broader field. It is concluded that predictive coding has a firm grounding in basic neuroscience and that, unsurprisingly, there remains much to learn.

Predictive neural computations in the cerebellum contribute to motor planning and faster behavioral responses in larval zebrafish

The ability to predict the future based on past experience lies at the core of the brain's ability to adapt behavior. However, the neural mechanisms that participate in generating and updating predictions are not clearly understood. Further, the evolutionary antecedents and the prevalence of predictive processing among vertebrates are even less explored. Here, we show evidence of predictive processing via the involvement of cerebellar circuits in larval zebrafish. We presented stereotyped optic flow stimuli to larval zebrafish to evoke swims and discovered that lesioning the cerebellum abolished prediction-dependent modulation of swim latency. When expectations of optic flow direction did not match with reality, error signals arrive at Purkinje cells via the olivary climbing fibers, whereas granule cells and Purkinje cells encode signals of expectation. Strong neural representations of expectation correlate with faster swim responses and vice versa. In sum, our results show evidence for predictive processing in nonmammalian vertebrates with the involvement of cerebellum, an evolutionarily conserved brain structure.

Estimating Time and Rhythm by Predicting External Stimuli

In this chapter, we present recent findings from our group showing that elapsed time, interval timing, and rhythm maintenance might be achieved by the well-known ability of the brain to predict the future states of the world. The difference between predictions and actual sensory evidence is used to generate perceptual and behavioral adjustments that help subjects achieve desired behavioral goals. Concretely, we show that (1) accumulating prediction errors is a plausible strategy humans could use to determine whether a train of consecutive stimuli arrives at regular or irregular intervals. By analyzing the behavior of human and non-human primate subjects performing rhythm perception tasks, we demonstrate that (2) the ability to estimate elapsed time and internally maintain rhythms is shared across primates and humans. Neurophysiological recordings show that (3) the medial premotor cortex engages in rhythm entrainment and maintains oscillatory activity that reveals an internal metronome's spatial and temporal characteristics. Finally, we demonstrate that (4) the amplitude of gamma oscillations within this cortex increases proportionally to the total elapsed time. In conjunction with our most recent experiments, our results suggest that timing might be achieved by an internal simulation of the sensory stimuli and the motor commands that define the timing task that needs to be performed.

Silences, spikes and bursts: Three-part knot of the neural code

When a neuron breaks silence, it can emit action potentials in a number of patterns. Some responses are so sudden and intense that electrophysiologists felt the need to single them out, labelling action potentials emitted at a particularly high frequency with a metonym - bursts. Is there more to bursts than a figure of speech? After all, sudden bouts of high-frequency firing are expected to occur whenever inputs surge. The burst coding hypothesis advances that the neural code has three syllables: silences, spikes and bursts. We review evidence supporting this ternary code in terms of devoted mechanisms for burst generation, synaptic transmission and synaptic plasticity. We also review the learning and attention theories for which such a triad is beneficial.

Activity in primate visual cortex is minimally driven by spontaneous movements

Organisms process sensory information in the context of their own moving bodies, an idea referred to as embodiment. This idea is important for developmental neuroscience, robotics and systems neuroscience. The mechanisms supporting embodiment are unknown, but a manifestation could be the observation in mice of brain-wide neuromodulation, including in the primary visual cortex, driven by task-irrelevant spontaneous body movements. We tested this hypothesis in macaque monkeys (Macaca mulatta), a primate model for human vision, by simultaneously recording visual cortex activity and facial and body movements. We also sought a direct comparison using an analogous approach to those used in mouse studies. Here we found that activity in the primate visual cortex (V1, V2 and V3/V3A) was associated with the animals' own movements, but this modulation was largely explained by the impact of the movements on the retinal image, that is, by changes in visual input. These results indicate that visual cortex in primates is minimally driven by spontaneous movements and may reflect species-specific sensorimotor strategies.

Stimulus-Specific Prediction Error Neurons in Mouse Auditory Cortex

Comparing expectation with experience is an important neural computation performed throughout the brain and is a hallmark of predictive processing. Experiments that alter the sensory outcome of an animal's behavior reveal enhanced neural responses to unexpected self-generated stimuli, indicating that populations of neurons in sensory cortex may reflect prediction errors (PEs), mismatches between expectation and experience. However, enhanced neural responses to self-generated stimuli could also arise through nonpredictive mechanisms, such as the movement-based facilitation of a neuron's inherent sound responses. If sensory prediction error neurons exist in sensory cortex, it is unknown whether they manifest as general error responses, or respond with specificity to errors in distinct stimulus dimensions. To answer these questions, we trained mice of either sex to expect the outcome of a simple sound-generating behavior and recorded auditory cortex activity as mice heard either the expected sound or sounds that deviated from expectation in one of multiple distinct dimensions. Our data reveal that the auditory cortex learns to suppress responses to self-generated sounds along multiple acoustic dimensions simultaneously. We identify a distinct population of auditory cortex neurons that are not responsive to passive sounds or to the expected sound but that encode prediction errors. These prediction error neurons are abundant only in animals with a learned motor-sensory expectation, and encode one or two specific violations rather than a generic error signal. Together, these findings reveal that cortical predictions about self-generated sounds have specificity in multiple simultaneous dimensions and that cortical prediction error neurons encode specific violations from expectation.SIGNIFICANCE STATEMENT Audette et. al record neural activity in the auditory cortex while mice perform a sound-generating forelimb movement and measure neural responses to sounds that violate an animal's expectation in different ways. They find that predictions about self-generated sounds are highly specific across multiple stimulus dimensions and that a population of typically nonsound-responsive neurons respond to sounds that violate an animal's expectation in a specific way. These results identify specific prediction error (PE) signals in the mouse auditory cortex and suggest that errors may be calculated early in sensory processing.

Internal feedback in the cortical perception-action loop enables fast and accurate behavior

Animals move smoothly and reliably in unpredictable environments. Models of sensorimotor control, drawing on control theory, have assumed that sensory information from the environment leads to actions, which then act back on the environment, creating a single, unidirectional perception-action loop. However, the sensorimotor loop contains internal delays in sensory and motor pathways, which can lead to unstable control. We show here that these delays can be compensated by internal feedback signals that flow backward, from motor toward sensory areas. This internal feedback is ubiquitous in neural sensorimotor systems, and we show how internal feedback compensates internal delays. This is accomplished by filtering out self-generated and other predictable changes so that unpredicted, actionable information can be rapidly transmitted toward action by the fastest components, effectively compressing the sensory input to more efficiently use feedforward pathways: Tracts of fast, giant neurons necessarily convey less accurate signals than tracts with many smaller neurons, but they are crucial for fast and accurate behavior. We use a mathematically tractable control model to show that internal feedback has an indispensable role in achieving state estimation, localization of function (how different parts of the cortex control different parts of the body), and attention, all of which are crucial for effective sensorimotor control. This control model can explain anatomical, physiological, and behavioral observations, including motor signals in the visual cortex, heterogeneous kinetics of sensory receptors, and the presence of giant cells in the cortex of humans as well as internal feedback patterns and unexplained heterogeneity in neural systems.

How 'visual' is the visual cortex? The interactions between the visual cortex and other sensory, motivational and motor systems as enabling factors for visual perception

The definition of the visual cortex is primarily based on the evidence that lesions of this area impair visual perception. However, this does not exclude that the visual cortex may process more information than of retinal origin alone, or that other brain structures contribute to vision. Indeed, research across the past decades has shown that non-visual information, such as neural activity related to reward expectation and value, locomotion, working memory and other sensory modalities, can modulate primary visual cortical responses to retinal inputs. Nevertheless, the function of this non-visual information is poorly understood. Here we review recent evidence, coming primarily from studies in rodents, arguing that non-visual and motor effects in visual cortex play a role in visual processing itself, for instance disentangling direct auditory effects on visual cortex from effects of sound-evoked orofacial movement. These findings are placed in a broader framework casting vision in terms of predictive processing under control of frontal, reward- and motor-related systems. In contrast to the prevalent notion that vision is exclusively constructed by the visual cortical system, we propose that visual percepts are generated by a larger network-the extended visual system-spanning other sensory cortices, supramodal areas and frontal systems. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Local minimization of prediction errors drives learning of invariant object representations in a generative network model of visual perception

The ventral visual processing hierarchy of the cortex needs to fulfill at least two key functions: perceived objects must be mapped to high-level representations invariantly of the precise viewing conditions, and a generative model must be learned that allows, for instance, to fill in occluded information guided by visual experience. Here, we show how a multilayered predictive coding network can learn to recognize objects from the bottom up and to generate specific representations via a top-down pathway through a single learning rule: the local minimization of prediction errors. Trained on sequences of continuously transformed objects, neurons in the highest network area become tuned to object identity invariant of precise position, comparable to inferotemporal neurons in macaques. Drawing on this, the dynamic properties of invariant object representations reproduce experimentally observed hierarchies of timescales from low to high levels of the ventral processing stream. The predicted faster decorrelation of error-neuron activity compared to representation neurons is of relevance for the experimental search for neural correlates of prediction errors. Lastly, the generative capacity of the network is confirmed by reconstructing specific object images, robust to partial occlusion of the inputs. By learning invariance from temporal continuity within a generative model, the approach generalizes the predictive coding framework to dynamic inputs in a more biologically plausible way than self-supervised networks with non-local error-backpropagation. This was achieved simply by shifting the training paradigm to dynamic inputs, with little change in architecture and learning rule from static input-reconstructing Hebbian predictive coding networks.

Molecularly targetable cell types in mouse visual cortex have distinguishable prediction error responses

Predictive processing postulates the existence of prediction error neurons in cortex. Neurons with both negative and positive prediction error response properties have been identified in layer 2/3 of visual cortex, but whether they correspond to transcriptionally defined subpopulations is unclear. Here we used the activity-dependent, photoconvertible marker CaMPARI2 to tag neurons in layer 2/3 of mouse visual cortex during stimuli and behaviors designed to evoke prediction errors. We performed single-cell RNA-sequencing on these populations and found that previously annotated Adamts2 and Rrad layer 2/3 transcriptional cell types were enriched when photolabeling during stimuli that drive negative or positive prediction error responses, respectively. Finally, we validated these results functionally by designing artificial promoters for use in AAV vectors to express genetically encoded calcium indicators. Thus, transcriptionally distinct cell types in layer 2/3 that can be targeted using AAV vectors exhibit distinguishable negative and positive prediction error responses.

Sensation and expectation are embedded in mouse motor cortical activity

During behavior, the motor cortex sends copies of motor-related signals to sensory cortices. It remains unclear whether these corollary discharge signals strictly encode movement or whether they also encode sensory experience and expectation. Here, we combine closed-loop behavior with large-scale physiology, projection-pattern specific recordings, and circuit perturbations to show that neurons in mouse secondary motor cortex (M2) encode sensation and are influenced by expectation. When a movement unexpectedly produces a sound, M2 becomes dominated by sound-evoked activity. Sound responses in M2 are inherited partially from the auditory cortex and are routed back to the auditory cortex, providing a path for the dynamic exchange of sensory-motor information during behavior. When the acoustic consequences of a movement become predictable, M2 responses to self-generated sounds are selectively gated off. These changes in single-cell responses are reflected in population dynamics, which are influenced by both sensation and expectation. Together, these findings reveal the rich embedding of sensory and expectation signals in motor cortical activity.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.