Reduced choice-related activity and correlated noise accompany perceptual deficits following unilateral vestibular lesion

Visuo-vestibular heading perception: a model system to study multi-sensory decision making

Integrating noisy signals across time as well as sensory modalities, a process named multi-sensory decision making (MSDM), is an essential strategy for making more accurate and sensitive decisions in complex environments. Although this field is just emerging, recent extraordinary works from different perspectives, including computational theory, psychophysical behaviour and neurophysiology, begin to shed new light onto MSDM. In the current review, we focus on MSDM by using a model system of visuo-vestibular heading. Combining well-controlled behavioural paradigms on virtual-reality systems, single-unit recordings, causal manipulations and computational theory based on spiking activity, recent progress reveals that vestibular signals contain complex temporal dynamics in many brain regions, including unisensory, multi-sensory and sensory-motor association areas. This challenges the brain for cue integration across time and sensory modality such as optic flow which mainly contains a motion velocity signal. In addition, new evidence from the higher-level decision-related areas, mostly in the posterior and frontal/prefrontal regions, helps revise our conventional thought on how signals from different sensory modalities may be processed, converged, and moment-by-moment accumulated through neural circuits for forming a unified, optimal perceptual decision. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Temporal and spatial properties of vestibular signals for perception of self-motion

It is well recognized that the vestibular system is involved in numerous important cognitive functions, including self-motion perception, spatial orientation, locomotion, and vector-based navigation, in addition to basic reflexes, such as oculomotor or body postural control. Consistent with this rationale, vestibular signals exist broadly in the brain, including several regions of the cerebral cortex, potentially allowing tight coordination with other sensory systems to improve the accuracy and precision of perception or action during self-motion. Recent neurophysiological studies in animal models based on single-cell resolution indicate that vestibular signals exhibit complex spatiotemporal dynamics, producing challenges in identifying their exact functions and how they are integrated with other modality signals. For example, vestibular and optic flow could provide congruent and incongruent signals regarding spatial tuning functions, reference frames, and temporal dynamics. Comprehensive studies, including behavioral tasks, neural recording across sensory and sensory-motor association areas, and causal link manipulations, have provided some insights into the neural mechanisms underlying multisensory self-motion perception.

How Peripheral Vestibular Damage Affects Velocity Storage: a Causative Explanation

Velocity storage is a centrally-mediated mechanism that processes peripheral vestibular inputs. One prominent aspect of velocity storage is its effect on dynamic responses to yaw rotation. Specifically, when normal human subjects are accelerated to constant angular yaw velocity, horizontal eye movements and perceived angular velocity decay exponentially with a time constant circa 15-30 s, even though the input from the vestibular periphery decays much faster (~ 6 s). Peripheral vestibular damage causes a time constant reduction, which is useful for clinical diagnoses, but a mechanistic explanation for the relationship between vestibular damage and changes in these behavioral dynamics is lacking. It has been hypothesized that Bayesian optimization determines ideal velocity storage dynamics based on statistics of vestibular noise and experienced motion. Specifically, while a longer time constant would make the central estimate of angular head velocity closer to actual head motion, it may also result in the accumulation of neural noise which simultaneously degrades precision. Thus, the brain may balance these two effects by determining the time constant that optimizes behavior. We applied a Bayesian optimal Kalman filter to determine the ideal velocity storage time constant for unilateral damage. Predicted time constants were substantially lower than normal and similar to patients. Building on our past work showing that Bayesian optimization explains age-related changes in velocity storage, we also modeled interactions between age-related hair cell loss and peripheral damage. These results provide a plausible mechanistic explanation for changes in velocity storage after peripheral damage. Results also suggested that even after peripheral damage, noise originating in the periphery or early central processing may remain relevant in neurocomputations. Overall, our findings support the hypothesis that the brain optimizes velocity storage based on the vestibular signal-to-noise ratio.

Vestibular Precision at the Level of Perception, Eye Movements, Posture, and Neurons

Precision and accuracy are two fundamental properties of any system, including the nervous system. Reduced precision (i.e., imprecision) results from the presence of neural noise at each level of sensory, motor, and perceptual processing. This review has three objectives: (1) to show the importance of studying vestibular precision, and specifically that studying accuracy without studying precision ignores fundamental aspects of the vestibular system; (2) to synthesize key hypotheses about precision in vestibular perception, the vestibulo-ocular reflex, posture, and neurons; and (3) to show that groups of studies that are thoughts to be distinct (e.g., perceptual thresholds, subjective visual vertical variability, neuronal variability) are actually "two sides of the same coin" - because the methods used allow results to be related to the standard deviation of a Gaussian distribution describing the underlying neural noise. Vestibular precision varies with age, stimulus amplitude, stimulus frequency, body orientation, motion direction, pathology, medication, and electrical/mechanical vestibular stimulation, but does not vary with sex. The brain optimizes precision during integration of vestibular cues with visual, auditory, and/or somatosensory cues. Since a common concern with precision metrics is time required for testing, we describe approaches to optimize data collection and provide evidence that fatigue and session effects are minimal. Finally, we summarize how precision is an individual trait that is correlated with clinical outcomes in patients as well as with performance in functional tasks like balance. These findings highlight the importance of studying vestibular precision and accuracy, and that knowledge gaps remain.

Time Course of Sensory Substitution for Gravity Sensing in Visual Vertical Orientation Perception following Complete Vestibular Loss

Loss of vestibular function causes severe acute symptoms of dizziness and disorientation, yet the brain can adapt and regain near to normal locomotor and orientation function through sensory substitution. Animal studies quantifying functional recovery have yet been limited to reflexive eye movements. Here, we studied the interplay between vestibular and proprioceptive graviception in macaque monkeys trained in an earth-vertical visual orientation (subjective visual vertical; SVV) task and measured the time course of sensory substitution for gravity perception following complete bilateral vestibular loss (BVL). Graviceptive gain, defined as the ratio of perceived versus actual tilt angle, decreased to 20% immediately following labyrinthectomy, and recovered to nearly prelesion levels with a time constant of approximately three weeks of postsurgery testing. We conclude that proprioception accounts for up to 20% of gravity sensing in normal animals, and is re-weighted to substitute completely perceptual graviception after vestibular loss. We show that these results can be accounted for by an optimal sensory fusion model.

Vestibular processing during natural self-motion: implications for perception and action

How the brain computes accurate estimates of our self-motion relative to the world and our orientation relative to gravity in order to ensure accurate perception and motor control is a fundamental neuroscientific question. Recent experiments have revealed that the vestibular system encodes this information during everyday activities using pathway-specific neural representations. Furthermore, new findings have established that vestibular signals are selectively combined with extravestibular information at the earliest stages of central vestibular processing in a manner that depends on the current behavioural goal. These findings have important implications for our understanding of the brain mechanisms that ensure accurate perception and behaviour during everyday activities and for our understanding of disorders of vestibular processing.

Adaptation of spatio-temporal convergent properties in central vestibular neurons in monkeys

The spatio-temporal convergent (STC) response occurs in central vestibular cells when dynamic and static inputs are activated. The functional significance of STC behavior is not fully understood. Whether STC is a property of some specific central vestibular neurons, or whether it is a response that can be induced in any neuron at some frequencies is unknown. It is also unknown how the change in orientation of otolith polarization vector (orientation adaptation) affects STC behavior. A new complex model, that includes inputs with regular and irregular discharges from both canal and otolith afferents, was applied to experimental data to determine how many convergent inputs are sufficient to explain the STC behavior as a function of frequency and orientation adaptation. The canal-otolith and otolith-only neurons were recorded in the vestibular nuclei of three monkeys. About 42% (11/26 canal-otolith and 3/7 otolith-only) neurons showed typical STC responses at least at one frequency before orientation adaptation. After orientation adaptation in side-down head position for 2 h, some canal-otolith and otolith-only neurons altered their STC responses. Thus, STC is a property of weights of the regular and irregular vestibular afferent inputs to central vestibular neurons which appear and/or disappear based on stimulus frequency and orientation adaptation. This indicates that STC properties are more common for central vestibular neurons than previously assumed. While gravity-dependent adaptation is also critically dependent on stimulus frequency and orientation adaptation, we propose that STC behavior is also linked to the neural network responsible for localized contextual learning during gravity-dependent adaptation.

Feedback determines the structure of correlated variability in primary visual cortex

The variable responses of sensory neurons tend to be weakly correlated (spike-count correlation, r_sc). This is widely thought to reflect noise in shared afferents, in which case r_sc can limit the reliability of sensory coding. However, it could also be due to feedback from higher-order brain regions. Currently, the relative contributions of these sources are unknown. We addressed this by recording from populations of V1 neurons in macaques performing different discrimination tasks involving the same visual input. We found that the structure of r_sc (the way r_sc varied with neuronal stimulus preference) changed systematically with task instruction. Therefore, even at the earliest stage in the cortical visual hierarchy, r_sc structure during task performance primarily reflects feedback dynamics. Consequently, previous proposals for how r_sc constrains sensory processing need not apply. Furthermore, we show that correlations between the activity of single neurons and choice depend on feedback engaged by the task.

Decision-Related Activity in Macaque V2 for Fine Disparity Discrimination Is Not Compatible with Optimal Linear Readout

Fine judgments of stereoscopic depth rely mainly on relative judgments of depth (relative binocular disparity) between objects, rather than judgments of the distance to where the eyes are fixating (absolute disparity). In macaques, visual area V2 is the earliest site in the visual processing hierarchy for which neurons selective for relative disparity have been observed (Thomas et al., 2002). Here, we found that, in macaques trained to perform a fine disparity discrimination task, disparity-selective neurons in V2 were highly selective for the task, and their activity correlated with the animals' perceptual decisions (unexplained by the stimulus). This may partially explain similar correlations reported in downstream areas. Although compatible with a perceptual role of these neurons for the task, the interpretation of such decision-related activity is complicated by the effects of interneuronal "noise" correlations between sensory neurons. Recent work has developed simple predictions to differentiate decoding schemes (Pitkow et al., 2015) without needing measures of noise correlations, and found that data from early sensory areas were compatible with optimal linear readout of populations with information-limiting correlations. In contrast, our data here deviated significantly from these predictions. We additionally tested this prediction for previously reported results of decision-related activity in V2 for a related task, coarse disparity discrimination (Nienborg and Cumming, 2006), thought to rely on absolute disparity. Although these data followed the predicted pattern, they violated the prediction quantitatively. This suggests that optimal linear decoding of sensory signals is not generally a good predictor of behavior in simple perceptual tasks.

SIGNIFICANCE STATEMENT

Activity in sensory neurons that correlates with an animal's decision is widely believed to provide insights into how the brain uses information from sensory neurons. Recent theoretical work developed simple predictions to differentiate decoding schemes, and found support for optimal linear readout of early sensory populations with information-limiting correlations. Here, we observed decision-related activity for neurons in visual area V2 of macaques performing fine disparity discrimination, as yet the earliest site for this task. These findings, and previously reported results from V2 in a different task, deviated from the predictions for optimal linear readout of a population with information-limiting correlations. Our results suggest that optimal linear decoding of early sensory information is not a general decoding strategy used by the brain.

Inference in the Brain: Statistics Flowing in Redundant Population Codes

It is widely believed that the brain performs approximate probabilistic inference to estimate causal variables in the world from ambiguous sensory data. To understand these computations, we need to analyze how information is represented and transformed by the actions of nonlinear recurrent neural networks. We propose that these probabilistic computations function by a message-passing algorithm operating at the level of redundant neural populations. To explain this framework, we review its underlying concepts, including graphical models, sufficient statistics, and message-passing, and then describe how these concepts could be implemented by recurrently connected probabilistic population codes. The relevant information flow in these networks will be most interpretable at the population level, particularly for redundant neural codes. We therefore outline a general approach to identify the essential features of a neural message-passing algorithm. Finally, we argue that to reveal the most important aspects of these neural computations, we must study large-scale activity patterns during moderately complex, naturalistic behaviors.

Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance

To go beyond qualitative models of the biological substrate of object recognition, we ask: can a single ventral stream neuronal linking hypothesis quantitatively account for core object recognition performance over a broad range of tasks? We measured human performance in 64 object recognition tests using thousands of challenging images that explore shape similarity and identity preserving object variation. We then used multielectrode arrays to measure neuronal population responses to those same images in visual areas V4 and inferior temporal (IT) cortex of monkeys and simulated V1 population responses. We tested leading candidate linking hypotheses and control hypotheses, each postulating how ventral stream neuronal responses underlie object recognition behavior. Specifically, for each hypothesis, we computed the predicted performance on the 64 tests and compared it with the measured pattern of human performance. All tested hypotheses based on low- and mid-level visually evoked activity (pixels, V1, and V4) were very poor predictors of the human behavioral pattern. However, simple learned weighted sums of distributed average IT firing rates exactly predicted the behavioral pattern. More elaborate linking hypotheses relying on IT trial-by-trial correlational structure, finer IT temporal codes, or ones that strictly respect the known spatial substructures of IT ("face patches") did not improve predictive power. Although these results do not reject those more elaborate hypotheses, they suggest a simple, sufficient quantitative model: each object recognition task is learned from the spatially distributed mean firing rates (100 ms) of ∼60,000 IT neurons and is executed as a simple weighted sum of those firing rates. Significance statement: We sought to go beyond qualitative models of visual object recognition and determine whether a single neuronal linking hypothesis can quantitatively account for core object recognition behavior. To achieve this, we designed a database of images for evaluating object recognition performance. We used multielectrode arrays to characterize hundreds of neurons in the visual ventral stream of nonhuman primates and measured the object recognition performance of >100 human observers. Remarkably, we found that simple learned weighted sums of firing rates of neurons in monkey inferior temporal (IT) cortex accurately predicted human performance. Although previous work led us to expect that IT would outperform V4, we were surprised by the quantitative precision with which simple IT-based linking hypotheses accounted for human behavior.

Neuronal thresholds and choice-related activity of otolith afferent fibers during heading perception

How activity of sensory neurons leads to perceptual decisions remains a challenge to understand. Correlations between choices and single neuron firing rates have been found early in vestibular processing, in the brainstem and cerebellum. To investigate the origins of choice-related activity, we have recorded from otolith afferent fibers while animals performed a fine heading discrimination task. We find that afferent fibers have similar discrimination thresholds as central cells, and the most sensitive fibers have thresholds that are only twofold or threefold greater than perceptual thresholds. Unlike brainstem and cerebellar nuclei neurons, spike counts from afferent fibers do not exhibit trial-by-trial correlations with perceptual decisions. This finding may reflect the fact that otolith afferent responses are poorly suited for driving heading perception because they fail to discriminate self-motion from changes in orientation relative to gravity. Alternatively, if choice probabilities reflect top-down inference signals, they are not relayed to the vestibular periphery.

Responses of non-eye movement central vestibular neurons to sinusoidal horizontal translation in compensated macaques after unilateral labyrinthectomy

After vestibular labyrinth injury, behavioral deficits partially recover through the process of vestibular compensation. The present study was performed to improve our understanding of the physiology of the macaque vestibular system in the compensated state (>7 wk) after unilateral labyrinthectomy (UL). Three groups of vestibular nucleus neurons were included: pre-UL control neurons, neurons ipsilateral to the lesion, and neurons contralateral to the lesion. The firing responses of neurons sensitive to linear acceleration in the horizontal plane were recorded during sinusoidal horizontal translation directed along six different orientations (30° apart) at 0.5 Hz and 0.2 g peak acceleration (196 cm/s(2)). This data defined the vector of best response for each neuron in the horizontal plane, along which sensitivity, symmetry, detection threshold, and variability of firing were determined. Additionally, the responses of the same cells to translation over a series of frequencies (0.25-5.0 Hz) either in the interaural or naso-occipital orientation were obtained to define the frequency response characteristics in each group. We found a decrease in sensitivity, increase in threshold, and alteration in orientation of best responses in the vestibular nuclei after UL. Additionally, the phase relationship of the best neural response to translational stimulation changed with UL. The symmetry of individual neuron responses in the excitatory and inhibitory directions was unchanged by UL. Bilateral central utricular neurons still demonstrated two-dimension tuning after UL, consistent with spatio-temporal convergence from a single vestibular end-organ. These neuronal data correlate with known behavioral deficits after unilateral vestibular compromise.

Long-term deficits in motion detection thresholds and spike count variability after unilateral vestibular lesion

The vestibular system operates in a push-pull fashion using signals from both labyrinths and an intricate bilateral organization. Unilateral vestibular lesions cause well-characterized motor deficits that are partially compensated over time and whose neural correlates have been traced in the mean response modulation of vestibular nuclei cells. Here we compare both response gains and neural detection thresholds of vestibular nuclei and semicircular canal afferent neurons in intact vs. unilateral-lesioned macaques using three-dimensional rotation and translation stimuli. We found increased stimulus-driven spike count variability and detection thresholds in semicircular canal afferents, although mean responses were unchanged, after contralateral labyrinth lesion. Analysis of trial-by-trial spike count correlations of a limited number of simultaneously recorded pairs of canal afferents suggests increased noise correlations after lesion. In addition, we also found persistent, chronic deficits in rotation detection thresholds of vestibular nuclei neurons, which were larger in the ipsilesional than the contralesional brain stem. These deficits, which persisted several months after lesion, were due to lower rotational response gains, whereas spike count variability was similar in intact and lesioned animals. In contrast to persistent deficits in rotation threshold, translation detection thresholds were not different from those in intact animals. These findings suggest that, after compensation, a single labyrinth is sufficient to recover motion sensitivity and normal thresholds for the otolith, but not the semicircular canal, system.

Decision-related activity in sensory neurons may depend on the columnar architecture of cerebral cortex

Many studies have reported correlations between the activity of sensory neurons and animals' judgments in discrimination tasks. Here, we suggest that such neuron-behavior correlations may require a cortical map for the task relevant features. This would explain why studies using discrimination tasks based on disparity in area V1 have not found these correlations: V1 contains no map for disparity. This scheme predicts that activity of V1 neurons correlates with decisions in an orientation-discrimination task. To test this prediction, we trained two macaque monkeys in a coarse orientation discrimination task using band-pass-filtered dynamic noise. The two orientations were always 90° apart and task difficulty was controlled by varying the orientation bandwidth of the filter. While the trained animals performed this task, we recorded from orientation-selective V1 neurons (n = 82, n = 31 for Monkey 1, n = 51 for Monkey 2). For both monkeys, we observed significant correlation (quantified as "choice probabilities") of the V1 activity with the monkeys' perceptual judgments (mean choice probability 0.54, p = 10(-5)). In one of these animals, we had previously measured choice probabilities in a disparity discrimination task in V1, which had been at chance (0.49, not significantly different from 0.5). The choice probabilities in this monkey for the orientation discrimination task were significantly larger than those for the disparity discrimination task (p = 0.032). These results are predicted by our suggestion that choice probabilities are only observed for cortical sensory neurons that are organized in maps for the task-relevant feature.