Contribution of large scale biases in decoding of direction-of-motion from high-resolution fMRI data in human early visual cortex

Encoding of continuous perceptual choices in human early visual cortex

Introduction

Research on the neural mechanisms of perceptual decision-making has typically focused on simple categorical choices, say between two alternative motion directions. Studies on such discrete alternatives have often suggested that choices are encoded either in a motor-based or in an abstract, categorical format in regions beyond sensory cortex.

Methods

In this study, we used motion stimuli that could vary anywhere between 0° and 360° to assess how the brain encodes choices for features that span the full sensory continuum. We employed a combination of neuroimaging and encoding models based on Gaussian process regression to assess how either stimuli or choices were encoded in brain responses.

Results

We found that single-voxel tuning patterns could be used to reconstruct the trial-by-trial physical direction of motion as well as the participants' continuous choices. Importantly, these continuous choice signals were primarily observed in early visual areas. The tuning properties in this region generalized between choice encoding and stimulus encoding, even for reports that reflected pure guessing.

Discussion

We found only little information related to the decision outcome in regions beyond visual cortex, such as parietal cortex, possibly because our task did not involve differential motor preparation. This could suggest that decisions for continuous stimuli take can place already in sensory brain regions, potentially using similar mechanisms to the sensory recruitment in visual working memory.

Identifying cortical areas that underlie the transformation from 2D retinal to 3D head-centric motion signals

Accurate motion perception requires that the visual system integrate the 2D retinal motion signals received by the two eyes into a single representation of 3D motion. However, most experimental paradigms present the same stimulus to the two eyes, signaling motion limited to a 2D fronto-parallel plane. Such paradigms are unable to dissociate the representation of 3D head-centric motion signals (i.e., 3D object motion relative to the observer) from the associated 2D retinal motion signals. Here, we used stereoscopic displays to present separate motion signals to the two eyes and examined their representation in visual cortex using fMRI. Specifically, we presented random-dot motion stimuli that specified various 3D head-centric motion directions. We also presented control stimuli, which matched the motion energy of the retinal signals, but were inconsistent with any 3D motion direction. We decoded motion direction from BOLD activity using a probabilistic decoding algorithm. We found that 3D motion direction signals can be reliably decoded in three major clusters in the human visual system. Critically, in early visual cortex (V1-V3), we found no significant difference in decoding performance between stimuli specifying 3D motion directions and the control stimuli, suggesting that these areas represent the 2D retinal motion signals, rather than 3D head-centric motion itself. In voxels in and surrounding hMT and IPS0 however, decoding performance was consistently superior for stimuli that specified 3D motion directions compared to control stimuli. Our results reveal the parts of the visual processing hierarchy that are critical for the transformation of retinal into 3D head-centric motion signals and suggest a role for IPS0 in their representation, in addition to its sensitivity to 3D object structure and static depth.

Population Models, Not Analyses, of Human Neuroscience Measurements

Selectivity for many basic properties of visual stimuli, such as orientation, is thought to be organized at the scale of cortical columns, making it difficult or impossible to measure directly with noninvasive human neuroscience measurement. However, computational analyses of neuroimaging data have shown that selectivity for orientation can be recovered by considering the pattern of response across a region of cortex. This suggests that computational analyses can reveal representation encoded at a finer spatial scale than is implied by the spatial resolution limits of measurement techniques. This potentially opens up the possibility to study a much wider range of neural phenomena that are otherwise inaccessible through noninvasive measurement. However, as we review in this article, a large body of evidence suggests an alternative hypothesis to this superresolution account: that orientation information is available at the spatial scale of cortical maps and thus easily measurable at the spatial resolution of standard techniques. In fact, a population model shows that this orientation information need not even come from single-unit selectivity for orientation tuning, but instead can result from population selectivity for spatial frequency. Thus, a categorical error of interpretation can result whereby orientation selectivity can be confused with spatial frequency selectivity. This is similarly problematic for the interpretation of results from numerous studies of more complex representations and cognitive functions that have built upon the computational techniques used to reveal stimulus orientation. We suggest in this review that these interpretational ambiguities can be avoided by treating computational analyses as models of the neural processes that give rise to measurement. Building upon the modeling tradition in vision science using considerations of whether population models meet a set of core criteria is important for creating the foundation for a cumulative and replicable approach to making valid inferences from human neuroscience measurements. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Directional Visual Motion Is Represented in the Auditory and Association Cortices of Early Deaf Individuals

Individuals who are deaf since early life may show enhanced performance at some visual tasks, including discrimination of directional motion. The neural substrates of such behavioral enhancements remain difficult to identify in humans, although neural plasticity has been shown for early deaf people in the auditory and association cortices, including the primary auditory cortex (PAC) and STS region, respectively. Here, we investigated whether neural responses in auditory and association cortices of early deaf individuals are reorganized to be sensitive to directional visual motion. To capture direction-selective responses, we recorded fMRI responses frequency-tagged to the 0.1-Hz presentation of central directional (100% coherent random dot) motion persisting for 2 sec contrasted with nondirectional (0% coherent) motion for 8 sec. We found direction-selective responses in the STS region in both deaf and hearing participants, but the extent of activation in the right STS region was 5.5 times larger for deaf participants. Minimal but significant direction-selective responses were also found in the PAC of deaf participants, both at the group level and in five of six individuals. In response to stimuli presented separately in the right and left visual fields, the relative activation across the right and left hemispheres was similar in both the PAC and STS region of deaf participants. Notably, the enhanced right-hemisphere activation could support the right visual field advantage reported previously in behavioral studies. Taken together, these results show that the reorganized auditory cortices of early deaf individuals are sensitive to directional motion. Speculatively, these results suggest that auditory and association regions can be remapped to support enhanced visual performance.

Inverted Encoding Models of Human Population Response Conflate Noise and Neural Tuning Width

Channel-encoding models offer the ability to bridge different scales of neuronal measurement by interpreting population responses, typically measured with BOLD imaging in humans, as linear sums of groups of neurons (channels) tuned for visual stimulus properties. Inverting these models to form predicted channel responses from population measurements in humans seemingly offers the potential to infer neuronal tuning properties. Here, we test the ability to make inferences about neural tuning width from inverted encoding models. We examined contrast invariance of orientation selectivity in human V1 (both sexes) and found that inverting the encoding model resulted in channel response functions that became broader with lower contrast, thus apparently violating contrast invariance. Simulations showed that this broadening could be explained by contrast-invariant single-unit tuning with the measured decrease in response amplitude at lower contrast. The decrease in response lowers the signal-to-noise ratio of population responses that results in poorer population representation of orientation. Simulations further showed that increasing signal to noise makes channel response functions less sensitive to underlying neural tuning width, and in the limit of zero noise will reconstruct the channel function assumed by the model regardless of the bandwidth of single units. We conclude that our data are consistent with contrast-invariant orientation tuning in human V1. More generally, our results demonstrate that population selectivity measures obtained by encoding models can deviate substantially from the behavior of single units because they conflate neural tuning width and noise and are therefore better used to estimate the uncertainty of decoded stimulus properties.SIGNIFICANCE STATEMENT It is widely recognized that perceptual experience arises from large populations of neurons, rather than a few single units. Yet, much theory and experiment have examined links between single units and perception. Encoding models offer a way to bridge this gap by explicitly interpreting population activity as the aggregate response of many single neurons with known tuning properties. Here we use this approach to examine contrast-invariant orientation tuning of human V1. We show with experiment and modeling that due to lower signal to noise, contrast-invariant orientation tuning of single units manifests in population response functions that broaden at lower contrast, rather than remain contrast-invariant. These results highlight the need for explicit quantitative modeling when making a reverse inference from population response profiles to single-unit responses.

The impact of ultra-high field MRI on cognitive and computational neuroimaging

The ability to measure functional brain responses non-invasively with ultra high field MRI (7 T and above) represents a unique opportunity in advancing our understanding of the human brain. Compared to lower fields (3 T and below), ultra high field MRI has an increased sensitivity, which can be used to acquire functional images with greater spatial resolution, and greater specificity of the blood oxygen level dependent (BOLD) signal to the underlying neuronal responses. Together, increased resolution and specificity enable investigating brain functions at a submillimeter scale, which so far could only be done with invasive techniques. At this mesoscopic spatial scale, perception, cognition and behavior can be probed at the level of fundamental units of neural computations, such as cortical columns, cortical layers, and subcortical nuclei. This represents a unique and distinctive advantage that differentiates ultra high from lower field imaging and that can foster a tighter link between fMRI and computational modeling of neural networks. So far, functional brain mapping at submillimeter scale has focused on the processing of sensory information and on well-known systems for which extensive information is available from invasive recordings in animals. It remains an open challenge to extend this methodology to uniquely human functions and, more generally, to systems for which animal models may be problematic. To succeed, the possibility to acquire high-resolution functional data with large spatial coverage, the availability of computational models of neural processing as well as accurate biophysical modeling of neurovascular coupling at mesoscopic scale all appear necessary.

Exploring structure and function of sensory cortex with 7T MRI

In this paper, we present an overview of 7T magnetic resonance imaging (MRI) studies of the detailed function and anatomy of sensory areas of the human brain. We discuss the motivation for the studies, with particular emphasis on increasing the spatial resolution of functional MRI (fMRI) using reduced field-of-view (FOV) data acquisitions. MRI at ultra-high-field (UHF) - defined here as 7T and above - has several advantages over lower field strengths. The intrinsic signal-to-noise ratio (SNR) of images is higher at UHF, and coupled with the increased blood-oxygen-level-dependent (BOLD) signal change, this results in increased BOLD contrast-to-noise ratio (CNR), which can be exploited to improve spatial resolution or detect weaker signals. Additionally, the BOLD signal from the intra-vascular (IV) compartment is relatively diminished compared to lower field strengths. Together, these properties make 7T functional MRI an attractive proposition for high spatial specificity measures. But with the advantages come some challenges. For example, increased vulnerability to susceptibility-induced geometric distortions and signal loss in EPI acquisitions tend to be much larger. Some of these technical issues can be addressed with currently available tools and will be discussed. We highlight the key methodological considerations for high resolution functional and structural imaging at 7 T. We then present recent data using the high spatial resolution available at UHF in studies of the visual and somatosensory cortex to highlight promising developments in this area.

Spatial scale and distribution of neurovascular signals underlying decoding of orientation and eye of origin from fMRI data

Multivariate pattern analysis of functional magnetic resonance imaging (fMRI) data is widely used, yet the spatial scales and origin of neurovascular signals underlying such analyses remain unclear. We compared decoding performance for stimulus orientation and eye of origin from fMRI measurements in human visual cortex with predictions based on the columnar organization of each feature and estimated the spatial scales of patterns driving decoding. Both orientation and eye of origin could be decoded significantly above chance in early visual areas (V1-V3). Contrary to predictions based on a columnar origin of response biases, decoding performance for eye of origin in V2 and V3 was not significantly lower than that in V1, nor did decoding performance for orientation and eye of origin differ significantly. Instead, response biases for both features showed large-scale organization, evident as a radial bias for orientation, and a nasotemporal bias for eye preference. To determine whether these patterns could drive classification, we quantified the effect on classification performance of binning voxels according to visual field position. Consistent with large-scale biases driving classification, binning by polar angle yielded significantly better decoding performance for orientation than random binning in V1-V3. Similarly, binning by hemifield significantly improved decoding performance for eye of origin. Patterns of orientation and eye preference bias in V2 and V3 showed a substantial degree of spatial correlation with the corresponding patterns in V1, suggesting that response biases in these areas originate in V1. Together, these findings indicate that multivariate classification results need not reflect the underlying columnar organization of neuronal response selectivities in early visual areas.NEW & NOTEWORTHY Large-scale response biases can account for decoding of orientation and eye of origin in human early visual areas V1-V3. For eye of origin this pattern is a nasotemporal bias; for orientation it is a radial bias. Differences in decoding performance across areas and stimulus features are not well predicted by differences in columnar-scale organization of each feature. Large-scale biases in extrastriate areas are spatially correlated with those in V1, suggesting biases originate in primary visual cortex.

Motion direction biases and decoding in human visual cortex

Functional magnetic resonance imaging (fMRI) studies have relied on multivariate analysis methods to decode visual motion direction from measurements of cortical activity. Above-chance decoding has been commonly used to infer the motion-selective response properties of the underlying neural populations. Moreover, patterns of reliable response biases across voxels that underlie decoding have been interpreted to reflect maps of functional architecture. Using fMRI, we identified a direction-selective response bias in human visual cortex that: (1) predicted motion-decoding accuracy; (2) depended on the shape of the stimulus aperture rather than the absolute direction of motion, such that response amplitudes gradually decreased with distance from the stimulus aperture edge corresponding to motion origin; and 3) was present in V1, V2, V3, but not evident in MT+, explaining the higher motion-decoding accuracies reported previously in early visual cortex. These results demonstrate that fMRI-based motion decoding has little or no dependence on the underlying functional organization of motion selectivity.

Physiological measurements using ultra-high field fMRI: a review

Functional MRI (fMRI) has grown to be the neuroimaging technique of choice for investigating brain function. This topical review provides an outline of fMRI methods and applications, with a particular emphasis on the recent advances provided by ultra-high field (UHF) scanners to allow functional mapping with greater sensitivity and improved spatial specificity. A short outline of the origin of the blood oxygenation level dependent (BOLD) contrast is provided, followed by a review of BOLD fMRI methods based on gradient-echo (GE) and spin-echo (SE) contrast. Phase based fMRI measures, as well as perfusion contrast obtained with the technique of arterial spin labelling (ASL), are also discussed. An overview of 7 T based functional neuroimaging is provided, outlining the potential advances to be made and technical challenges to be addressed.

An investigation of the white matter microstructure in motion detection using diffusion MRI

One of the most widely investigated functions of the brain is vision. Whereas special attention is often paid to motion detection and its modulation by attention, comparatively still little is known about the structural background of this function. We therefore, examined the white matter microstructural background of coherent motion detection. A random-dot kinematogram paradigm was used to measure the sensitivity of healthy individuals׳ to movement coherence. The potential correlation was investigated between the motion detection threshold and the white matter microstructure as measured by high angular resolution diffusion MRI. The Track Based Spatial Statistics method was used to address this correlation and probabilistic tractography to reveal the connection between identified regions. A significant positive correlation was found between the behavioural data and the local fractional anisotropy in the posterior part of the right superior frontal gyrus, the right juxta-cortical superior parietal lobule, the left parietal white matter, the left superior temporal gyrus and the left optic radiation. Probabilistic tractography identified pathways that are highly similar to the segregated attention networks, which have a crucial role in the paradigm. This study draws attention to the structural determinant of a behavioural function.

Coarse-scale biases for spirals and orientation in human visual cortex

Multivariate decoding analyses are widely applied to functional magnetic resonance imaging (fMRI) data, but there is controversy over their interpretation. Orientation decoding in primary visual cortex (V1) reflects coarse-scale biases, including an over-representation of radial orientations. But fMRI responses to clockwise and counter-clockwise spirals can also be decoded. Because these stimuli are matched for radial orientation, while differing in local orientation, it has been argued that fine-scale columnar selectivity for orientation contributes to orientation decoding. We measured fMRI responses in human V1 to both oriented gratings and spirals. Responses to oriented gratings exhibited a complex topography, including a radial bias that was most pronounced in the peripheral representation, and a near-vertical bias that was most pronounced near the foveal representation. Responses to clockwise and counter-clockwise spirals also exhibited coarse-scale organization, at the scale of entire visual quadrants. The preference of each voxel for clockwise or counter-clockwise spirals was predicted from the preferences of that voxel for orientation and spatial position (i.e., within the retinotopic map). Our results demonstrate a bias for local stimulus orientation that has a coarse spatial scale, is robust across stimulus classes (spirals and gratings), and suffices to explain decoding from fMRI responses in V1.

On the definition of signal-to-noise ratio and contrast-to-noise ratio for FMRI data

Signal-to-noise ratio, the ratio between signal and noise, is a quantity that has been well established for MRI data but is still subject of ongoing debate and confusion when it comes to fMRI data. fMRI data are characterised by small activation fluctuations in a background of noise. Depending on how the signal of interest and the noise are identified, signal-to-noise ratio for fMRI data is reported by using many different definitions. Since each definition comes with a different scale, interpreting and comparing signal-to-noise ratio values for fMRI data can be a very challenging job. In this paper, we provide an overview of existing definitions. Further, the relationship with activation detection power is investigated. Reference tables and conversion formulae are provided to facilitate comparability between fMRI studies.

Linking brain imaging signals to visual perception

The rapid advances in brain imaging technology over the past 20 years are affording new insights into cortical processing hierarchies in the human brain. These new data provide a complementary front in seeking to understand the links between perceptual and physiological states. Here we review some of the challenges associated with incorporating brain imaging data into such "linking hypotheses," highlighting some of the considerations needed in brain imaging data acquisition and analysis. We discuss work that has sought to link human brain imaging signals to existing electrophysiological data and opened up new opportunities in studying the neural basis of complex perceptual judgments. We consider a range of approaches when using human functional magnetic resonance imaging to identify brain circuits whose activity changes in a similar manner to perceptual judgments and illustrate these approaches by discussing work that has studied the neural basis of 3D perception and perceptual learning. Finally, we describe approaches that have sought to understand the information content of brain imaging data using machine learning and work that has integrated multimodal data to overcome the limitations associated with individual brain imaging approaches. Together these approaches provide an important route in seeking to understand the links between physiological and psychological states.

Decoding working memory of stimulus contrast in early visual cortex

Most studies of the early stages of visual analysis (V1-V3) have focused on the properties of neurons that support processing of elemental features of a visual stimulus or scene, such as local contrast, orientation, or direction of motion. Recent evidence from electrophysiology and neuroimaging studies, however, suggests that early visual cortex may also play a role in retaining stimulus representations in memory for short periods. For example, fMRI responses obtained during the delay period between two presentations of an oriented visual stimulus can be used to decode the remembered stimulus orientation with multivariate pattern analysis. Here, we investigated whether orientation is a special case or if this phenomenon generalizes to working memory traces of other visual features. We found that multivariate classification of fMRI signals from human visual cortex could be used to decode the contrast of a perceived stimulus even when the mean response changes were accounted for, suggesting some consistent spatial signal for contrast in these areas. Strikingly, we found that fMRI responses also supported decoding of contrast when the stimulus had to be remembered. Furthermore, classification generalized from perceived to remembered stimuli and vice versa, implying that the corresponding pattern of responses in early visual cortex were highly consistent. In additional analyses, we show that stimulus decoding here is driven by biases depending on stimulus eccentricity. This places important constraints on the interpretation for decoding stimulus properties for which cortical processing is known to vary with eccentricity, such as contrast, color, spatial frequency, and temporal frequency.

Integration of motion responses underlying directional motion anisotropy in human early visual cortical areas

Recent imaging studies have reported directional motion biases in human visual cortex when perceiving moving random dot patterns. It has been hypothesized that these biases occur as a result of the integration of motion detector activation along the path of motion in visual cortex. In this study we investigate the nature of such motion integration with functional MRI (fMRI) using different motion stimuli. Three types of moving random dot stimuli were presented, showing either coherent motion, motion with spatial decorrelations or motion with temporal decorrelations. The results from the coherent motion stimulus reproduced the centripetal and centrifugal directional motion biases in V1, V2 and V3 as previously reported. The temporally decorrelated motion stimulus resulted in both centripetal and centrifugal biases similar to coherent motion. In contrast, the spatially decorrelated motion stimulus resulted in small directional motion biases that were only present in parts of visual cortex coding for higher eccentricities of the visual field. In combination with previous results, these findings indicate that biased motion responses in early visual cortical areas most likely depend on the spatial integration of a simultaneously activated motion detector chain.

Human sensorimotor cortex represents conflicting visuomotor mappings

Behavioral studies have shown that humans can adapt to conflicting sensorimotor mappings that cause interference after intensive training. While previous research works indicate the involvement of distinct brain regions for different types of motor learning (e.g., kinematics vs dynamics), the neural mechanisms underlying joint adaptation to conflicting mappings within the same type of perturbation (e.g., different angles of visuomotor rotation) remain unclear. To reveal the neural substrates that represent multiple sensorimotor mappings, we examined whether different mappings could be classified with multivoxel activity patterns of functional magnetic resonance imaging data. Participants simultaneously adapted to opposite rotational perturbations (+90° and - 90°) during visuomotor tracking. To dissociate differences in movement kinematics with rotation types, we used two distinct patterns of target motion and tested generalization of the classifier between different combinations of rotation and motion types. Results showed that the rotation types were classified significantly above chance using activities in the primary sensorimotor cortex and the supplementary motor area, despite no significant difference in averaged signal amplitudes within the region. In contrast, low-level sensorimotor components, including tracking error and movement speed, were best classified using activities of the early visual cortex. Our results reveal that the sensorimotor cortex represents different visuomotor mappings, which permits joint learning and switching between conflicting sensorimotor skills.