Task-related activity in human visual cortex

Neural mechanisms of resource allocation in working memory

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. We decoded higher-priority memoranda with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization between and within participants. Trial-wise variability in the magnitude of delay activity in the frontal cortex predicted differences in decoded precision between low- and high-priority items in visual cortex. These results support a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in the visual cortex according to behavioral relevance, thus identifying a neural mechanism for resource allocation.

Pupil-Linked Arousal Modulates Precision of Stimulus Representation in Cortex

Neural responses are naturally variable from one moment to the next, even when the stimulus is held constant. What factors might underlie this variability in neural population activity? We hypothesized that spontaneous fluctuations in cortical stimulus representations are created by changes in arousal state. We tested the hypothesis using a combination of fMRI, probabilistic decoding methods, and pupillometry. Human participants (20 female, 12 male) were presented with gratings of random orientation. Shortly after viewing the grating, participants reported its orientation and gave their level of confidence in this judgment. Using a probabilistic fMRI decoding technique, we quantified the precision of the stimulus representation in the visual cortex on a trial-by-trial basis. Pupil size was recorded and analyzed to index the observer's arousal state. We found that the precision of the cortical stimulus representation, reported confidence, and variability in the behavioral orientation judgments varied from trial to trial. Interestingly, these trial-by-trial changes in cortical and behavioral precision and confidence were linked to pupil size and its temporal rate of change. Specifically, when the cortical stimulus representation was more precise, the pupil dilated more strongly prior to stimulus onset and remained larger during stimulus presentation. Similarly, stronger pupil dilation during stimulus presentation was associated with higher levels of subjective confidence, a secondary measure of sensory precision, as well as improved behavioral performance. Taken together, our findings support the hypothesis that spontaneous fluctuations in arousal state modulate the fidelity of the stimulus representation in the human visual cortex, with clear consequences for behavior.

Leveraging Multi-Echo EPI to Enhance BOLD Sensitivity in Task-based Olfactory fMRI

Functional magnetic resonance imaging (fMRI) using blood-oxygenation-level-dependent (BOLD) contrast relies on gradient echo echo-planar imaging (GE-EPI) to quantify dynamic susceptibility changes associated with the hemodynamic response to neural activity. However, acquiring BOLD fMRI in human olfactory regions is particularly challenging due to their proximity to the sinuses where large susceptibility gradients induce magnetic field distortions. BOLD fMRI of the human olfactory system is further complicated by respiratory artifacts that are highly correlated with event onsets in olfactory tasks. Multi-echo EPI (ME-EPI) acquires gradient echo data at multiple echo times (TEs) during a single acquisition and can leverage signal evolution over the multiple echo times to enhance BOLD sensitivity and reduce artifactual signal contributions. In the current study, we developed a ME-EPI acquisition protocol for olfactory task-based fMRI and demonstrated significant improvement in BOLD signal sensitivity over conventional single-echo EPI (1E-EPI). The observed improvement arose from both an increase in BOLD signal changes through a T 2 * -weighted echo combination and a reduction in non-BOLD artifacts through the application of the Multi-Echo Independent Components Analysis (ME-ICA) denoising method. This study represents one of the first direct comparisons between 1E-EPI and ME-EPI in high-susceptibility regions and provides compelling evidence in favor of using ME-EPI for future task-based fMRI studies.

Brain representations of motion and position in the double-drift illusion

In the 'double-drift' illusion, local motion within a window moving in the periphery of the visual field alters the window's perceived path. The illusion is strong even when the eyes track a target whose motion matches the window so that the stimulus remains stable on the retina. This implies that the illusion involves the integration of retinal signals with non-retinal eye-movement signals. To identify where in the brain this integration occurs, we measured BOLD fMRI responses in visual cortex while subjects experienced the double-drift illusion. We then used a combination of univariate and multivariate decoding analyses to identify (1) which brain areas were sensitive to the illusion and (2) whether these brain areas contained information about the illusory stimulus trajectory. We identified a number of cortical areas that responded more strongly during the illusion than a control condition that was matched for low-level stimulus properties. Only in area hMT+ was it possible to decode the illusory trajectory. We additionally performed a number of important controls that rule out possible low-level confounds. Concurrent eye tracking confirmed that subjects accurately tracked the moving target; we were unable to decode the illusion trajectory using eye position measurements recorded during fMRI scanning, ruling out explanations based on differences in oculomotor behavior. Our results provide evidence for a perceptual representation in human visual cortex that incorporates extraretinal information.

Neural mechanisms of resource allocation in working memory

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. Higher-priority memoranda were decoded with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization. Remarkably, trialwise variability in the magnitude of delay activity in frontal cortex predicted differences in decoded precision between low and high-priority items in visual cortex. These results suggest a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in visual cortex according to behavioral relevance, thus, identifying a neural mechanism for resource allocation.

Pupil Size Is Sensitive to Low-Level Stimulus Features, Independent of Arousal-Related Modulation

Similar to a camera aperture, pupil size adjusts to the surrounding luminance. Unlike a camera, pupil size is additionally modulated both by stimulus properties and by cognitive processes, including attention and arousal, though the interdependence of these factors is unclear. We hypothesized that different stimulus properties interact to jointly modulate pupil size while remaining independent from the impact of arousal. We measured pupil responses from human observers to equiluminant stimuli during a demanding rapid serial visual presentation (RSVP) task at fixation and tested how response amplitude depends on contrast, spatial frequency, and reward level. We found that under constant luminance, unattended stimuli evoke responses that are separable from changes caused by general arousal or attention. We further uncovered a double-dissociation between task-related responses and stimulus-evoked responses, suggesting that different sources of pupil size modulation are independent of one another. Our results shed light on neural pathways underlying pupillary response.

Spoken language processing activates the primary visual cortex

Primary visual cortex (V1) is generally thought of as a low-level sensory area that primarily processes basic visual features. Although there is evidence for multisensory effects on its activity, these are typically found for the processing of simple sounds and their properties, for example spatially or temporally-congruent simple sounds. However, in congenitally blind individuals, V1 is involved in language processing, with no evidence of major changes in anatomical connectivity that could explain this seemingly drastic functional change. This is at odds with current accounts of neural plasticity, which emphasize the role of connectivity and conserved function in determining a neural tissue's role even after atypical early experiences. To reconcile what appears to be unprecedented functional reorganization with known accounts of plasticity limitations, we tested whether V1's multisensory roles include responses to spoken language in sighted individuals. Using fMRI, we found that V1 in normally sighted individuals was indeed activated by comprehensible spoken sentences as compared to an incomprehensible reversed speech control condition, and more strongly so in the left compared to the right hemisphere. Activation in V1 for language was also significant and comparable for abstract and concrete words, suggesting it was not driven by visual imagery. Last, this activation did not stem from increased attention to the auditory onset of words, nor was it correlated with attentional arousal ratings, making general attention accounts an unlikely explanation. Together these findings suggest that V1 responds to spoken language even in sighted individuals, reflecting the binding of multisensory high-level signals, potentially to predict visual input. This capability might be the basis for the strong V1 language activation observed in people born blind, re-affirming the notion that plasticity is guided by pre-existing connectivity and abilities in the typically developed brain.

Representations in human primary visual cortex drift over time

Primary sensory regions are believed to instantiate stable neural representations, yet a number of recent rodent studies suggest instead that representations drift over time. To test whether sensory representations are stable in human visual cortex, we analyzed a large longitudinal dataset of fMRI responses to images of natural scenes. We fit the fMRI responses using an image-computable encoding model and tested how well the model generalized across sessions. We found systematic changes in model fits that exhibited cumulative drift over many months. Convergent analyses pinpoint changes in neural responsivity as the source of the drift, while population-level representational dissimilarities between visual stimuli were unchanged. These observations suggest that downstream cortical areas may read-out a stable representation, even as representations within V1 exhibit drift.

Tasks and their role in visual neuroscience

Vision is widely used as a model system to gain insights into how sensory inputs are processed and interpreted by the brain. Historically, careful quantification and control of visual stimuli have served as the backbone of visual neuroscience. There has been less emphasis, however, on how an observer's task influences the processing of sensory inputs. Motivated by diverse observations of task-dependent activity in the visual system, we propose a framework for thinking about tasks, their role in sensory processing, and how we might formally incorporate tasks into our models of vision.

Engaging distributed cortical and cerebellar networks through motor execution, observation, and imagery

When we interact with the environment around us, we are sometimes active participants, making directed physical motor movements and other times only mentally engaging with our environment, taking in sensory information and internally planning our next move without directed physical movement. Traditionally, cortical motor regions and key subcortical structures such as the cerebellum have been tightly linked to motor initiation, coordination, and directed motor behavior. However, recent neuroimaging studies have noted the activation of the cerebellum and wider cortical networks specifically during various forms of motor processing, including the observations of actions and mental rehearsal of movements through motor imagery. This phenomenon of cognitive engagement of traditional motor networks raises the question of how these brain regions are involved in the initiation of movement without physical motor output. Here, we will review evidence for distributed brain network activation during motor execution, observation, and imagery in human neuroimaging studies as well as the potential for cerebellar involvement specifically in motor-related cognition. Converging evidence suggests that a common global brain network is involved in both movement execution and motor observation or imagery, with specific task-dependent shifts in these global activation patterns. We will further discuss underlying cross-species anatomical support for these cognitive motor-related functions as well as the role of cerebrocerebellar communication during action observation and motor imagery.

High-frequency neuronal signal better explains multi-phase BOLD response

Visual stimulation-evoked blood-oxygen-level dependent (BOLD) responses can exhibit more complex temporal dynamics than a simple monophasic response. For instance, BOLD responses sometimes include a phase of positive response followed by a phase of post-stimulus undershoot. Whether the BOLD response during these phases reflects the underlying neuronal signal fluctuations or is contributed by non-neuronal physiological factors remains elusive. When presenting blocks of sustained (i.e. DC) light ON-OFF stimulations to unanesthetized rats, we observed that the response following a decrease in illumination (i.e. OFF stimulation-evoked BOLD response) in the visual cortices displayed reproducible multiple phases, including an initial positive BOLD response, followed by an undershoot and then an overshoot before the next ON trial. This multi-phase BOLD response did not result from the entrainment of the periodic stimulation structure. When we measured the neural correlates of these responses, we found that the high-frequency band from the LFP power (300 - 3000 Hz, multi-unit activity (MUA)), but not the power in the gamma band (30 - 100 Hz) exhibited the same multiphasic dynamics as the BOLD signal. This study suggests that the post-stimulus phases of the BOLD response can be better explained by the high-frequency neuronal signal.

Task-related hemodynamic responses in human early visual cortex are modulated by task difficulty and behavioral performance

Early visual cortex exhibits widespread hemodynamic responses in the absence of visual stimulation, which are entrained to the timing of a task and not predicted by local spiking or local field potential. Such task-related responses (TRRs) covary with reward magnitude and physiological signatures of arousal. It is unknown, however, if TRRs change on a trial-to-trial basis according to behavioral performance and task difficulty. If so, this would suggest that TRRs reflect arousal on a trial-to-trial timescale and covary with critical task and behavioral variables. We measured functional magnetic resonance imaging blood-oxygen-level-dependent (fMRI-BOLD) responses in the early visual cortex of human observers performing an orientation discrimination task consisting of separate easy and hard runs of trials. Stimuli were presented in a small portion of one hemifield, but the fMRI response was measured in the ipsilateral hemisphere, far from the stimulus representation and focus of spatial attention. TRRs scaled in amplitude with task difficulty, behavioral accuracy, reaction time, and lapses across trials. These modulations were not explained by the influence of respiration, cardiac activity, or head movement on the fMRI signal. Similar modulations with task difficulty and behavior were observed in pupil size. These results suggest that TRRs reflect arousal and behavior on the timescale of individual trials.

When the Whole Is Less Than the Sum of Its Parts: Maximum Object Category Information and Behavioral Prediction in Multiscale Activation Patterns

Neural codes are reflected in complex neural activation patterns. Conventional electroencephalography (EEG) decoding analyses summarize activations by averaging/down-sampling signals within the analysis window. This diminishes informative fine-grained patterns. While previous studies have proposed distinct statistical features capable of capturing variability-dependent neural codes, it has been suggested that the brain could use a combination of encoding protocols not reflected in any one mathematical feature alone. To check, we combined 30 features using state-of-the-art supervised and unsupervised feature selection procedures (n = 17). Across three datasets, we compared decoding of visual object category between these 17 sets of combined features, and between combined and individual features. Object category could be robustly decoded using the combined features from all of the 17 algorithms. However, the combination of features, which were equalized in dimension to the individual features, were outperformed across most of the time points by the multiscale feature of Wavelet coefficients. Moreover, the Wavelet coefficients also explained the behavioral performance more accurately than the combined features. These results suggest that a single but multiscale encoding protocol may capture the EEG neural codes better than any combination of protocols. Our findings put new constraints on the models of neural information encoding in EEG.

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence.

State-related neural influences on fMRI connectivity estimation

The spatiotemporal structure of functional magnetic resonance imaging (fMRI) signals has provided a valuable window into the network underpinnings of human brain function and dysfunction. Although some cross-regional temporal correlation patterns (functional connectivity; FC) exhibit a high degree of stability across individuals and species, there is growing acknowledgment that measures of FC can exhibit marked changes over a range of temporal scales. Further, FC can covary with experimental task demands and ongoing neural processes linked to arousal, consciousness and perception, cognitive and affective state, and brain-body interactions. The increased recognition that such interrelated neural processes modulate FC measurements has raised both challenges and new opportunities in using FC to investigate brain function. Here, we review recent advances in the quantification of neural effects that shape fMRI FC and discuss the broad implications of these findings in the design and analysis of fMRI studies. We also discuss how a more complete understanding of the neural factors that shape FC measurements can resolve apparent inconsistencies in the literature and lead to more interpretable conclusions from fMRI studies.

Local and distributed cortical markers of effort expenditure during sustained goal pursuit

The adaptive adjustment of behavior in pursuit of desired goals is critical for survival. To accomplish this complex feat, individuals must weigh the potential benefits of a given action against time, energy, and resource costs. Here, we examine brain responses associated with willingness to exert physical effort during the sustained pursuit of desired goals. Our analyses reveal a distributed pattern of brain activity in aspects of ventral medial prefrontal cortex that tracks with trial-level variability in effort expenditure. Indicating the brain represents echoes of effort at the point of feedback, whole-brain searchlights identified signals reflecting past effort expenditure in medial and lateral prefrontal cortices, encompassing broad swaths of frontoparietal and dorsal attention networks. These data have important implications for our understanding of how the brain's valuation mechanisms contend with the complexity of real-world dynamic environments with relevance for the study of behavior across health and disease.

Associations between ghrelin and leptin and neural food cue reactivity in a fasted and sated state

Food cue exposure can trigger eating. Food cue reactivity (FCR) is a conditioned response to food cues and includes physiological responses and activation of reward-related brain areas. FCR can be affected by hunger and weight status. The appetite-regulating hormones ghrelin and leptin play a pivotal role in homeostatic as well as hedonic eating. We examined the association between ghrelin and leptin levels and neural FCR in the fasted and sated state and the association between meal-induced changes in ghrelin and neural FCR, and in how far these associations are related to BMI and HOMA-IR. Data from 109 participants from three European centers (age 50±18 y, BMI 27±5 kg/m²) who performed a food viewing task during fMRI after an overnight fast and after a standardized meal were analyzed. Blood samples were drawn prior to the viewing task in which high-caloric, low-caloric and non-food images were shown. Fasting ghrelin was positively associated with neural FCR in the inferior and superior occipital gyrus in the fasted state. This was partly attributable to BMI and HOMA-IR. These brain regions are involved in visual attention, suggesting that individuals with higher fasting ghrelin have heightened attention to food cues. Leptin was positively associated with high calorie FCR in the medial prefrontal cortex (PFC) in the fasted state and to neural FCR in the left supramarginal gyrus in the fasted versus sated state, when correcting for BMI and HOMA-IR, respectively. This PFC region is involved in assessing anticipated reward value, suggesting that for individuals with higher leptin levels high-caloric foods are more salient than low-caloric foods, but foods in general are not more salient than non-foods. There were no associations between ghrelin and leptin and neural FCR in the sated state, nor between meal-induced changes in ghrelin and neural FCR. In conclusion, we show modest associations between ghrelin and leptin and neural FCR in a relatively large sample of European adults with a broad age and BMI range. Our findings indicate that people with higher leptin levels for their weight status and people with higher ghrelin levels may be more attracted to high caloric foods when hungry. The results of the present study form a foundation for future studies to test whether food intake and (changes in) weight status can be predicted by the association between (mainly fasting) ghrelin and leptin levels and neural FCR.