Encoding model of temporal processing in human visual cortex

Temporal dynamics of short-term neural adaptation across human visual cortex

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses between lower and higher visual brain areas and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.

Rethinking simultaneous suppression in visual cortex via compressive spatiotemporal population receptive fields

When multiple visual stimuli are presented simultaneously in the receptive field, the neural response is suppressed compared to presenting the same stimuli sequentially. The prevailing hypothesis suggests that this suppression is due to competition among multiple stimuli for limited resources within receptive fields, governed by task demands. However, it is unknown how stimulus-driven computations may give rise to simultaneous suppression. Using fMRI, we find simultaneous suppression in single voxels, which varies with both stimulus size and timing, and progressively increases up the visual hierarchy. Using population receptive field (pRF) models, we find that compressive spatiotemporal summation rather than compressive spatial summation predicts simultaneous suppression, and that increased simultaneous suppression is linked to larger pRF sizes and stronger compressive nonlinearities. These results necessitate a rethinking of simultaneous suppression as the outcome of stimulus-driven compressive spatiotemporal computations within pRFs, and open new opportunities to study visual processing capacity across space and time.

Intracortical recordings reveal vision-to-action cortical gradients driving human exogenous attention

Exogenous attention, the process that makes external salient stimuli pop-out of a visual scene, is essential for survival. How attention-capturing events modulate human brain processing remains unclear. Here we show how the psychological construct of exogenous attention gradually emerges over large-scale gradients in the human cortex, by analyzing activity from 1,403 intracortical contacts implanted in 28 individuals, while they performed an exogenous attention task. The timing, location and task-relevance of attentional events defined a spatiotemporal gradient of three neural clusters, which mapped onto cortical gradients and presented a hierarchy of timescales. Visual attributes modulated neural activity at one end of the gradient, while at the other end it reflected the upcoming response timing, with attentional effects occurring at the intersection of visual and response signals. These findings challenge multi-step models of attention, and suggest that frontoparietal networks, which process sequential stimuli as separate events sharing the same location, drive exogenous attention phenomena such as inhibition of return.

Temporal dynamics of short-term neural adaptation across human visual cortex

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses across the human visual hierarchy and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.

Characterizing Spatiotemporal Population Receptive Fields in Human Visual Cortex with fMRI

The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time-varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants (both females and males). We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (1) from early to later areas within a visual stream, spatial and temporal windows of pRFs progressively increase in size and show greater compressive nonlinearities, (2) later visual areas show diverging spatial and temporal windows across streams, and (3) within early visual areas (V1-V3), both spatial and temporal windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses using fMRI.

Dyslexia Due to Visual Impairments

Reading involves many different abilities that are necessary or sufficient conditions for fluent and flawless reading. The absence of one necessary or of all sufficient conditions is a cause of dyslexia. The present study investigates whether too short fixation times and an impaired ability to recognize a string of letters simultaneously are causes of dyslexia. The frequency and types of reading mistakes were investigated in a tachistoscopic pseudoword experiment with 100 children with dyslexia to test the impact of too short fixation times and the attempts of children with dyslexia to recognize more letters simultaneously than they can when reading pseudowords. The experiment demonstrates that all types of reading mistakes disappear when the fixation time increases and/or the number of letters that the children try to recognize simultaneously is reduced. The results cannot be interpreted as being due to altered visual crowding, impaired attention, or impaired phonological awareness, but can be regarded as an effect of impaired temporal summation and a dysfunction in the ventral stream of the visual system.

Polar angle asymmetries in visual perception and neural architecture

Human visual performance changes with visual field location. It is best at the center of gaze and declines with eccentricity, and also varies markedly with polar angle. These perceptual polar angle asymmetries are linked to asymmetries in the organization of the visual system. We review and integrate research quantifying how performance changes with visual field location and how this relates to neural organization at multiple stages of the visual system. We first briefly review how performance varies with eccentricity and the neural foundations of this effect. We then focus on perceptual polar angle asymmetries and their neural foundations. Characterizing perceptual and neural variations across and around the visual field contributes to our understanding of how the brain translates visual signals into neural representations which form the basis of visual perception.

Characterizing spatiotemporal population receptive fields in human visual cortex with fMRI

The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants. We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (i) from early to later areas within a visual stream, spatial and temporal integration windows of pRFs progressively increase in size and show greater compressive nonlinearities, (ii) later visual areas show diverging spatial and temporal integration windows across streams, and (iii) within early visual areas (V1-V3), both spatial and temporal integration windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses in the human brain using fMRI.

What has vision science taught us about functional MRI?

In the domain of human neuroimaging, much attention has been paid to the question of whether and how the development of functional magnetic resonance imaging (fMRI) has advanced our scientific knowledge of the human brain. However, the opposite question is also important; how has our knowledge of the visual system advanced our understanding of fMRI? Here, we discuss how and why scientific knowledge about the human and animal visual system has been used to answer fundamental questions about fMRI as a brain measurement tool and how these answers have contributed to scientific discoveries beyond vision science.

Luxotonic signals in human prefrontal cortex as a possible substrate for effects of light on mood and cognition

Studies with experimental animals have revealed a mood-regulating neural pathway linking intrinsically photosensitive retinal ganglion cells (ipRGCs) and the prefrontal cortex (PFC), involved in the pathophysiology of mood disorders. Since humans also have light-intensity-encoding ipRGCs, we asked whether a similar pathway exists in humans. Here, functional MRI was used to identify PFC regions and other areas exhibiting light-intensity-dependent signals. We report 26 human brain regions having activation that either monotonically decreases or monotonically increases with light intensity. Luxotonic-related activation occurred across the cerebral cortex, in diverse subcortical structures, and in the cerebellum, encompassing regions with functions related to visual image formation, motor control, cognition, and emotion. Light suppressed PFC activation, which monotonically decreased with increasing light intensity. The sustained time course of light-evoked PFC responses and their susceptibility to prior light exposure resembled those of ipRGCs. These findings offer a functional link between light exposure and PFC-mediated cognitive and affective phenomena.

Visual timing-tuned responses in human association cortices and response dynamics in early visual cortex

Quantifying the timing (duration and frequency) of brief visual events is vital to human perception, multisensory integration and action planning. Tuned neural responses to visual event timing have been found in association cortices, in areas implicated in these processes. Here we ask how these timing-tuned responses are related to the responses of early visual cortex, which monotonically increase with event duration and frequency. Using 7-Tesla functional magnetic resonance imaging and neural model-based analyses, we find a gradual transition from monotonically increasing to timing-tuned neural responses beginning in the medial temporal area (MT/V5). Therefore, across successive stages of visual processing, timing-tuned response components gradually become dominant over inherent sensory response modulation by event timing. This additional timing-tuned response component is independent of retinotopic location. We propose that this hierarchical emergence of timing-tuned responses from sensory processing areas quantifies sensory event timing while abstracting temporal representations from spatial properties of their inputs.

Early visual cortical responses increase with event duration and frequency, while later timing-tuned responses quantify event timing. Here, the authors show timing tuning gradually emerges up the visual hierarchy, and separates temporal and spatial event features.

Trial-by-trial predictions of subjective time from human brain activity

Human experience of time exhibits systematic, context-dependent deviations from clock time; for example, time is experienced differently at work than on holiday. Here we test the proposal that differences from clock time in subjective experience of time arise because time estimates are constructed by accumulating the same quantity that guides perception: salient events. Healthy human participants watched naturalistic, silent videos of up to 24 seconds in duration and estimated their duration while fMRI was acquired. We were able to reconstruct trial-by-trial biases in participants’ duration reports, which reflect subjective experience of duration, purely from salient events in their visual cortex BOLD activity. By contrast, salient events in neither of two control regions–auditory and somatosensory cortex–were predictive of duration biases. These results held despite being able to (trivially) predict clock time from all three brain areas. Our results reveal that the information arising during perceptual processing of a dynamic environment provides a sufficient basis for reconstructing human subjective time duration.

Our perception of time isn’t like a clock; it varies depending on other aspects of experience, such as what we see and hear in that moment. Previous studies have shown that differences in simple features, such as an image being larger or smaller, or brighter or dimmer, can change how we perceive time for those experiences. But in everyday life, the properties of these simple features can change frequently, presenting a challenge to understanding real-world time perception based on simple lab experiments. To overcome this problem, we developed a computational model of human time perception based on tracking changes in neural activity across brain regions involved in sensory processing (using non-invasive brain imaging). By measuring changes in brain activity patterns across these regions, our approach accommodates the different and changing feature combinations present in natural scenarios, such as walking on a busy street. Our model reproduces people’s duration reports for natural videos (up to almost half a minute long) and, most importantly, predicts whether a person reports a scene as relatively shorter or longer–the biases in time perception that reflect how natural experience of time deviates from clock time.

Temporal perceptual learning distinguishes between empty and filled intervals

Temporal perceptual learning (TPL) refers to improved temporal performance as a result of training with sub-second intervals. Most studies on TPL have focused on empty intervals (i.e. intervals marked by two brief stimuli); however, scholars have suggested that filled intervals (i.e. intervals presented as continuous sensory inputs) might have different underlying mechanisms. Therefore, the current study aimed to test whether empty and filled intervals yield similar TPL performance and whether such learning effects could transfer mutually. To this end, we trained two groups of participants with empty and filled intervals of 200 ms for four days, respectively. We found that the empty-interval group clearly improved their timing performances after training, and such an effect transferred to filled intervals of 200 ms. By contrast, the filled-interval group had neither learning nor transfer effect. Our results further shed light on the distinct mechanisms between empty and filled intervals in time perception while simultaneously replicating the classical findings on TPL involving empty intervals.

Auditory timing-tuned neural responses in the human auditory cortices

Perception of sub-second auditory event timing supports multisensory integration, and speech and music perception and production. Neural populations tuned for the timing (duration and rate) of visual events were recently described in several human extrastriate visual areas. Here we ask whether the brain also contains neural populations tuned for auditory event timing, and whether these are shared with visual timing. Using 7T fMRI, we measured responses to white noise bursts of changing duration and rate. We analyzed these responses using neural response models describing different parametric relationships between event timing and neural response amplitude. This revealed auditory timing-tuned responses in the primary auditory cortex, and auditory association areas of the belt, parabelt and premotor cortex. While these areas also showed tonotopic tuning for auditory pitch, pitch and timing preferences were not consistently correlated. Auditory timing-tuned response functions differed between these areas, though without clear hierarchical integration of responses. The similarity of auditory and visual timing tuned responses, together with the lack of overlap between the areas showing these responses for each modality, suggests modality-specific responses to event timing are computed similarly but from different sensory inputs, and then transformed differently to suit the needs of each modality.

Relative precision of top-down attentional modulations is lower in early visual cortex compared to mid- and high-level visual areas

Top-down spatial attention enhances cortical representations of behaviorally relevant visual information and increases the precision of perceptual reports. However, little is known about the relative precision of top-down attentional modulations in different visual areas, especially compared with the highly precise stimulus-driven responses that are observed in early visual cortex. For example, the precision of attentional modulations in early visual areas may be limited by the relatively coarse spatial selectivity and the anatomical connectivity of the areas in prefrontal cortex that generate and relay the top-down signals. Here, we used functional MRI (fMRI) and human participants to assess the precision of bottom-up spatial representations evoked by high-contrast stimuli across the visual hierarchy. Then, we examined the relative precision of top-down attentional modulations in the absence of spatially specific bottom-up drive. Whereas V1 showed the largest relative difference between the precision of top-down attentional modulations and the precision of bottom-up modulations, midlevel areas such as V4 showed relatively smaller differences between the precision of top-down and bottom-up modulations. Overall, this interaction between visual areas (e.g., V1 vs. V4) and the relative precision of top-down and bottom-up modulations suggests that the precision of top-down attentional modulations is limited by the representational fidelity of areas that generate and relay top-down feedback signals.NEW & NOTEWORTHY When the relative precision of purely top-down and bottom-up signals were compared across visual areas, early visual areas like V1 showed higher bottom-up precision compared with top-down precision. In contrast, midlevel areas showed similar levels of top-down and bottom-up precision. This result suggests that the precision of top-down attentional modulations may be limited by the relatively coarse spatial selectivity and the anatomical connectivity of the areas generating and relaying the signals.

A population receptive field model of the magnetoencephalography response

Computational models which predict the neurophysiological response from experimental stimuli have played an important role in human neuroimaging. One type of computational model, the population receptive field (pRF), has been used to describe cortical responses at the millimeter scale using functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG). However, pRF models are not widely used for non-invasive electromagnetic field measurements (EEG/MEG), because individual sensors pool responses originating from several centimeter of cortex, containing neural populations with widely varying spatial tuning. Here, we introduce a forward-modeling approach in which pRFs estimated from fMRI data are used to predict MEG sensor responses. Subjects viewed contrast-reversing bar stimuli sweeping across the visual field in separate fMRI and MEG sessions. Individual subject's pRFs were modeled on the cortical surface at the millimeter scale using the fMRI data. We then predicted cortical time series and projected these predictions to MEG sensors using a biophysical MEG forward model, accounting for the pooling across cortex. We compared the predicted MEG responses to observed visually evoked steady-state responses measured in the MEG session. We found that pRF parameters estimated by fMRI could explain a substantial fraction of the variance in steady-state MEG sensor responses (up to 60% in individual sensors). Control analyses in which we artificially perturbed either pRF size or pRF position reduced MEG prediction accuracy, indicating that MEG data are sensitive to pRF properties derived from fMRI. Our model provides a quantitative approach to link fMRI and MEG measurements, thereby enabling advances in our understanding of spatiotemporal dynamics in human visual field maps.

Is Developmental Dyslexia Due to a Visual and Not a Phonological Impairment?

It is a widely held belief that developmental dyslexia (DD) is a phonological disorder in which readers have difficulty associating graphemes with their corresponding phonemes. In contrast, the magnocellular theory of dyslexia assumes that DD is a visual disorder caused by dysfunctional magnocellular neural pathways. The review explores arguments for and against these theories. Recent results have shown that DD is caused by (1) a reduced ability to simultaneously recognize sequences of letters that make up words, (2) longer fixation times required to simultaneously recognize strings of letters, and (3) amplitudes of saccades that do not match the number of simultaneously recognized letters. It was shown that pseudowords that could not be recognized simultaneously were recognized almost without errors when the fixation time was extended. However, there is an individual maximum number of letters that each reader with DD can recognize simultaneously. Findings on the neurobiological basis of temporal summation have shown that a necessary prolongation of fixation times is due to impaired processing mechanisms of the visual system, presumably involving magnocells and parvocells. An area in the mid-fusiform gyrus also appears to play a significant role in the ability to simultaneously recognize words and pseudowords. The results also contradict the assumption that DD is due to a lack of eye movement control. The present research does not support the assumption that DD is caused by a phonological disorder but shows that DD is due to a visual processing dysfunction.

Imaging faster neural dynamics with fast fMRI: A need for updated models of the hemodynamic response

Fast fMRI enables the detection of neural dynamics over timescales of hundreds of milliseconds, suggesting it may provide a new avenue for studying subsecond neural processes in the human brain. The magnitudes of these fast fMRI dynamics are far greater than predicted by canonical models of the hemodynamic response. Several studies have established nonlinear properties of the hemodynamic response that have significant implications for fast fMRI. We first review nonlinear properties of the hemodynamic response function that may underlie fast fMRI signals. We then illustrate the breakdown of canonical hemodynamic response models in the context of fast neural dynamics. We will then argue that the canonical hemodynamic response function is not likely to reflect the BOLD response to neuronal activity driven by sparse or naturalistic stimuli or perhaps to spontaneous neuronal fluctuations in the resting state. These properties suggest that fast fMRI is capable of tracking surprisingly fast neuronal dynamics, and we discuss the neuroscientific questions that could be addressed using this approach.

Holistic face recognition is an emergent phenomenon of spatial processing in face-selective regions

Spatial processing by receptive fields is a core property of the visual system. However, it is unknown how spatial processing in high-level regions contributes to recognition behavior. As face inversion is thought to disrupt typical holistic processing of information in faces, we mapped population receptive fields (pRFs) with upright and inverted faces in the human visual system. Here we show that in face-selective regions, but not primary visual cortex, pRFs and overall visual field coverage are smaller and shifted downward in response to face inversion. From these measurements, we successfully predict the relative behavioral detriment of face inversion at different positions in the visual field. This correspondence between neural measurements and behavior demonstrates how spatial processing in face-selective regions may enable holistic perception. These results not only show that spatial processing in high-level visual regions is dynamically used towards recognition, but also suggest a powerful approach for bridging neural computations by receptive fields to behavior.

Temporal stability of stimulus representation increases along rodent visual cortical hierarchies

Cortical representations of brief, static stimuli become more invariant to identity-preserving transformations along the ventral stream. Likewise, increased invariance along the visual hierarchy should imply greater temporal persistence of temporally structured dynamic stimuli, possibly complemented by temporal broadening of neuronal receptive fields. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. By probing the rat analog of the ventral stream with movies, we uncovered a hierarchy of temporal scales, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations grew gradually along the hierarchy. A database of recordings from mouse showed similar trends, additionally revealing dependencies on the behavioral state. Overall, these findings show that visual representations become progressively more stable along rodent visual processing hierarchies, with an important contribution provided by intrinsic processing.

Population Models, Not Analyses, of Human Neuroscience Measurements

Selectivity for many basic properties of visual stimuli, such as orientation, is thought to be organized at the scale of cortical columns, making it difficult or impossible to measure directly with noninvasive human neuroscience measurement. However, computational analyses of neuroimaging data have shown that selectivity for orientation can be recovered by considering the pattern of response across a region of cortex. This suggests that computational analyses can reveal representation encoded at a finer spatial scale than is implied by the spatial resolution limits of measurement techniques. This potentially opens up the possibility to study a much wider range of neural phenomena that are otherwise inaccessible through noninvasive measurement. However, as we review in this article, a large body of evidence suggests an alternative hypothesis to this superresolution account: that orientation information is available at the spatial scale of cortical maps and thus easily measurable at the spatial resolution of standard techniques. In fact, a population model shows that this orientation information need not even come from single-unit selectivity for orientation tuning, but instead can result from population selectivity for spatial frequency. Thus, a categorical error of interpretation can result whereby orientation selectivity can be confused with spatial frequency selectivity. This is similarly problematic for the interpretation of results from numerous studies of more complex representations and cognitive functions that have built upon the computational techniques used to reveal stimulus orientation. We suggest in this review that these interpretational ambiguities can be avoided by treating computational analyses as models of the neural processes that give rise to measurement. Building upon the modeling tradition in vision science using considerations of whether population models meet a set of core criteria is important for creating the foundation for a cumulative and replicable approach to making valid inferences from human neuroscience measurements. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Dyslexic Readers Improve without Training When Using a Computer-Guided Reading Strategy

Background: Flawless reading presupposes the ability to simultaneously recognize a sequence of letters, to fixate words at a given location for a given time, to exert eye movements of a given amplitude, and to retrieve phonems rapidly from memory. Poor reading performance may be due to an impairment of at least one of these abilities. Objectives: It was investigated whether reading performance of dyslexic children can be improved by changing the reading strategy without any previous training. Methods: 60 dyslexic German children read a text without and with the help of a computer. A tailored computer program subdivided the text into segments that consisted of no more letters than the children could simultaneously recognize, indicated the location in the segments to which the gaze should be directed, indicated how long the gaze should be directed to each segment, which reading saccades the children should execute, and when the children should pronounce the segments. The computer aided reading was not preceded by any training. Results: It was shown that the rate of reading mistakes dropped immediately by 69.97% if a computer determined the reading process. Computer aided reading reached the highest effect size of Cohen d = 2.649. Conclusions: The results show which abilities are indispensable for reading, that the impairment of at least one of the abilities leads to reading deficiencies that are diagnosed as dyslexia, and that a computer-guided, altered reading strategy immediately reduces the rate of reading mistakes. There was no evidence that dyslexia is due to a lack of eye movement control or reduced visual attention.

Organization of directed functional connectivity among nodes of ventral attention network reveals the common network mechanisms underlying saliency processing across distinct spatial and spatio-temporal scales

Previous neuroimaging studies have extensively evaluated the structural and functional connectivity of the Ventral Attention Network (VAN) and its role in reorienting attention in the presence of a salient (pop-out) stimulus. However, a detailed understanding of the "directed" functional connectivity within the VAN during the process of reorientation remains elusive. Functional magnetic resonance imaging (fMRI) studies have not adequately addressed this issue due to a lack of appropriate temporal resolution required to capture this dynamic process. The present study investigates the neural changes associated with processing salient distractors operating at a slow and a fast time scale using custom-designed experiment involving visual search on static images and dynamic motion tracking, respectively. We recorded high-density scalp electroencephalography (EEG) from healthy human volunteers, obtained saliency-specific behavioral and spectral changes during the tasks, localized the sources underlying the spectral power modulations with individual-specific structural MRI scans, reconstructed the waveforms of the sources and finally, investigated the causal relationships between the sources using spectral Granger-Geweke Causality (GGC). We found that salient stimuli processing, across tasks with varying spatio-temporal complexities, involves a characteristic modulation in the alpha frequency band which is executed primarily by the nodes of the VAN constituting the temporo-parietal junction (TPJ), the insula and the lateral prefrontal cortex (lPFC). The directed functional connectivity results further revealed the presence of bidirectional interactions among prominent nodes of right-lateralized VAN, corresponding only to the trials with saliency. Thus, our study elucidates the invariant network mechanisms for processing saliency in visual attention tasks across diverse time-scales.

Dissociated modulations of multivoxel activation patterns in the ventral and dorsal visual pathways by the temporal dynamics of stimuli

INTRODUCTION

Previous studies suggested temporal limitations of visual object identification in the ventral pathway. Moreover, multivoxel pattern analyses (MVPA) of fMRI activation have shown reliable encoding of various object categories including faces and tools in the ventral pathway. By contrast, the dorsal pathway is involved in reaching a target and grasping a tool, and quicker in processing the temporal dynamics of stimulus change. However, little is known about how activation patterns in both pathways may change according to the temporal dynamics of stimulus change.

METHODS

Here, we measured fMRI responses of two consecutive stimuli with varying interstimulus intervals (ISIs), and we compared how the two visual pathways respond to the dynamics of stimuli by using MVPA and information-based searchlight mapping.

RESULTS

We found that the temporal dynamics of stimuli modulate responses of the two visual pathways in opposite directions. Specifically, slower temporal dynamics (longer ISIs) led to greater activity and better MVPA results in the ventral pathway. However, faster temporal dynamics (shorter ISIs) led to greater activity and better MVPA results in the dorsal pathway.

CONCLUSIONS

These results are the first to show how temporal dynamics of stimulus change modulated multivoxel fMRI activation pattern change. And such temporal dynamic response function in different ROIs along the two visual pathways may shed lights on understanding functional relationship and organization of these ROIs.

Diverse Temporal Dynamics of Repetition Suppression Revealed by Intracranial Recordings in the Human Ventral Temporal Cortex

Repeated stimulus presentations commonly produce decreased neural responses-a phenomenon known as repetition suppression (RS) or adaptation-in ventral temporal cortex (VTC) of humans and nonhuman primates. However, the temporal features of RS in human VTC are not well understood. To fill this gap in knowledge, we utilized the precise spatial localization and high temporal resolution of electrocorticography (ECoG) from nine human subjects implanted with intracranial electrodes in the VTC. The subjects viewed nonrepeated and repeated images of faces with long-lagged intervals and many intervening stimuli between repeats. We report three main findings: 1) robust RS occurs in VTC for activity in high-frequency broadband (HFB), but not lower-frequency bands; 2) RS of the HFB signal is associated with lower peak magnitude (PM), lower total responses, and earlier peak responses; and 3) RS effects occur early within initial stages of stimulus processing and persist for the entire stimulus duration. We discuss these findings in the context of early and late components of visual perception, as well as theoretical models of repetition suppression.

Investigating face and house discrimination at foveal to parafoveal locations reveals category-specific characteristics

Since perceptual and neural face sensitivity is associated with a foveal bias, and neural place sensitivity is associated with a peripheral bias (integration over space), we hypothesized that face perception ability will decline more with eccentricity than place perception ability. We also wanted to examine whether face perception ability would show a left visual field (LeVF) bias due to earlier reports suggesting right hemisphere dominance for faces, or would show an upper or lower visual field bias. Participants performed foveal and parafoveal face and house discrimination tasks for upright or inverted stimuli (≤4°) while their eye movements were monitored. Low-level visual tasks were also measured. The eccentricity-related accuracy reductions were evident for all categories. Through detailed analyses we found (i) a robust face inversion effect across the parafovea, while for houses an opposite effect was found, (ii) higher eccentricity-related sensitivity for face performance than for house performance (via inverted vs. upright within-category eccentricity-driven reductions), (iii) within-category but not across-category performance associations across eccentricities, and (iv) no hemifield biases. Our central to parafoveal investigations suggest that high-level vision processing may be reflected in behavioural performance.

A Network of Topographic Maps in Human Association Cortex Hierarchically Transforms Visual Timing-Selective Responses

Accurately timing sub-second sensory events is crucial when perceiving our dynamic world. This ability allows complex human behaviors that require timing-dependent multisensory integration and action planning. Such behaviors include perception and performance of speech, music, driving, and many sports. How are responses to sensory event timing processed for multisensory integration and action planning? We measured responses to viewing systematically changing visual event timing using ultra-high-field fMRI. We analyzed these responses with neural population response models selective for event duration and frequency, following behavioral, computational, and macaque action planning results and comparisons to alternative models. We found systematic local changes in timing preferences (recently described in supplementary motor area) in an extensive network of topographic timing maps, mirroring sensory cortices and other quantity processing networks. These timing maps were partially left lateralized and widely spread, from occipital visual areas through parietal multisensory areas to frontal action planning areas. Responses to event duration and frequency were closely linked. As in sensory cortical maps, response precision varied systematically with timing preferences, and timing selectivity systematically varied between maps. Progressing from posterior to anterior maps, responses to multiple events were increasingly integrated, response selectivity narrowed, and responses focused increasingly on the middle of the presented timing range. These timing maps largely overlap with numerosity and visual field map networks. In both visual timing map and visual field map networks, selective responses and topographic map organization may facilitate hierarchical transformations by allowing neural populations to interact over minimal distances.

•

Many brain areas show neural responses to specific ranges of visual event timing

•

Timing preferences change gradually in these areas, forming topographic timing maps

•

Neural response properties hierarchically transform from visual to premotor areas

•

Timing, numerosity, and visual field map networks are distinct but largely overlap

Measurement of ultra-fast signal progression related to face processing by 7T fMRI

Given that the brain is a dynamic system, the temporal characteristics of brain function are important. Previous functional magnetic resonance imaging (fMRI) studies have attempted to overcome the limitations of temporal resolution to investigate dynamic states of brain activity. However, finding an fMRI method with sufficient temporal resolution to keep up with the progress of neuronal signals in the brain is challenging. This study aimed to detect between‐hemisphere signal progression, occurring on a timescale of tens of milliseconds, in the ventral brain regions involved in face processing. To this end, we devised an inter‐stimulus interval (ISI) stimulation scheme and used a 7T MRI system to obtain fMRI signals with a high signal‐to‐noise ratio. We conducted two experiments: one to measure signal suppression depending on the ISI and another to measure the relationship between the amount of suppression and the ISI. These two experiments enabled us to measure the signal transfer time from a brain region in the ventral visual stream to its counterpart in the opposite hemisphere through the corpus callosum. These findings demonstrate the feasibility of using fMRI to measure ultra‐fast signals (tens of milliseconds) and could facilitate the elucidation of further aspects of dynamic brain function.

Predicting neuronal dynamics with a delayed gain control model

Visual neurons respond to static images with specific dynamics: neuronal responses sum sub-additively over time, reduce in amplitude with repeated or sustained stimuli (neuronal adaptation), and are slower at low stimulus contrast. Here, we propose a simple model that predicts these seemingly disparate response patterns observed in a diverse set of measurements-intracranial electrodes in patients, fMRI, and macaque single unit spiking. The model takes a time-varying contrast time course of a stimulus as input, and produces predicted neuronal dynamics as output. Model computation consists of linear filtering, expansive exponentiation, and a divisive gain control. The gain control signal relates to but is slower than the linear signal, and this delay is critical in giving rise to predictions matched to the observed dynamics. Our model is simpler than previously proposed related models, and fitting the model to intracranial EEG data uncovers two regularities across human visual field maps: estimated linear filters (temporal receptive fields) systematically differ across and within visual field maps, and later areas exhibit more rapid and substantial gain control. The model is further generalizable to account for dynamics of contrast-dependent spike rates in macaque V1, and amplitudes of fMRI BOLD in human V1.

Visual temporal frequency preference shows a distinct cortical architecture using fMRI

Studies of visual temporal frequency preference typically examine frequencies under 20 Hz and measure local activity to evaluate the sensitivity of different cortical areas to variations in temporal frequencies. Most of these studies have not attempted to map preferred temporal frequency within and across visual areas, nor have they explored in detail, stimuli at gamma frequency, which recent research suggests may have potential clinical utility. In this study, we address this gap by using functional magnetic resonance imaging (fMRI) to measure response to flickering visual stimuli varying in frequency from 1 to 40 Hz. We apply stimulation in both a block design to examine task response and a steady-state design to examine functional connectivity. We observed distinct activation patterns between 1 Hz and 40 Hz stimuli. We also found that the correlation between medial thalamus and visual cortex was modulated by the temporal frequency. The modulation functions and tuned frequencies are different for the visual activity and thalamo-visual correlations. Using both fMRI activity and connectivity measurements, we show evidence for a temporal frequency specific organization across the human visual system.

Neural adaptation to faces reveals racial outgroup homogeneity effects in early perception

The tendency to view members of social outgroups as interchangeable has long been considered a core component of intergroup bias and a precursor to stereotyping and discrimination. However, the early perceptual nature of these intergroup biases is poorly understood. Here, we used a functional MRI adaptation paradigm to assess how face-selective brain regions respond to variation in physical similarity among racial ingroup (White) and outgroup (Black) faces. We conclude that differences emerge in the different tuning properties of early face-selective cortex for racial ingroup and outgroup faces and mirror behavioral differences in memory and perception of racial ingroup versus outgroup faces. These results suggest that outgroup deindividuation emerges at some of the earliest stages of perception.

A hallmark of intergroup biases is the tendency to individuate members of one’s own group but process members of other groups categorically. While the consequences of these biases for stereotyping and discrimination are well-documented, their early perceptual underpinnings remain less understood. Here, we investigated the neural mechanisms of this effect by testing whether high-level visual cortex is differentially tuned in its sensitivity to variation in own-race versus other-race faces. Using a functional MRI adaptation paradigm, we measured White participants’ habituation to blocks of White and Black faces that parametrically varied in their groupwise similarity. Participants showed a greater tendency to individuate own-race faces in perception, showing both greater release from adaptation to unique identities and increased sensitivity in the adaptation response to physical difference among faces. These group differences emerge in the tuning of early face-selective cortex and mirror behavioral differences in the memory and perception of own- versus other-race faces. Our results suggest that biases for other-race faces emerge at some of the earliest stages of sensory perception.

Distinguishing Hemodynamics from Function in the Human LGN Using a Temporal Response Model

We developed a temporal population receptive field model to differentiate the neural and hemodynamic response functions (HRF) in the human lateral geniculate nucleus (LGN). The HRF in the human LGN is dominated by the richly vascularized hilum, a structure that serves as a point of entry for blood vessels entering the LGN and supplying the substrates of central vision. The location of the hilum along the ventral surface of the LGN and the resulting gradient in the amplitude of the HRF across the extent of the LGN have made it difficult to segment the human LGN into its more interesting magnocellular and parvocellular regions that represent two distinct visual processing streams. Here, we show that an intrinsic clustering of the LGN responses to a variety of visual inputs reveals the hilum, and further, that this clustering is dominated by the amplitude of the HRF. We introduced a temporal population receptive field model that includes separate sustained and transient temporal impulse response functions that vary on a much short timescale than the HRF. When we account for the HRF amplitude, we demonstrate that this temporal response model is able to functionally segregate the residual responses according to their temporal properties.

Differential sustained and transient temporal processing across visual streams

How do high-level visual regions process the temporal aspects of our visual experience? While the temporal sensitivity of early visual cortex has been studied with fMRI in humans, temporal processing in high-level visual cortex is largely unknown. By modeling neural responses with millisecond precision in separate sustained and transient channels, and introducing a flexible encoding framework that captures differences in neural temporal integration time windows and response nonlinearities, we predict fMRI responses across visual cortex for stimuli ranging from 33 ms to 20 s. Using this innovative approach, we discovered that lateral category-selective regions respond to visual transients associated with stimulus onsets and offsets but not sustained visual information. Thus, lateral category-selective regions compute moment-to-moment visual transitions, but not stable features of the visual input. In contrast, ventral category-selective regions process both sustained and transient components of the visual input. Our model revealed that sustained channel responses to prolonged stimuli exhibit adaptation, whereas transient channel responses to stimulus offsets are surprisingly larger than for stimulus onsets. This large offset transient response may reflect a memory trace of the stimulus when it is no longer visible, whereas the onset transient response may reflect rapid processing of new items. Together, these findings reveal previously unconsidered, fundamental temporal mechanisms that distinguish visual streams in the human brain. Importantly, our results underscore the promise of modeling brain responses with millisecond precision to understand the underlying neural computations.

How does the brain encode the timing of our visual experience? Using functional magnetic resonance imaging (fMRI) and a generative temporal model with millisecond resolution, we discovered that visual regions in the lateral and ventral processing streams fundamentally differ in their temporal processing of the visual input. Regions in lateral temporal cortex process visual transients associated with the beginning and ending of the stimulus, but not its stable aspects. That is, lateral regions appear to compute moment-to-moment changes in the visual input. In contrast, regions in ventral temporal cortex process both stable and transient components of the visual input, even as the response to the former exhibits adaptation. Surprisingly, the model predicts that in ventral regions responses to stimulus endings are larger than beginnings. We suggest that ending responses may reflect a memory trace of the stimulus, when it is no longer visible, and the beginning responses may reflect processing of new inputs. Together, these findings (i) reveal a fundamental temporal mechanism that distinguishes visual streams and (ii) highlight both the importance and utility of modeling brain responses with millisecond precision to understand the temporal dynamics of neural computations in the human brain.

How Visual Cortical Organization Is Altered by Ophthalmologic and Neurologic Disorders

Receptive fields are a core property of cortical organization. Modern neuroimaging allows routine access to visual population receptive fields (pRFs), enabling investigations of clinical disorders. Yet how the underlying neural circuitry operates is controversial. The controversy surrounds observations that measurements of pRFs can change in healthy adults as well as in patients with a range of ophthalmological and neurological disorders. The debate relates to the balance between plasticity and stability of the underlying neural circuitry. We propose that to move the debate forward, the field needs to define the implied mechanism. First, we review the pRF changes in both healthy subjects and those with clinical disorders. Then, we propose a computational model that describes how pRFs can change in healthy humans. We assert that we can correctly interpret the pRF changes in clinical disorders only if we establish the capabilities and limitations of pRF dynamics in healthy humans with mechanistic models that provide quantitative predictions.

The functional neuroanatomy of face perception: from brain measurements to deep neural networks

A central goal in neuroscience is to understand how processing within the ventral visual stream enables rapid and robust perception and recognition. Recent neuroscientific discoveries have significantly advanced understanding of the function, structure and computations along the ventral visual stream that serve as the infrastructure supporting this behaviour. In parallel, significant advances in computational models, such as hierarchical deep neural networks (DNNs), have brought machine performance to a level that is commensurate with human performance. Here, we propose a new framework using the ventral face network as a model system to illustrate how increasing the neural accuracy of present DNNs may allow researchers to test the computational benefits of the functional architecture of the human brain. Thus, the review (i) considers specific neural implementational features of the ventral face network, (ii) describes similarities and differences between the functional architecture of the brain and DNNs, and (iii) provides a hypothesis for the computational value of implementational features within the brain that may improve DNN performance. Importantly, this new framework promotes the incorporation of neuroscientific findings into DNNs in order to test the computational benefits of fundamental organizational features of the visual system.

A quantitative framework for motion visibility in human cortex

Despite the central use of motion visibility to reveal the neural basis of perception, perceptual decision making, and sensory inference there exists no comprehensive quantitative framework establishing how motion visibility parameters modulate human cortical response. Random-dot motion stimuli can be made less visible by reducing image contrast or motion coherence, or by shortening the stimulus duration. Because each of these manipulations modulates the strength of sensory neural responses they have all been extensively used to reveal cognitive and other nonsensory phenomena such as the influence of priors, attention, and choice-history biases. However, each of these manipulations is thought to influence response in different ways across different cortical regions and a comprehensive study is required to interpret this literature. Here, human participants observed random-dot stimuli varying across a large range of contrast, coherence, and stimulus durations as we measured blood-oxygen-level dependent responses. We developed a framework for modeling these responses that quantifies their functional form and sensitivity across areas. Our framework demonstrates the sensitivity of all visual areas to each parameter, with early visual areas V1-V4 showing more parametric sensitivity to changes in contrast and V3A and the human middle temporal area to coherence. Our results suggest that while motion contrast, coherence, and duration share cortical representation, they are encoded with distinct functional forms and sensitivity. Thus, our quantitative framework serves as a reference for interpretation of the vast perceptual literature manipulating these parameters and shows that different manipulations of visibility will have different effects across human visual cortex and need to be interpreted accordingly. NEW & NOTEWORTHY Manipulations of motion visibility have served as a key tool for understanding the neural basis for visual perception. Here we measured human cortical response to changes in visibility across a comprehensive range of motion visibility parameters and modeled these with a quantitative framework. Our quantitative framework can be used as a reference for linking human cortical response to perception and underscores that different manipulations of motion visibility can have greatly different effects on cortical representation.

Facial Expression Enhances Emotion Perception Compared to Vocal Prosody: Behavioral and fMRI Studies

Facial and vocal expressions are essential modalities mediating the perception of emotion and social communication. Nonetheless, currently little is known about how emotion perception and its neural substrates differ across facial expression and vocal prosody. To clarify this issue, functional MRI scans were acquired in Study 1, in which participants were asked to discriminate the valence of emotional expression (angry, happy or neutral) from facial, vocal, or bimodal stimuli. In Study 2, we used an affective priming task (unimodal materials as primers and bimodal materials as target) and participants were asked to rate the intensity, valence, and arousal of the targets. Study 1 showed higher accuracy and shorter response latencies in the facial than in the vocal modality for a happy expression. Whole-brain analysis showed enhanced activation during facial compared to vocal emotions in the inferior temporal-occipital regions. Region of interest analysis showed a higher percentage signal change for facial than for vocal anger in the superior temporal sulcus. Study 2 showed that facial relative to vocal priming of anger had a greater influence on perceived emotion for bimodal targets, irrespective of the target valence. These findings suggest that facial expression is associated with enhanced emotion perception compared to equivalent vocal prosodies.

Development differentially sculpts receptive fields across early and high-level human visual cortex

Receptive fields (RFs) processing information in restricted parts of the visual field are a key property of visual system neurons. However, how RFs develop in humans is unknown. Using fMRI and population receptive field (pRF) modeling in children and adults, we determine where and how pRFs develop across the ventral visual stream. Here we report that pRF properties in visual field maps, from the first visual area, V1, through the first ventro-occipital area, VO1, are adult-like by age 5. However, pRF properties in face-selective and character-selective regions develop into adulthood, increasing the foveal coverage bias for faces in the right hemisphere and words in the left hemisphere. Eye-tracking indicates that pRF changes are related to changing fixation patterns on words and faces across development. These findings suggest a link between face and word viewing behavior and the differential development of pRFs across visual cortex, potentially due to competition on foveal coverage.

Population receptive fields (pRFs) in the visual system are key information-processors, but how they develop is unknown. Here, authors use fMRI and pRF modeling in children and adults to show that in the ventral stream only pRFs in face- and word-selective regions continue to develop, mirroring changes in viewing behavior.

Compressive Temporal Summation in Human Visual Cortex

Combining sensory inputs over space and time is fundamental to vision. Population receptive field models have been successful in characterizing spatial encoding throughout the human visual pathways. A parallel question, how visual areas in the human brain process information distributed over time, has received less attention. One challenge is that the most widely used neuroimaging method, fMRI, has coarse temporal resolution compared with the time-scale of neural dynamics. Here, via carefully controlled temporally modulated stimuli, we show that information about temporal processing can be readily derived from fMRI signal amplitudes in male and female subjects. We find that all visual areas exhibit subadditive summation, whereby responses to longer stimuli are less than the linear prediction from briefer stimuli. We also find fMRI evidence that the neural response to two stimuli is reduced for brief interstimulus intervals (indicating adaptation). These effects are more pronounced in visual areas anterior to V1-V3. Finally, we develop a general model that shows how these effects can be captured with two simple operations: temporal summation followed by a compressive nonlinearity. This model operates for arbitrary temporal stimulation patterns and provides a simple and interpretable set of computations that can be used to characterize neural response properties across the visual hierarchy. Importantly, compressive temporal summation directly parallels earlier findings of compressive spatial summation in visual cortex describing responses to stimuli distributed across space. This indicates that, for space and time, cortex uses a similar processing strategy to achieve higher-level and increasingly invariant representations of the visual world.SIGNIFICANCE STATEMENT Combining sensory inputs over time is fundamental to seeing. Two important temporal phenomena are summation, the accumulation of sensory inputs over time, and adaptation, a response reduction for repeated or sustained stimuli. We investigated these phenomena in the human visual system using fMRI. We built predictive models that operate on arbitrary temporal patterns of stimulation using two simple computations: temporal summation followed by a compressive nonlinearity. Our new temporal compressive summation model captures (1) subadditive temporal summation, and (2) adaptation. We show that the model accounts for systematic differences in these phenomena across visual areas. Finally, we show that for space and time, the visual system uses a similar strategy to achieve increasingly invariant representations of the visual world.