Determinants and mechanisms of attentional modulation of neural processing

Event-related potential (ERP) evidence for visual processing differences in children and adults with cystinosis (CTNS gene mutations)

BACKGROUND

Cystinosis, a rare lysosomal storage disease caused by mutations in the CTNS gene, is characterized by cystine crystallization and accumulation within multiple tissues, including kidney and brain. Its impact on neural function appears mild relative to its effects on other organs during early disease, but since therapeutic advances have led to substantially increased life expectancy, neurological implications are of increasing interest, necessitating deeper understanding of the impact of cystinosis on neurocognitive function. Behavioral difficulties have been reported in cystinosis in the visual domain. Very little is known, however, about how the brains of people living with cystinosis process visual information. This is especially interesting given that cystine accumulation in the cornea and posterior ocular structures is a hallmark of cystinosis.

METHODS

Here, high-density scalp electrophysiology was recorded to visual stimuli (during a Go/No-Go task) to investigate visual processing in individuals with cystinosis, compared to age-matched controls. Analyses focused on early stages of cortical visual processing.

RESULTS

The groups differed in their initial cortical response, with individuals with cystinosis exhibiting a significantly larger visual evoked potential (VEP) in the 130-150 ms time window. The groups also differed in the associations between neural responses and verbal abilities: While controls with higher IQ scores presented larger neural responses, that relationship was not observed in cystinosis.

CONCLUSIONS

The enlarged VEP in cystinosis could be the result of cortical hyperexcitability and/or differences in attentional engagement and explain, at least partially, the visual and visual-spatial difficulties described in this population.

Event-related potential (ERP) evidence for early visual processing differences in children and adults with Cystinosis (CTNS gene mutations)

Background

Cystinosis, a rare lysosomal storage disease caused by mutations in the CTNS gene, is characterized by cystine crystallization and accumulation within multiple tissues, including kidney and brain. Its impact on neural function appears mild relative to its effects on other organs during early disease, but since therapeutic advances have led to substantially increased life expectancy, neurological implications are of increasing interest, necessitating deeper understanding of the impact of cystinosis on neurocognitive function. Behavioral difficulties have been reported in cystinosis in the visual domain. Very little is known, however, about how the brains of people living with cystinosis process visual information. This is especially interesting given that cystine accumulation in the cornea and posterior ocular structures is a hallmark of cystinosis.

Methods

Here, high-density scalp electrophysiology was recorded to visual stimuli (during a Go/No-Go task) to investigate early visual processing in individuals with cystinosis, compared to age-matched controls. Analyses focused on early stages of cortical visual processing.

Results

The groups differed in their initial cortical response, with individuals with cystinosis exhibiting a significantly larger visual evoked potential (VEP) in the 130-150 ms time window. The groups also differed in the associations between neural responses and verbal abilities: While controls with higher IQ scores presented larger neural responses, that relationship was not observed in cystinosis.

Conclusions

The enlarged VEP in cystinosis could be the result of cortical hyperexcitability and/or differences in attentional engagement and explain, at least partially, the visual and visual-spatial difficulties described in this population.

Neural modulations in the auditory cortex during internal and external attention tasks: A single-patient intracranial recording study

Brain sensory processing is not passive, but is rather modulated by our internal state. Different research methods such as non-invasive imaging methods and intracranial recording of the local field potential (LFP) have been used to study to what extent sensory processing and the auditory cortex in particular are modulated by selective attention. However, at the level of the single- or multi-units the selective attention in humans has not been tested. In addition, most previous research on selective attention has explored externally-oriented attention, but attention can be also directed inward (i.e., internal attention), like spontaneous self-generated thoughts and mind-wandering. In the present study we had a rare opportunity to record multi-unit activity (MUA) in the auditory cortex of a patient. To complement, we also analyzed the LFP signal of the macro-contact in the auditory cortex. Our experiment consisted of two conditions with periodic beeping sounds. The participants were asked either to count the beeps (i.e., an "external attention" condition) or to recall the events of the previous day (i.e., an "internal attention" condition). We found that the four out of seven recorded units in the auditory cortex showed increased firing rates in "external attention" compared to "internal attention" condition. The beginning of this attentional modulation varied across multi-units between 30-50 msec and 130-150 msec from stimulus onset, a result that is compatible with an early selection view. The LFP evoked potential and induced high gamma activity both showed attentional modulation starting at about 70-80 msec. As the control, for the same experiment we recorded MUA activity in the amygdala and hippocampus of two additional patients. No major attentional modulation was found in the control regions. Overall, we believe that our results provide new empirical information and support for existing theoretical views on selective attention and spontaneous self-generated cognition.

Parallel fast and slow recurrent cortical processing mediates target and distractor selection in visual search

Visual search has been commonly used to study the neural correlates of attentional allocation in space. Recent electrophysiological research has disentangled distractor processing from target processing, showing that these mechanisms appear to operate in parallel and show electric fields of opposite polarity. Nevertheless, the localization and exact nature of this activity is unknown. Here, using MEG in humans, we provide a spatiotemporal characterization of target and distractor processing in visual cortex. We demonstrate that source activity underlying target- and distractor-processing propagates in parallel as fast and slow sweep from higher to lower hierarchical levels in visual cortex. Importantly, the fast propagating target-related source activity bypasses intermediate levels to go directly to V1, and this V1 activity correlates with behavioral performance. These findings suggest that reentrant processing is important for both selection and attenuation of stimuli, and such processing operates in parallel feedback loops.

Sarah E. Donohue et al. characterize the spatiotemporal propagation of target and distractor processing in the human visual cortex. They show that these signals propagate in parallel as fast and slow sweeps from higher to lower hierarchical levels, and that the fast target processing signal can bypass intermediate levels correlating with behavioral performance.

Odor Identification in Rats: Behavioral and Electrophysiological Evidence of Learned Olfactory-Auditory Associations

The ability to recognize and identify a smell is highly dependent on multisensory context and expectation, for example, hearing the name of the odor source. Here, we develop a novel auditory-odor association task in rats, wherein the animal learns that a specific auditory tone, when associated with a specific odor, predicts reward (Go signal), whereas the same tone associated with a different odor, or vice versa, is not (No-Go signal). The tone occurs prior to the onset of the odor, allowing physiological analyses of sensory-evoked local field potential (LFP) activity to each stimulus in primary auditory cortex and anterior piriform cortex (aPCX). In trained animals that have acquired the task, both auditory and subsequent olfactory cues activate β band oscillations in both the auditory cortex and PCX, suggesting multisensory integration. Naive animals show no such multisensory responses, suggesting the response is learned. In addition to the learned multisensory evoked responses, functional connectivity between auditory cortex and PCX, as assessed with spectral coherence and phase lag index (PLI), is enhanced. Importantly, both the multi-sensory evoked responses and the functional connectivity are context-dependent. In trained animals, the same auditory stimuli presented in the home cage evoke no responses in auditory cortex or PCX, and functional connectivity between the sensory cortices is reduced. Together, the results demonstrate how learning and context shape the expression of multisensory cortical processing. Given that odor identification impairment is associated with preclinical dementia in humans, the mechanisms suggested here may help develop experimental models to assess effects of neuropathology on behavior.

Mindfulness meditators show altered distributions of early and late neural activity markers of attention in a response inhibition task

Attention is vital for optimal behavioural performance in every-day life. Mindfulness meditation has been shown to enhance attention. However, the components of attention altered by meditation and the related neural activities are underexplored. In particular, the contributions of inhibitory processes and sustained attention are not well understood. To address these points, 34 meditators were compared to 28 age and gender matched controls during electroencephalography (EEG) recordings of neural activity during a Go/Nogo response inhibition task. This task generates a P3 event related potential, which is related to response inhibition processes in Nogo trials, and attention processes across both trial types. Compared with controls, meditators were more accurate at responding to Go and Nogo trials. Meditators showed a more frontally distributed P3 to both Go and Nogo trials, suggesting more frontal involvement in sustained attention rather than activity specific to response inhibition. Unexpectedly, meditators also showed increased positivity over the right parietal cortex prior to visual information reaching the occipital cortex (during the pre-C1 window). Both results were positively related to increased accuracy across both groups. The results suggest that meditators show altered engagement of neural regions related to attention, including both higher order processes generated by frontal regions, and sensory anticipation processes generated by poster regions. This activity may reflect an increased capacity to modulate a range of neural processes in order to meet task requirements. This increased capacity may underlie the improved attentional function observed in mindfulness meditators.

Contribution of FEF to Attentional Periodicity during Visual Search: A TMS Study

Visual search, looking for a target embedded among distractors, has long been used to study attention. Current theories postulate a two-stage process in which early visual areas perform feature extraction, whereas higher-order regions perform attentional selection. Such a model implies iterative communication between low- and high-level regions to sequentially select candidate targets in the array, focus attention on these elements, and eventually permit target recognition. This leads to two independent predictions: (1) high-level, attentional regions and (2) early visual regions should both be involved periodically during the search. Here, we used transcranial magnetic stimulation (TMS) applied over the frontal eye field (FEF) in humans, known to be involved in attentional selection, at various delays while observers performed a difficult, attentional search task. We observed a periodic pattern of interference at ∼6 Hz (theta) suggesting that the FEF is periodically involved during this difficult search task. We further compared this result with two previous studies (Dugué et al., 2011, 2015a) in which a similar TMS procedure was applied over the early visual cortex (V1) while observers performed the same task. This analysis revealed the same pattern of interference, i.e., V1 is periodically involved during this difficult search task, at the theta frequency. Past V1 evidence reappraised for this paper, together with our current FEF results, confirm both of our independent predictions, and suggest that difficult search is supported by low- and high-level regions, each involved periodically at the theta frequency.

Spiking Suppression Precedes Cued Attentional Enhancement of Neural Responses in Primary Visual Cortex

Attending to a visual stimulus increases its detectability, even if gaze is directed elsewhere. This covert attentional selection is known to enhance spiking across many brain areas, including the primary visual cortex (V1). Here we investigate the temporal dynamics of attention-related spiking changes in V1 of macaques performing a task that separates attentional selection from the onset of visual stimulation. We found that preceding attentional enhancement there was a sharp, transient decline in spiking following presentation of an attention-guiding cue. This disruption of V1 spiking was not observed in a task-naïve subject that passively observed the same stimulus sequence, suggesting that sensory activation is insufficient to cause suppression. Following this suppression, attended stimuli evoked more spiking than unattended stimuli, matching previous reports of attention-related activity in V1. Laminar analyses revealed a distinct pattern of activation in feedback-associated layers during both the cue-induced suppression and subsequent attentional enhancement. These findings suggest that top-down modulation of V1 spiking can be bidirectional and result in either suppression or enhancement of spiking responses.

Transcranial Magnetic Stimulation Reveals Intrinsic Perceptual and Attentional Rhythms

Oscillatory brain activity has functional relevance for perceptual and cognitive processes, as proven by numerous electrophysiology studies accumulating over the years. However, only within the past two decades have researchers been able to study the causal role of such oscillations using transcranial magnetic stimulation (TMS) technology. Two complementary approaches exist. A majority of research employs rhythmic TMS (rTMS) to entrain oscillatory activity and investigate its effect on targeted brain functions. On the other hand, single pulses of TMS (spTMS) that can be delivered with a high spatio-temporal resolution, can be used to precisely probe the state of the system. In this mini-review, we concentrate on this second approach. We argue that, with no a priori hypothesis on the oscillatory frequency of the targeted cortical regions, spTMS can help establish causal links between spontaneous oscillatory activity and perceptual and cognitive functions. Notably, this approach helped to demonstrate that the occipital cortex is periodically involved during specific attentional tasks at the theta (~5 Hz) frequency. We propose that this frequency reflects periodic inter-areal communication for attentional exploration and selection. In the future, clever combination of non-invasive recording and stimulation with well-controlled psychophysics protocols will allow us to further our understanding of the role of brain oscillations for human brain functions.

The neural dynamics of somatosensory processing and adaptation across childhood: a high-density electrical mapping study

Young children are often hyperreactive to somatosensory inputs hardly noticed by adults, as exemplified by irritation to seams or labels in clothing. The neurodevelopmental mechanisms underlying changes in sensory reactivity are not well understood. Based on the idea that neurodevelopmental changes in somatosensory processing and/or changes in sensory adaptation might underlie developmental differences in somatosensory reactivity, high-density electroencephalography was used to examine how the nervous system responds and adapts to repeated vibrotactile stimulation over childhood. Participants aged 6-18 yr old were presented with 50-ms vibrotactile stimuli to the right wrist over the median nerve at 5 blocked interstimulus intervals (ranging from ∼7 to ∼1 stimulus per second). Somatosensory evoked potentials (SEPs) revealed three major phases of activation within the first 200 ms, with scalp topographies suggestive of neural generators in contralateral somatosensory cortex. Although overall SEPs were highly similar for younger, middle, and older age groups (6.1-9.8, 10.0-12.9, and 13.0-17.8 yr old), there were significant age-related amplitude differences in initial and later phases of the SEP. In contrast, robust adaptation effects for fast vs. slow presentation rates were observed that did not differ as a function of age. A greater amplitude response in the later portion of the SEP was observed for the youngest group and may be related to developmental changes in responsivity to somatosensory stimuli. These data suggest the protracted development of the somatosensory system over childhood, whereas adaptation, as assayed in this study, is largely in place by ∼7 yr of age.

Spatial Attention and Temporal Expectation Under Timed Uncertainty Predictably Modulate Neuronal Responses in Monkey V1

The brain uses attention and expectation as flexible devices for optimizing behavioral responses associated with expected but unpredictably timed events. The neural bases of attention and expectation are thought to engage higher cognitive loci; however, their influence at the level of primary visual cortex (V1) remains unknown. Here, we asked whether single-neuron responses in monkey V1 were influenced by an attention task of unpredictable duration. Monkeys covertly attended to a spot that remained unchanged for a fixed period and then abruptly disappeared at variable times, prompting a lever release for reward. We show that monkeys responded progressively faster and performed better as the trial duration increased. Neural responses also followed monkey's task engagement-there was an early, but short duration, response facilitation, followed by a late but sustained increase during the time monkeys expected the attention spot to disappear. This late attentional modulation was significantly and negatively correlated with the reaction time and was well explained by a modified hazard function. Such bimodal, time-dependent changes were, however, absent in a task that did not require explicit attentional engagement. Thus, V1 neurons carry reliable signals of attention and temporal expectation that correlate with predictable influences on monkeys' behavioral responses.

Virtual reality and consciousness inference in dreaming

This article explores the notion that the brain is genetically endowed with an innate virtual reality generator that - through experience-dependent plasticity - becomes a generative or predictive model of the world. This model, which is most clearly revealed in rapid eye movement (REM) sleep dreaming, may provide the theater for conscious experience. Functional neuroimaging evidence for brain activations that are time-locked to rapid eye movements (REMs) endorses the view that waking consciousness emerges from REM sleep - and dreaming lays the foundations for waking perception. In this view, the brain is equipped with a virtual model of the world that generates predictions of its sensations. This model is continually updated and entrained by sensory prediction errors in wakefulness to ensure veridical perception, but not in dreaming. In contrast, dreaming plays an essential role in maintaining and enhancing the capacity to model the world by minimizing model complexity and thereby maximizing both statistical and thermodynamic efficiency. This perspective suggests that consciousness corresponds to the embodied process of inference, realized through the generation of virtual realities (in both sleep and wakefulness). In short, our premise or hypothesis is that the waking brain engages with the world to predict the causes of sensations, while in sleep the brain's generative model is actively refined so that it generates more efficient predictions during waking. We review the evidence in support of this hypothesis - evidence that grounds consciousness in biophysical computations whose neuronal and neurochemical infrastructure has been disclosed by sleep research.

Neural mechanisms of rapid sensitivity to syntactic anomaly

Recent psycholinguistic models hypothesize that anticipatory processing can speed the response to linguistic input during language comprehension by pre-activating representations necessary for word recognition. We investigated the neurocognitive mechanisms of anticipatory processing by recording event-related potentials (ERPs) to syntactically anomalous (The thief was caught byfor police) and well-formed (e.g., The thief was caught bythe police) sentences. One group of participants saw anomalies elicited by the same word in every instance (e.g., for; low-variability stimuli), providing high affordances for predictions about the word-form appearing in the critical position. A second group saw anomalies elicited by seven different prepositions (at, of, on, for, from, over, with; high-variability stimuli) across the study, creating a more difficult prediction task. Syntactic category anomalies enhanced the occipital-temporal N170 component of the ERP, indicating rapid sensitivity – within 200 ms of word-onset – to syntactic anomaly. For low-variability but not the high-variability stimuli, syntactic anomaly also enhanced the earlier occipital-temporal P1 component, around 130 ms after word-onset, indicating that affordances for prediction engendered earlier sensitivity to syntactic anomaly. Independent components analysis revealed three sources within the ERP signal whose functional dynamics were consistent with predictive processing and early responses to syntactic anomaly. Distributed neural source modeling (sLORETA) of these early active sources produced a candidate network for early responses to words during reading in the right posterior occipital, left occipital-temporal, and medial parietal cortex.

Two cognitive and neural systems for endogenous and exogenous spatial attention

Orienting of spatial attention is a family of phylogenetically old mechanisms developed to select information for further processing. Information can be selected via top-down or endogenous mechanisms, depending on the goals of the observers or on the task at hand. Moreover, salient and potentially dangerous events also attract spatial attention via bottom-up or exogenous mechanisms, allowing a rapid and efficient reaction to unexpected but important events. Fronto-parietal brain networks have been demonstrated to play an important role in supporting spatial attentional orienting, although there is no consensus on whether there is a single attentional system supporting both endogenous and exogenous attention, or two anatomical and functionally different attentional systems. In the present paper we review behavioral evidence emphasizing the differential characteristics of both systems, as well as their possible interactions for the control of the final orienting response. Behavioral studies reporting qualitative differences between the effects of both systems as well as double dissociations of the effects of endogenous and exogenous attention on information processing, suggest that they constitute two independent attentional systems, rather than a single one. Recent models of attentional orienting in humans have put forward the hypothesis of a dorsal fronto-parietal network for orienting spatial attention, and a more ventral fronto-parietal network for detecting unexpected but behaviorally relevant events. Non-invasive neurostimulation techniques, as well as neuropsychological data, suggest that endogenous and exogenous attention are implemented in overlapping, although partially segregated, brain circuits. Although more research is needed in order to refine our anatomical and functional knowledge of the brain circuits underlying spatial attention, we conclude that endogenous and exogenous spatial orienting constitute two independent attentional systems, with different behavioral effects, and partially distinct neural substrates.

Intermodal attention modulates visual processing in dorsal and ventral streams

Attending to visual objects while ignoring information from other modalities is necessary for performing difficult visual discriminations, but it is unclear how selecting between sensory modalities alters processing within the visual system. We used an audio-visual intermodal selective attention paradigm with fMRI to study the effects of visual attention on cortical activity in the absence of competitive interactions between multiple visual stimuli. Complex stimuli (faces and words) activated higher visual areas even in the absence of visual attention. These stimulus-dependent activations (SDAs) covered foveal retinotopic cortex, extended ventrally to the anterior fusiform gyrus and dorsally to include multiple distinct foci in the intraparietal sulcus (IPS). Attention amplified the baseline response in posterior retinotopic regions and altered activity in different ways in the extrastriate dorsal and ventral pathways. The majority of the IPS was strongly and exclusively activated by visual attention: attention-related modulations (ARMs) encompassed and spread well beyond the focal SDAs. In contrast, in the fusiform gyrus only a small subset of the regions activated by unattended stimuli showed ARMs. Ventral cortex was also heterogeneous: we found a distinct ventrolateral region in the occipitotemporal sulcus (OTS) that was activated exclusively by attention, showing neither SDAs nor any significant stimulus preferences. Attention-dependent activations in the IPS and the OTS suggest that these regions play critical roles in intermodal visual attention.

Rapid Interactions between Lexical Semantic and Word Form Analysis during Word Recognition in Context: Evidence from ERPs

We used ERPs to investigate the time course of interactions between lexical semantic and sublexical visual word form processing during word recognition. Participants read sentence-embedded pseudowords that orthographically resembled a contextually supported real word (e.g., “She measured the flour so she could bake a ceke…”) or did not (e.g., “She measured the flour so she could bake a tont…”) along with nonword consonant strings (e.g., “She measured the flour so she could bake a srdt…”). Pseudowords that resembled a contextually supported real word (“ceke”) elicited an enhanced positivity at 130 msec (P130), relative to real words (e.g., “She measured the flour so she could bake a cake…”). Pseudowords that did not resemble a plausible real word (“tont”) enhanced the N170 component, as did nonword consonant strings (“srdt”). The effect pattern shows that the visual word recognition system is, perhaps, counterintuitively, more rapidly sensitive to minor than to flagrant deviations from contextually predicted inputs. The findings are consistent with rapid interactions between lexical and sublexical representations during word recognition, in which rapid lexical access of a contextually supported word (CAKE) provides top–down excitation of form features (“cake”), highlighting the anomaly of an unexpected word “ceke.”

Electrostimulation as a prosthesis for repair of information flow in a computer model of neocortex

Damage to a cortical area reduces not only information transmitted to other cortical areas, but also activation of these areas. This phenomenon, whereby the dynamics of a follower area are dramatically altered, is typically manifested as a marked reduction in activity. Ideally, neuroprosthetic stimulation would replace both information and activation. However, replacement of activation alone may be valuable as a means of restoring dynamics and information processing of other signals in this multiplexing system. We used neuroprosthetic stimulation in a computer model of the cortex to repair activation dynamics, using a simple repetitive stimulation to replace the more complex, naturalistic stimulation that had been removed. We found that we were able to restore activity in terms of neuronal firing rates. Additionally, we were able to restore information processing, measured as a restoration of causality between an experimentally recorded signal fed into the in silico brain and a cortical output. These results indicate that even simple neuroprosthetics that do not restore lost information may nonetheless be effective in improving the functionality of surrounding areas of cortex.

Visual attention: the past 25 years

This review focuses on covert attention and how it alters early vision. I explain why attention is considered a selective process, the constructs of covert attention, spatial endogenous and exogenous attention, and feature-based attention. I explain how in the last 25 years research on attention has characterized the effects of covert attention on spatial filters and how attention influences the selection of stimuli of interest. This review includes the effects of spatial attention on discriminability and appearance in tasks mediated by contrast sensitivity and spatial resolution; the effects of feature-based attention on basic visual processes, and a comparison of the effects of spatial and feature-based attention. The emphasis of this review is on psychophysical studies, but relevant electrophysiological and neuroimaging studies and models regarding how and where neuronal responses are modulated are also discussed.

Action understanding and active inference

We have suggested that the mirror-neuron system might be usefully understood as implementing Bayes-optimal perception of actions emitted by oneself or others. To substantiate this claim, we present neuronal simulations that show the same representations can prescribe motor behavior and encode motor intentions during action-observation. These simulations are based on the free-energy formulation of active inference, which is formally related to predictive coding. In this scheme, (generalised) states of the world are represented as trajectories. When these states include motor trajectories they implicitly entail intentions (future motor states). Optimizing the representation of these intentions enables predictive coding in a prospective sense. Crucially, the same generative models used to make predictions can be deployed to predict the actions of self or others by simply changing the bias or precision (i.e. attention) afforded to proprioceptive signals. We illustrate these points using simulations of handwriting to illustrate neuronally plausible generation and recognition of itinerant (wandering) motor trajectories. We then use the same simulations to produce synthetic electrophysiological responses to violations of intentional expectations. Our results affirm that a Bayes-optimal approach provides a principled framework, which accommodates current thinking about the mirror-neuron system. Furthermore, it endorses the general formulation of action as active inference.

Attention, uncertainty, and free-energy

We suggested recently that attention can be understood as inferring the level of uncertainty or precision during hierarchical perception. In this paper, we try to substantiate this claim using neuronal simulations of directed spatial attention and biased competition. These simulations assume that neuronal activity encodes a probabilistic representation of the world that optimizes free-energy in a Bayesian fashion. Because free-energy bounds surprise or the (negative) log-evidence for internal models of the world, this optimization can be regarded as evidence accumulation or (generalized) predictive coding. Crucially, both predictions about the state of the world generating sensory data and the precision of those data have to be optimized. Here, we show that if the precision depends on the states, one can explain many aspects of attention. We illustrate this in the context of the Posner paradigm, using the simulations to generate both psychophysical and electrophysiological responses. These simulated responses are consistent with attentional bias or gating, competition for attentional resources, attentional capture and associated speed-accuracy trade-offs. Furthermore, if we present both attended and non-attended stimuli simultaneously, biased competition for neuronal representation emerges as a principled and straightforward property of Bayes-optimal perception.

Predictive coding or evidence accumulation? False inference and neuronal fluctuations

Perceptual decisions can be made when sensory input affords an inference about what generated that input. Here, we report findings from two independent perceptual experiments conducted during functional magnetic resonance imaging (fMRI) with a sparse event-related design. The first experiment, in the visual modality, involved forced-choice discrimination of coherence in random dot kinematograms that contained either subliminal or periliminal motion coherence. The second experiment, in the auditory domain, involved free response detection of (non-semantic) near-threshold acoustic stimuli. We analysed fluctuations in ongoing neural activity, as indexed by fMRI, and found that neuronal activity in sensory areas (extrastriate visual and early auditory cortex) biases perceptual decisions towards correct inference and not towards a specific percept. Hits (detection of near-threshold stimuli) were preceded by significantly higher activity than both misses of identical stimuli or false alarms, in which percepts arise in the absence of appropriate sensory input. In accord with predictive coding models and the free-energy principle, this observation suggests that cortical activity in sensory brain areas reflects the precision of prediction errors and not just the sensory evidence or prediction errors per se.

The free-energy principle: a unified brain theory?

A free-energy principle has been proposed recently that accounts for action, perception and learning. This Review looks at some key brain theories in the biological (for example, neural Darwinism) and physical (for example, information theory and optimal control theory) sciences from the free-energy perspective. Crucially, one key theme runs through each of these theories - optimization. Furthermore, if we look closely at what is optimized, the same quantity keeps emerging, namely value (expected reward, expected utility) or its complement, surprise (prediction error, expected cost). This is the quantity that is optimized under the free-energy principle, which suggests that several global brain theories might be unified within a free-energy framework.

Searching for targets within the spatial layout of visual short-term memory

Recent studies have revealed that the internal representations that we construct from the environment and maintain in visual short-term memory (VSTM) to guide behavior are highly flexible and can be selectively modulated according to our task goals and expectations. In the current study, we conducted two experiments to compare and contrast neural mechanisms of selective attention related to searching for target items within perceptual versus VSTM representations. We used event-related potentials to investigate whether searching for relevant target items from within VSTM representations involves spatially specific biasing of neural activity in a manner analogous to that which occurs during visual search for target items in perceptual arrays. The results, replicated across the two experiments, revealed that selection of a target object within a search array maintained in VSTM proceeds through a similar mechanism as that in the perceptual domain. In line with previous results, N2pc potentials were obtained when targets were identified within a perceptual visual-search array. Interestingly, equivalent N2pcs, with similar time courses and scalp distributions, were also elicited when target items were identified within a VSTM representation. The findings reinforce the notion of highly flexible VSTM representations that can be modulated according to task goals and suggest a large degree of overlap in the spatially specific neural mechanisms of target selection across the perceptual and VSTM domains.

The free-energy principle: a rough guide to the brain?

This article reviews a free-energy formulation that advances Helmholtz's agenda to find principles of brain function based on conservation laws and neuronal energy. It rests on advances in statistical physics, theoretical biology and machine learning to explain a remarkable range of facts about brain structure and function. We could have just scratched the surface of what this formulation offers; for example, it is becoming clear that the Bayesian brain is just one facet of the free-energy principle and that perception is an inevitable consequence of active exchange with the environment. Furthermore, one can see easily how constructs like memory, attention, value, reinforcement and salience might disclose their simple relationships within this framework.

Intermodal auditory, visual, and tactile attention modulates early stages of neural processing

We used event-related potentials (ERPs) and gamma band oscillatory responses (GBRs) to examine whether intermodal attention operates early in the auditory, visual, and tactile modalities. To control for the effects of spatial attention, we spatially coregistered all stimuli and varied the attended modality across counterbalanced blocks in an intermodal selection task. In each block, participants selectively responded to either auditory, visual, or vibrotactile stimuli from the stream of intermodal events. Auditory and visual ERPs were modulated at the latencies of early cortical processing, but attention manifested later for tactile ERPs. For ERPs, auditory processing was modulated at the latency of the Na (29 msec), which indexes early cortical or thalamocortical processing and the subsequent P1 (90 msec) ERP components. Visual processing was modulated at the latency of the early phase of the C1 (62-72 msec) thought to be generated in the primary visual cortex and the subsequent P1 and N1 (176 msec). Tactile processing was modulated at the latency of the N160 (165 msec) likely generated in the secondary association cortex. Intermodal attention enhanced early sensory GBRs for all three modalities: auditory (onset 57 msec), visual (onset 47 msec), and tactile (onset 27 msec). Together, these results suggest that intermodal attention enhances neural processing relatively early in the sensory stream independent from differential effects of spatial and intramodal selective attention.

Induced Gamma-band Activity Elicited by Visual Representation of Unattended Objects

Object recognition is achieved through neural mechanisms reliant on the activity of distributed neural assemblies that are thought to be coordinated by synchronous firing in the gamma-band range (>20 Hz). An outstanding question focuses on the extent to which the role of gamma oscillations in object recognition is dependent on attention. Attentional mechanisms determine the allocation of perceptual resources to objects in complex scenes biasing the outcome of their mutual competitive interactions. Would object-related enhancements in gamma activity also occur for unattended objects when perceptual resources are traded off to the processing of concurrent visual material? The present electroencephalogram study investigated event-related potentials and evoked (time- and phase-locked) and induced (non-time- and phase-locked to stimulus onset) gamma-band activity (GBA) using a visual discrimination task of low or high perceptual load at fixation. The task was performed while task-irrelevant familiar or unfamiliar objects coappeared in the surrounding central area. Attentional focus was kept at fixation by varying perceptual load between trials; in such conditions, only holistic object processing or low-level perceptual processing, requiring little or no attention, are thought to occur. Although evoked GBA remained unmodulated, induced GBA enhancements, specific to familiar object presentations, were observed, thus providing evidence for cortical visual representation of unattended objects. In addition, the effect was mostly driven by object-specific activity under low load, implying that, in cluttered or complex scenes, attentional selection likely plays a more significant role in object representation.

Low-frequency neuronal oscillations as instruments of sensory selection

Neuroelectric oscillations reflect rhythmic shifting of neuronal ensembles between high and low excitability states. In natural settings, important stimuli often occur in rhythmic streams, and when oscillations entrain to an input rhythm their high excitability phases coincide with events in the stream, effectively amplifying neuronal input responses. When operating in a 'rhythmic mode', attention can use these differential excitability states as a mechanism of selection by simply enforcing oscillatory entrainment to a task-relevant input stream. When there is no low-frequency rhythm that oscillations can entrain to, attention operates in a 'continuous mode', characterized by extended increase in gamma synchrony. We review the evidence for early sensory selection by oscillatory phase-amplitude modulations, its mechanisms and its perceptual and behavioral consequences.

Hierarchical models in the brain

This paper describes a general model that subsumes many parametric models for continuous data. The model comprises hidden layers of state-space or dynamic causal models, arranged so that the output of one provides input to another. The ensuing hierarchy furnishes a model for many types of data, of arbitrary complexity. Special cases range from the general linear model for static data to generalised convolution models, with system noise, for nonlinear time-series analysis. Crucially, all of these models can be inverted using exactly the same scheme, namely, dynamic expectation maximization. This means that a single model and optimisation scheme can be used to invert a wide range of models. We present the model and a brief review of its inversion to disclose the relationships among, apparently, diverse generative models of empirical data. We then show that this inversion can be formulated as a simple neural network and may provide a useful metaphor for inference and learning in the brain.

Models are essential to make sense of scientific data, but they may also play a central role in how we assimilate sensory information. In this paper, we introduce a general model that generates or predicts diverse sorts of data. As such, it subsumes many common models used in data analysis and statistical testing. We show that this model can be fitted to data using a single and generic procedure, which means we can place a large array of data analysis procedures within the same unifying framework. Critically, we then show that the brain has, in principle, the machinery to implement this scheme. This suggests that the brain has the capacity to analyse sensory input using the most sophisticated algorithms currently employed by scientists and possibly models that are even more elaborate. The implications of this work are that we can understand the structure and function of the brain as an inference machine. Furthermore, we can ascribe various aspects of brain anatomy and physiology to specific computational quantities, which may help understand both normal brain function and how aberrant inferences result from pathological processes associated with psychiatric disorders.

Top-down control of human visual cortex by frontal and parietal cortex in anticipatory visual spatial attention

Advance information about an impending stimulus facilitates its subsequent identification and ensuing behavioral responses. This facilitation is thought to be mediated by top-down control signals from frontal and parietal cortex that modulate sensory cortical activity. Here we show, using Granger causality measures on blood oxygen level-dependent time series, that frontal eye field (FEF) and intraparietal sulcus (IPS) activity predicts visual occipital activity before an expected visual stimulus. Top-down levels of Granger causality from FEF and IPS to visual occipital cortex were significantly greater than both bottom-up and mean cortex-wide levels in all individual subjects and the group. In the group and most individual subjects, Granger causality was significantly greater from FEF to IPS than from IPS to FEF, and significantly greater from both FEF and IPS to intermediate-tier than lower-tier ventral visual areas. Moreover, top-down Granger causality from right IPS to intermediate-tier areas was predictive of correct behavioral performance. These results suggest that FEF and IPS modulate visual occipital cortex, and FEF modulates IPS, in relation to visual attention. The current approach may prove advantageous for the investigation of interregional directed influences in other human brain functions.

Temporal characteristics of audiovisual information processing

In complex natural environments, auditory and visual information often have to be processed simultaneously. Previous functional magnetic resonance imaging (fMRI) studies focused on the spatial localization of brain areas involved in audiovisual (AV) information processing, but the temporal characteristics of AV information flow in these regions remained unclear. In this study, we used fMRI and a novel information-theoretic approach to study the flow of AV sensory information. Subjects passively perceived sounds and images of objects presented either alone or simultaneously. Applying the measure of mutual information, we computed for each voxel the latency in which the blood oxygenation level-dependent signal had the highest information content about the preceding stimulus. The results indicate that, after AV stimulation, the earliest informative activity occurs in right Heschl's gyrus, left primary visual cortex, and the posterior portion of the superior temporal gyrus, which is known as a region involved in object-related AV integration. Informative activity in the anterior portion of superior temporal gyrus, middle temporal gyrus, right occipital cortex, and inferior frontal cortex was found at a later latency. Moreover, AV presentation resulted in shorter latencies in multiple cortical areas compared with isolated auditory or visual presentation. The results provide evidence for bottom-up processing from primary sensory areas into higher association areas during AV integration in humans and suggest that AV presentation shortens processing time in early sensory cortices.

Neurophysiology of the BOLD fMRI signal in awake monkeys

BACKGROUND

Simultaneous intracortical recordings of neural activity and blood-oxygen-level-dependent (BOLD) functional magnetic resonance imaging (fMRI) in primary visual cortex of anesthetized monkeys demonstrated varying degrees of correlation between fMRI signals and the different types of neural activity, such as local field potentials (LFPs), multiple-unit activity (MUA), and single-unit activity (SUA). One important question raised by the aforementioned investigation is whether the reported correlations also apply to alert subjects.

RESULTS

Monkeys were trained to perform a fixation task while stimuli within the receptive field of each recording site were used to elicit neural responses followed by a BOLD response. We show -- also in alert behaving monkeys -- that although both LFP and MUA make significant contributions to the BOLD response, LFPs are better and more reliable predictors of the BOLD signal. Moreover, when MUA responses adapt but LFP remains unaffected, the BOLD signal remains unaltered.

CONCLUSIONS

The persistent coupling of the BOLD signal to the field potential when LFP and MUA have different time evolutions suggests that BOLD is primarily determined by the local processing of inputs in a given cortical area. In the alert animal the largest portion of the BOLD signal's variance is explained by an LFP range (20-60 Hz) that is most likely related to neuromodulation. Finally, the similarity of the results in alert and anesthetized subjects indicates that at least in V1 anesthesia is not a confounding factor. This enables the comparison of human fMRI results with a plethora of electrophysiological results obtained in alert or anesthetized animals.

Free-energy and the brain

If one formulates Helmholtz's ideas about perception in terms of modern-day theories one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts. Using constructs from statistical physics it can be shown that the problems of inferring what cause our sensory input and learning causal regularities in the sensorium can be resolved using exactly the same principles. Furthermore, inference and learning can proceed in a biologically plausible fashion. The ensuing scheme rests on Empirical Bayes and hierarchical models of how sensory information is generated. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of the brain's organisation and responses.In this paper, we suggest that these perceptual processes are just one emergent property of systems that conform to a free-energy principle. The free-energy considered here represents a bound on the surprise inherent in any exchange with the environment, under expectations encoded by its state or configuration. A system can minimise free-energy by changing its configuration to change the way it samples the environment, or to change its expectations. These changes correspond to action and perception respectively and lead to an adaptive exchange with the environment that is characteristic of biological systems. This treatment implies that the system's state and structure encode an implicit and probabilistic model of the environment. We will look at models entailed by the brain and how minimisation of free-energy can explain its dynamics and structure.

Feature-based attention modulates orientation-selective responses in human visual cortex

How does feature-based attention modulate neural responses? We used adaptation to quantify the effect of feature-based attention on orientation-selective responses in human visual cortex. Observers were adapted to two superimposed oblique gratings while attending to one grating only. We measured the magnitude of attention-induced orientation-selective adaptation both psychophysically, by the behavioral tilt aftereffect, and physiologically, using fMRI response adaptation. We found evidence for orientation-selective attentional modulation of neuronal responses-a lower fMRI response for the attended than the unattended orientation-in multiple visual areas, and a significant correlation between the magnitude of the tilt aftereffect and that of fMRI response adaptation in V1, the earliest site of orientation coding. These results show that feature-based attention can selectively increase the response of neuronal subpopulations that prefer the attended feature, even when the attended and unattended features are coded in the same visual areas and share the same retinotopic location.

Spatial attention changes excitability of human visual cortex to direct stimulation

Conscious perception depends not only on sensory input, but also on attention [1, 2]. Recent studies in monkeys [3-6] and humans [7-12] suggest that influences of spatial attention on visual awareness may reflect top-down influences on excitability of visual cortex. Here we tested this specifically, by providing direct input into human visual cortex via cortical transcranial magnetic stimulation (TMS) to produce illusory visual percepts, called phosphenes. We found that a lower TMS intensity was needed to elicit a conscious phosphene when its apparent spatial location was attended, rather than unattended. Our results indicate that spatial attention can enhance visual-cortex excitability, and visual awareness, even when sensory signals from the eye via the thalamic pathway are bypassed.

A free energy principle for the brain

By formulating Helmholtz's ideas about perception, in terms of modern-day theories, one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts: using constructs from statistical physics, the problems of inferring the causes of sensory input and learning the causal structure of their generation can be resolved using exactly the same principles. Furthermore, inference and learning can proceed in a biologically plausible fashion. The ensuing scheme rests on Empirical Bayes and hierarchical models of how sensory input is caused. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of cortical organisation and responses. In this paper, we show these perceptual processes are just one aspect of emergent behaviours of systems that conform to a free energy principle. The free energy considered here measures the difference between the probability distribution of environmental quantities that act on the system and an arbitrary distribution encoded by its configuration. The system can minimise free energy by changing its configuration to affect the way it samples the environment or change the distribution it encodes. These changes correspond to action and perception respectively and lead to an adaptive exchange with the environment that is characteristic of biological systems. This treatment assumes that the system's state and structure encode an implicit and probabilistic model of the environment. We will look at the models entailed by the brain and how minimisation of its free energy can explain its dynamics and structure.

Attentional modulation of early-stage visual processing in schizophrenia

This study shows that paying attention to the color of a visual stimulus is manifested by an early endogenous scalp-positive event-related brain potential (ERP) component, referred to as "selection positivity", that emerges within the first 100 ms after stimulus onset in healthy observers. In contrast, recently ill and chronically ill schizophrenia patients as well as patients at high risk for schizophrenia all failed to show this early ERP component while attending to color. These results suggest that a relatively early stage of visual-selective processing in posterior extrastriate cortex is disrupted in schizophrenia.

Sustained attention and apical dendrite activity in recurrent circuits

Recurrent neural activity is a pervasive mode of cortical operations and is believed to underlie cognitive functions of working memory, attention, and the generation of spontaneous activity during sleep . It is proposed here that activity in corticothalamic recurrent circuits underlies the sustaining of attention, and that extended durations of attention are made possible by the stabilizing effects of electrical activity in long apical dendrites of pyramidal neurons. Using the cue-target delay task as a framework, the present paper describes sustained attention during the cue-target delay as activity in recurrent circuits involving layer 5/6 pyramidal neurons. At target onset, persistent activity in apical dendrites of layer 2/3 pyramidal neurons (projected from the recurrent circuits) can enhance the processing of incoming pulse trains at basal dendrites. Apical dendrite activity is assumed to modulate the soma processing of layer 2/3 and layer 5/6 pyramidal neurons at subthreshold voltage levels. The variability of successive soma depolarizations from the apical dendrite strongly influences the stability of activity in the corticothalamic recurrent circuit. Lower variability promotes higher stability. According to the present model of apical dendrite function, soma depolarizations can be reduced in variability and maintained within subthreshold levels by increasing the distance that EPSPs propagate along the apical dendrite. The close relationship between sustained attention and the electrical field potentials produced by repeated EPSP propagations in apical dendrites is supported in a brief review of sustained attention experiments that have employed measures of EEG, ERS/ERD, ERP, and LFP.

Covert attention increases contrast sensitivity: Psychophysical, neurophysiological and neuroimaging studies

This chapter focuses on the effect of covert spatial attention on contrast sensitivity, a basic visual dimension where the best mechanistic understanding of attention has been achieved. I discuss how models of contrast sensitivity, as well as the confluence of psychophysical, single-unit recording, and neuroimaging studies, suggest that attention increases contrast sensitivity via contrast gain, an effect akin to a change in the physical contrast stimulus. I suggest possible research directions and ways to strengthen the interaction among different levels of analysis to further our understanding of visual attention.

Spatiotemporal dynamics of human object recognition processing: an integrated high-density electrical mapping and functional imaging study of "closure" processes

Humans are capable of recognizing objects, often despite highly adverse viewing conditions (e.g., occlusion). The term "perceptual closure" has been used to refer to the neural processes responsible for "filling-in" missing information in the visual image under such conditions. Closure phenomena have been linked to a group of object recognition areas, the so-called lateral-occipital complex (LOC). Here, we investigated the spatiotemporal dynamics of perceptual closure processes by coregistering data from high-density electrical recordings (ERPs) and functional magnetic resonance imaging (fMRI) while subjects participated in a perceptual closure task. Subjects were presented with highly fragmented images and control scrambled images. Fragmented images were calibrated to be 'just' recognizable as objects (that is, perceptual closure was necessary), whereas the scrambled images were unrecognizable. Comparison of responses to these two stimulus classes revealed the neural processes underlying perceptual closure. fMRI revealed an object recognition system that mediates these closure processes, the core of which consists of the LOC regions. ERP recordings resulted in the well-characterized N(CL) component (for negativity associated with closure), a robust relative negativity over bilateral occipito-temporal scalp that occurs in the 230-400 ms timeframe. Our investigations further revealed an extended network of dorsal and frontal regions, also involved in perceptual closure processes. Inverse source analysis showed that the major generators of N(CL) localized to the identical regions within LOC revealed by the fMRI recordings and detailed the temporal dynamics across these LOC regions including interactions between LOC and these other nodes of the object recognition circuit.

Biasing the brain's attentional set: II. effects of selective intersensory attentional deployments on subsequent sensory processing

This study used high-density mapping of human event-related potentials to examine the brain activity associated with selective information processing when subjects were cued on a trial-by-trial basis to perform a discrimination in either the visual or auditory modality. On each trial, word-cues (S1) instructed subjects to attend to features within one sensory-modality of an impending compound auditory-visual stimulus (S2) that arrived approximately 1-second following the cue. Subjects made a discrimination within the cued modality of the S2 stimulus. The spatio-temporal patterns of activity in response to the compound S2 stimulus were examined as a function of the sensory modality being attended. The earliest effects of intersensory attention on visual processing were seen subsequent to the initial activation of visual cortex, beginning at 80 ms and continuing into the P1 and N1 components of the visual ERP. The scalp-topography of this earliest modulation was consistent with modulation of activity in ventral visual stream areas. Thus, the locus of effects on visual S2 processing differed from the anticipatory parieto-occipital biasing activity that preceded S2 presentation. This pattern of effects strongly suggests that the anticipatory activity (following the cue) associated with sustaining the focus of attention during intersensory attention, at least in the context of this paradigm, does not operate as a simple gain mechanism in early visual sensory areas. Rather, attentional biasing can operate through a higher-order process whereby parieto-occipital cortices influence the subsequent flow of visual processing in the ventral stream.

Multisensory contributions to low-level, ‘unisensory’ processing

Neurobiologists have traditionally assumed that multisensory integration is a higher order process that occurs after sensory signals have undergone extensive processing through a hierarchy of unisensory subcortical and cortical regions. Recent findings, however, question this assumption. Studies in humans, nonhuman primates and other species demonstrate multisensory convergence in low level cortical structures that were generally believed to be unisensory in function. In addition to enriching current models of multisensory processing and perceptual functions, these new findings require a revision in our thinking about unisensory processing in low level cortical areas.

EVENT BASED SELF-SUPERVISED TEMPORAL INTEGRATION FOR MULTIMODAL SENSOR DATA

A method for synergistic integration of multimodal sensor data is proposed in this paper. This method is based on two aspects of the integration process: (1) achieving synergistic integration of two or more sensory modalities, and (2) fusing the various information streams at particular moments during processing. Inspired by psychophysical experiments, we propose a self-supervised learning method for achieving synergy with combined representations. Evidence from temporal registration and binding experiments indicates that different cues are processed individually at specific time intervals. Therefore, an event-based temporal co-occurrence principle is proposed for the integration process. This integration method was applied to a mobile robot exploring unfamiliar environments. Simulations showed that integration enhanced route recognition with many perceptual similarities; moreover, they indicate that a perceptual hierarchy of knowledge about instant movement contributes significantly to short-term navigation, but that visual perceptions have bigger impact over longer intervals.

Transient attention enhances perceptual performance and FMRI response in human visual cortex

When a visual stimulus suddenly appears, it captures attention, producing a transient improvement of performance on basic visual tasks. We investigate the effect of transient attention on stimulus representations in early visual areas using rapid event-related fMRI. Participants discriminated the orientation of one of two gratings preceded or followed by a nonpredictive peripheral cue. Compared to control conditions, precueing the target location improved performance and produced a larger fMRI response in corresponding retinotopic areas. This enhancement progressively increased from striate to extrastriate areas. Control conditions indicated that the enhanced fMRI response was not due to sensory summation of cue and target signals. Thus, an uninformative precue increases both perceptual performance and the concomitant stimulus-evoked activity in early visual areas. These results provide evidence regarding the retinotopically specific neural correlate for the effects of transient attention on early vision.

A feedback model of visual attention

Feedback connections are a prominent feature of cortical anatomy and are likely to have a significant functional role in neural information processing. We present a neural network model of cortical feedback that successfully simulates neurophysiological data associated with attention. In this domain, our model can be considered a more detailed, and biologically plausible, implementation of the biased competition model of attention. However, our model is more general as it can also explain a variety of other top-down processes in vision, such as figure/ground segmentation and contextual cueing. This model thus suggests that a common mechanism, involving cortical feedback pathways, is responsible for a range of phenomena and provides a unified account of currently disparate areas of research.

Task relevance enhances early transient and late slow-wave activity of distributed cortical sources

The primary purpose of these studies was to link together concepts related to attention/working memory and feedforward/feedback activity using MEG response profiles obtained in humans. Similar to recent studies of attention in monkeys, we show early "spike-like" activity (<200 ms poststimulus), most likely reflecting an early transient excitatory response mixed with feedback influences, followed by "slow-wave" activity (>200 ms poststimulus) in MEG cortical response profiles evoked by a visual working memory task. We experimentally dissociated the early transient activity from the later sustained activity (predominantly feedback) by conducting an auditory size classification task. Words, representing common objects, evoked activity in occipital cortex (presumably due to imagery) even though visual stimuli were not present in this task. The initial "spike" was absent from the response profile obtained from occipital cortex, leaving only "slow-wave" activity, thereby allowing us to characterize or profile feedback activity in this situation. Attention or task relevance enhanced the initial "spike" and "slow-wave" activity in visually responsive areas. Prefrontal activity, along the superior frontal sulcus, evoked by the working memory task, was active later in time than initial activity in visual cortex and later than the earliest effect of attention modulation in visual cortex.

Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing

Recent findings in both monkeys and humans indicate that multisensory convergence occurs in low-level cortical structures generally believed to be unisensory in function. In an in-depth treatment of this theme, this paper reviews anatomical and physiological findings relating to the convergence of visual, somatosensory and auditory signals at early stages of auditory cortical processing. We discuss the potential anatomical sources of the input, and the types of known projections, and attempt to integrate this information with the current hierarchical model of auditory processing. Finally, we consider the functional implications of multisensory integration in early sensory processing.