Seeing sounds and hearing sights: the influence of prior learning on current perception

The eye movement measure of memory and its relationship with explicit measures

This study examined whether the eye movement can be used to measure memory of past events and its relationship with the explicit measures. In Experiment 1, after studying a list of Chinese characters, the participants received a recognition memory test. For each trial the participants had to indicate, among one studied character and two nonstudied homonyms, which character they had studied. Participants' eye movements were monitored while they viewed the three-character test display. Both the time-course and response-locked measures showed that participants viewed the studied character longer than the nonstudied character regardless of their explicit response. Experiment 2 used a wagering task to assess participants' conscious awareness and found that wagering points predicted viewing time for the target better than the recognition accuracy did. These findings suggest that the effect of memory on viewing time occurs automatically and is weakly associated with subsequent conscious awareness of the studied event.

A Bayesian perspective on sensory and cognitive integration in pain perception and placebo analgesia

The placebo effect is a component of any response to a treatment (effective or inert), but we still ignore why it exists. We propose that placebo analgesia is a facet of pain perception, others being the modulating effects of emotions, cognition and past experience, and we suggest that a computational understanding of pain may provide a unifying explanation of these phenomena. Here we show how Bayesian decision theory can account for such features and we describe a model of pain that we tested against experimental data. Our model not only agrees with placebo analgesia, but also predicts that learning can affect pain perception in other unexpected ways, which experimental evidence supports. Finally, the model can also reflect the strategies used by pain perception, showing that modulation by disparate factors is intrinsic to the pain process.

Association with emotional information alters subsequent processing of neutral faces

The processing of emotional as compared to neutral information is associated with different patterns in eye movement and neural activity. However, the ‘emotionality’ of a stimulus can be conveyed not only by its physical properties, but also by the information that is presented with it. There is very limited work examining the how emotional information may influence the immediate perceptual processing of otherwise neutral information. We examined how presenting an emotion label for a neutral face may influence subsequent processing by using eye movement monitoring (EMM) and magnetoencephalography (MEG) simultaneously. Participants viewed a series of faces with neutral expressions. Each face was followed by a unique negative or neutral sentence to describe that person, and then the same face was presented in isolation again. Viewing of faces paired with a negative sentence was associated with increased early viewing of the eye region and increased neural activity between 600 and 1200 ms in emotion processing regions such as the cingulate, medial prefrontal cortex, and amygdala, as well as posterior regions such as the precuneus and occipital cortex. Viewing of faces paired with a neutral sentence was associated with increased activity in the parahippocampal gyrus during the same time window. By monitoring behavior and neural activity within the same paradigm, these findings demonstrate that emotional information alters subsequent visual scanning and the neural systems that are presumably invoked to maintain a representation of the neutral information along with its emotional details.

The perceptual nature of audiovisual interactions for semantic knowledge in young and elderly adults

Audiovisual interactions for familiar objects are at the core of perception. The nature of these interactions depends on the amodal--sensory abstracted--or modal--sensory-dependent--approach of knowledge. According to these approaches, the interactions should be respectively semantic and indirect or perceptual and direct. This issue is therefore a central question to memory and perception, yet the nature of these interactions remains unexplored in young and elderly adults. We used a cross-modal priming paradigm combined with a visual masking procedure of half of the auditory primes. The data demonstrated similar results in the young and elderly adult groups. The mask interfered with the priming effect in the semantically congruent condition, whereas the mask facilitated the processing of the visual target in the semantically incongruent condition. These findings indicate that audiovisual interactions are perceptual, and support the grounded cognition theory.

The hippocampus supports multiple cognitive processes through relational binding and comparison

It has been well established that the hippocampus plays a pivotal role in explicit long-term recognition memory. However, findings from amnesia, lesion and recording studies with non-human animals, eye-movement recording studies, and functional neuroimaging have recently converged upon a similar message: the functional reach of the hippocampus extends far beyond explicit recognition memory. Damage to the hippocampus affects performance on a number of cognitive tasks including recognition memory after short and long delays and visual discrimination. Additionally, with the advent of neuroimaging techniques that have fine spatial and temporal resolution, findings have emerged that show the elicitation of hippocampal responses within the first few 100 ms of stimulus/task onset. These responses occur for novel and previously viewed information during a time when perceptual processing is traditionally thought to occur, and long before overt recognition responses are made. We propose that the hippocampus is obligatorily involved in the binding of disparate elements across both space and time, and in the comparison of such relational memory representations. Furthermore, the hippocampus supports relational binding and comparison with or without conscious awareness for the relational representations that are formed, retrieved and/or compared. It is by virtue of these basic binding and comparison functions that the reach of the hippocampus extends beyond long-term recognition memory and underlies task performance in multiple cognitive domains.

Worth a glance: using eye movements to investigate the cognitive neuroscience of memory

Results of several investigations indicate that eye movements can reveal memory for elements of previous experience. These effects of memory on eye movement behavior can emerge very rapidly, changing the efficiency and even the nature of visual processing without appealing to verbal reports and without requiring conscious recollection. This aspect of eye movement based memory investigations is particularly useful when eye movement methods are used with special populations (e.g., young children, elderly individuals, and patients with severe amnesia), and also permits use of comparable paradigms in animals and humans, helping to bridge different memory literatures and permitting cross-species generalizations. Unique characteristics of eye movement methods have produced findings that challenge long-held views about the nature of memory, its organization in the brain, and its failures in special populations. Recently, eye movement methods have been successfully combined with neuroimaging techniques such as fMRI, single-unit recording, and magnetoencephalography, permitting more sophisticated investigations of memory. Ultimately, combined use of eye-tracking with neuropsychological and neuroimaging methods promises to provide a more comprehensive account of brain-behavior relationships and adheres to the "converging evidence" approach to cognitive neuroscience.

Cortical networks representing object categories and high-level attributes of familiar real-world action sounds

In contrast to visual object processing, relatively little is known about how the human brain processes everyday real-world sounds, transforming highly complex acoustic signals into representations of meaningful events or auditory objects. We recently reported a fourfold cortical dissociation for representing action (nonvocalization) sounds correctly categorized as having been produced by human, animal, mechanical, or environmental sources. However, it was unclear how consistent those network representations were across individuals, given potential differences between each participant's degree of familiarity with the studied sounds. Moreover, it was unclear what, if any, auditory perceptual attributes might further distinguish the four conceptual sound-source categories, potentially revealing what might drive the cortical network organization for representing acoustic knowledge. Here, we used functional magnetic resonance imaging to test participants before and after extensive listening experience with action sounds, and tested for cortices that might be sensitive to each of three different high-level perceptual attributes relating to how a listener associates or interacts with the sound source. These included the sound's perceived concreteness, effectuality (ability to be affected by the listener), and spatial scale. Despite some variation of networks for environmental sounds, our results verified the stability of a fourfold dissociation of category-specific networks for real-world action sounds both before and after familiarity training. Additionally, we identified cortical regions parametrically modulated by each of the three high-level perceptual sound attributes. We propose that these attributes contribute to the network-level encoding of category-specific acoustic knowledge representations.

A complementary analytic approach to examining medial temporal lobe sources using magnetoencephalography

Neuropsychological and neuroimaging findings reveal that the hippocampus is important for recognition memory. However, it is unclear when and whether the hippocampus contributes differentially to recognition of previously studied items (old) versus novel items (new), or contributes to a general processing requirement that is necessary for recognition of both types of information. To address this issue, we examined the temporal dynamics and spectral frequency underlying hippocampal activity during recognition of old/new complex scenes using magnetoencephalography (MEG). In order to provide converging evidence to existing literature in support of the potential of MEG to localize the hippocampus, we reconstructed brain source activity using the beamformer method and analyzed three types of processing-related signal changes by applying three different analysis methods: (1) Synthetic aperture magnetometry (SAM) revealed event related and non-event-related spectral power changes; (2) Inter-trial coherence (ITC) revealed time-locked changes in neural synchrony; and (3) Event-related SAM (ER-SAM) revealed averaged event-related responses over time. Hippocampal activity was evident for both old and new information within the theta frequency band and during the first 250 ms following stimulus onset. The early onset of hippocampal responses suggests that general comparison processes related to recognition of new/old information may occur obligatorily.

MEG correlates of bimodal encoding of faces and persons' names

Learning associations between people's faces and names is a universal cognitive function with important social implications. The goal of the present study was to examine brain activity patterns associated with cross-modal encoding of names and faces. Learning face-name pairs was compared to unimodal learning tasks using the same visual and auditory stimuli. Spatiotemporal brain activation profiles were obtained with magnetoencephalography using an automated source estimation method. Results showed activation foci in left (for names) and right (for faces) temporal lobe perisylvian cortices, predominantly right-hemisphere occipital and occipitotemporal regions (for faces), and right hemisphere dorsolateral prefrontal regions during the encoding phase for both types of stimuli presented in isolation. Paired (face-name) stimulus presentation elicited bilateral prefrontal and temporal lobe perisylvian activity for faces and enhanced visual cortex activation in response to names (compared to names in the unpaired condition). These findings indicate distinct patterns of brain activation during the formation of associations between meaningful visual and auditory stimuli.

The obligatory effects of memory on eye movements

Previous work has shown that eye movement behaviour is affected by previous experience, such that alterations in viewing patterns can be observed to previously viewed compared to novel displays. The current work addresses the extent to which such effects of memory on eye movement behaviour are obligatory; that is, we examined whether prior experience could alter subsequent eye movement behaviour under a variety of testing conditions, for stimuli that varied on the nature of the prior exposure. While task demands influenced whether viewing was predominantly directed to the novel versus familiar faces, viewing of the familiar faces was distinguished from viewing of the novel faces, regardless of whether the task required incidental encoding or intentional retrieval. Changes in scanning of previously viewed over novel faces emerged early in viewing; in particular, viewing duration of the first fixation to the familiar faces was often significantly different from the duration of the first fixation directed to the novel faces, regardless of whether prior exposure was solely in the context of the experiment or due to real-world exposure. These findings suggest that representations maintained in memory may be retrieved and compared with presented information obligatorily.