Neural correlates of perceptual narrowing in cross-species face-voice matching

Intermodulation frequencies reveal common neural assemblies integrating facial and vocal fearful expressions

Effective social communication depends on the integration of emotional expressions coming from the face and the voice. Although there are consistent reports on how seeing and hearing emotion expressions can be automatically integrated, direct signatures of multisensory integration in the human brain remain elusive. Here we implemented a multi-input electroencephalographic (EEG) frequency tagging paradigm to investigate neural populations integrating facial and vocal fearful expressions. High-density EEG was acquired in participants attending to dynamic fearful facial and vocal expressions tagged at different frequencies (fvis, faud). Beyond EEG activity at the specific unimodal facial and vocal emotion presentation frequencies, activity at intermodulation frequencies (IM) arising at the sums and differences of the harmonics of the stimulation frequencies (mfvis ± nfaud) were observed, suggesting non-linear integration of the visual and auditory emotion information into a unified representation. These IM provide evidence that common neural populations integrate signal from the two sensory streams. Importantly, IMs were absent in a control condition with mismatched facial and vocal emotion expressions. Our results provide direct evidence from non-invasive recordings in humans for common neural populations that integrate fearful facial and vocal emotional expressions.

The emergence of the multisensory brain: From the womb to the first steps

The becoming of the human being is a multisensory process that starts in the womb. By integrating spontaneous neuronal activity with inputs from the external world, the developing brain learns to make sense of itself through multiple sensory experiences. Over the past ten years, advances in neuroimaging and electrophysiological techniques have allowed the exploration of the neural correlates of multisensory processing in the newborn and infant brain, thus adding an important piece of information to behavioral evidence of early sensitivity to multisensory events. Here, we review recent behavioral and neuroimaging findings to document the origins and early development of multisensory processing, particularly showing that the human brain appears naturally tuned to multisensory events at birth, which requires multisensory experience to fully mature. We conclude the review by highlighting the potential uses and benefits of multisensory interventions in promoting healthy development by discussing emerging studies in preterm infants.

Calibrating Vision: Concepts and Questions

The idea that visual coding and perception are shaped by experience and adjust to changes in the environment or the observer is universally recognized as a cornerstone of visual processing, yet the functions and processes mediating these calibrations remain in many ways poorly understood. In this article we review a number of facets and issues surrounding the general notion of calibration, with a focus on plasticity within the encoding and representational stages of visual processing. These include how many types of calibrations there are - and how we decide; how plasticity for encoding is intertwined with other principles of sensory coding; how it is instantiated at the level of the dynamic networks mediating vision; how it varies with development or between individuals; and the factors that may limit the form or degree of the adjustments. Our goal is to give a small glimpse of an enormous and fundamental dimension of vision, and to point to some of the unresolved questions in our understanding of how and why ongoing calibrations are a pervasive and essential element of vision.

Visual attention for linguistic and non-linguistic body actions in non-signing and native signing children

Evidence from adult studies of deaf signers supports the dissociation between neural systems involved in processing visual linguistic and non-linguistic body actions. The question of how and when this specialization arises is poorly understood. Visual attention to these forms is likely to change with age and be affected by prior language experience. The present study used eye-tracking methodology with infants and children as they freely viewed alternating video sequences of lexical American sign language (ASL) signs and non-linguistic body actions (self-directed grooming action and object-directed pantomime). In Experiment 1, we quantified fixation patterns using an area of interest (AOI) approach and calculated face preference index (FPI) values to assess the developmental differences between 6 and 11-month-old hearing infants. Both groups were from monolingual English-speaking homes with no prior exposure to sign language. Six-month-olds attended the signer's face for grooming; but for mimes and signs, they were drawn to attend to the "articulatory space" where the hands and arms primarily fall. Eleven-month-olds, on the other hand, showed a similar attention to the face for all body action types. We interpret this to reflect an early visual language sensitivity that diminishes with age, just before the child's first birthday. In Experiment 2, we contrasted 18 hearing monolingual English-speaking children (mean age of 4.8 years) vs. 13 hearing children of deaf adults (CODAs; mean age of 5.7 years) whose primary language at home was ASL. Native signing children had a significantly greater face attentional bias than non-signing children for ASL signs, but not for grooming and mimes. The differences in the visual attention patterns that are contingent on age (in infants) and language experience (in children) may be related to both linguistic specialization over time and the emerging awareness of communicative gestural acts.

A Bayesian meta-analysis of infants' ability to perceive audio-visual congruence for speech

This paper quantifies the extent to which infants can perceive audio-visual congruence for speech information and assesses whether this ability changes with native language exposure over time. A hierarchical Bayesian robust regression model of 92 separate effect sizes extracted from 24 studies indicates a moderate effect size in a positive direction (0.35, CI [0.21: 0.50]). This result suggests that infants possess a robust ability to detect audio-visual congruence for speech. Moderator analyses, moreover, suggest that infants' audio-visual matching ability for speech emerges at an early point in the process of language acquisition and remains stable for both native and non-native speech throughout early development. A sensitivity analysis of the meta-analytic data, however, indicates that a moderate publication bias for significant results could shift the lower credible interval to include null effects. Based on these findings, we outline recommendations for new lines of enquiry and suggest ways to improve the replicability of results in future investigations.

Babies know bad dancing when they see it: Older but not younger infants discriminate between synchronous and asynchronous audiovisual musical displays

Movement to music is a universal human behavior, yet little is known about how observers perceive audiovisual synchrony in complex musical displays such as a person dancing to music, particularly during infancy and childhood. In the current study, we investigated how perception of musical audiovisual synchrony develops over the first year of life. We habituated infants to a video of a person dancing to music and subsequently presented videos in which the visual track was matched (synchronous) or mismatched (asynchronous) with the audio track. In a visual-only control condition, we presented the same visual stimuli with no sound. In Experiment 1, we found that older infants (8-12months) exhibited a novelty preference for the mismatched movie when both auditory information and visual information were available and showed no preference when only visual information was available. By contrast, younger infants (5-8months) in Experiment 2 did not discriminate matching stimuli from mismatching stimuli. This suggests that the ability to perceive musical audiovisual synchrony may develop during the second half of the first year of infancy.

How Does Experience Shape Early Development? Considering the Role of Top-Down Mechanisms

Perceptual development requires infants to adapt their perceptual systems to the structures and statistical information of their environment. In this way, perceptual development is not only important in its own right, but is a case study for behavioral and neural plasticity-powerful mechanisms that have the potential to support developmental change in numerous domains starting early in life. While it is widely assumed that perceptual development is a bottom-up process, where simple exposure to sensory input modifies perceptual representations starting early in the perceptual system, there are several critical phenomena in this literature that cannot be explained with an exclusively bottom-up model. This chapter proposes a complementary mechanism where nascent top-down information, feeding back from higher-level regions of the brain, helps to guide perceptual development. Supporting this theoretical proposal, recent behavioral and neuroimaging studies have established that young infants already have the capacity to engage in top-down modulation of their perceptual systems.

Listening to the calls of the wild: The role of experience in linking language and cognition in young infants

Well before they understand their first words, infants have begun to link language and cognition. This link is initially broad: At 3months, listening to both human and nonhuman primate vocalizations supports infants' object categorization, a building block of cognition. But by 6months, the link has narrowed: Only human vocalizations support categorization. What mechanisms underlie this rapid tuning process? Here, we document the crucial role of infants' experience as infants tune this link to cognition. Merely exposing infants to nonhuman primate vocalizations permits them to preserve, rather than sever, the link between these signals and categorization. Exposing infants to backward speech-a signal that fails to support categorization in the first year of life-does not have this advantage. This new evidence illuminates the central role of early experience as infants specify which signals, from an initially broad set, they will continue to link to core cognitive capacities.

Tuning the developing brain to emotional body expressions

Reading others' emotional body expressions is an essential social skill. Adults readily recognize emotions from body movements. However, it is unclear when in development infants become sensitive to bodily expressed emotions. We examined event-related brain potentials (ERPs) in 4- and 8-month-old infants in response to point-light displays (PLDs) of happy and fearful body expressions presented in two orientations (upright and inverted). The ERP results revealed that 8-month-olds but not 4-month-olds respond sensitively to the orientation and the emotion of the dynamic expressions. Specifically, 8-month-olds showed (i) an early (200-400 ms) orientation-sensitive positivity over frontal and central electrodes, and (ii) a late (700-1100 ms) emotion-sensitive positivity over temporal and parietal electrodes in the right hemisphere. These findings suggest that orientation-sensitive and emotion-sensitive brain processes, distinct in timing and topography, develop between 4 and 8 months of age.

Developmental and individual differences in the neural processing of dynamic expressions of pain and anger

We examined the processing of facial expressions of pain and anger in 8-month-old infants and adults by measuring event-related brain potentials (ERPs) and frontal EEG alpha asymmetry. The ERP results revealed that while adults showed a late positive potential (LPP) to emotional expressions that was enhanced to pain expressions, reflecting increased evaluation and emotional arousal to pain expressions, infants showed a negative component (Nc) to emotional expressions that was enhanced to angry expressions, reflecting increased allocation of attention to angry faces. Moreover, infants and adults showed opposite patterns in their frontal asymmetry responses to pain and anger, suggesting developmental differences in the motivational processes engendered by these facial expressions. These findings are discussed in the light of associated individual differences in infant temperament and adult dispositional empathy.

Early experience and multisensory perceptual narrowing

Perceptual narrowing reflects the effects of early experience and contributes in key ways to perceptual and cognitive development. Previous studies have found that unisensory perceptual sensitivity in young infants is broadly tuned such that they can discriminate native as well as non-native sensory inputs but that it is more narrowly tuned in older infants such that they only respond to native inputs. Recently, my coworkers and I discovered that multisensory perceptual sensitivity narrows as well. The present article reviews this new evidence in the general context of multisensory perceptual development and the effects of early experience. Together, the evidence on unisensory and multisensory narrowing shows that early experience shapes the emergence of perceptual specialization and expertise.

A unified coding strategy for processing faces and voices

Both faces and voices are rich in socially-relevant information, which humans are remarkably adept at extracting, including a person's identity, age, gender, affective state, personality, etc. Here, we review accumulating evidence from behavioral, neuropsychological, electrophysiological, and neuroimaging studies which suggest that the cognitive and neural processing mechanisms engaged by perceiving faces or voices are highly similar, despite the very different nature of their sensory input. The similarity between the two mechanisms likely facilitates the multi-modal integration of facial and vocal information during everyday social interactions. These findings emphasize a parsimonious principle of cerebral organization, where similar computational problems in different modalities are solved using similar solutions.