Infants' ability to match dynamic phonetic and gender information in the face and voice

Gendered speech development in early childhood: Evidence from a longitudinal study of vowel and consonant acoustics

Adults rate the speech of children assigned male at birth (AMAB) and assigned female at birth (AFAB) as young as 2.5 years of age differently on a scale of definitely a boy to definitely a girl (Munson et al., 2022), despite the lack of consistent sex dimorphism in children's speech production mechanisms. This study used longitudinal data to examine the acoustic differences between AMAB and AFAB children and the association between the acoustic measures and perceived gender ratings of children's speech. We found differences between AMAB and AFAB children in two acoustic parameters that mark gender in adult speech: the spectral centroid of /s/ and the overall scaling of resonant frequencies in vowels. These results demonstrate that children as young as 3 years old speak in ways that reflect their sex assigned at birth. We interpret this as evidence that children manipulate their speech apparatus volitionally to mark gender through speech.

Integration of audiovisual speech perception: From infancy to older adults

One of the most prevalent and relevant social experiences for humans - engaging in face-to-face conversations - is inherently multimodal. In the context of audiovisual (AV) speech perception, the visual cues from the speaker's face play a crucial role in language acquisition and in enhancing our comprehension of incoming auditory speech signals. Nonetheless, AV integration reflects substantial individual differences, which cannot be entirely accounted for by the information conveyed through the speech signal or the perceptual abilities of the individual. These differences illustrate changes in response to experience with auditory and visual sensory processing across the lifespan, and within a phase of life. To improve our understanding of integration of AV speech, the current work offers a perspective for understanding AV speech processing in relation to AV perception in general from a prelinguistic and a linguistic viewpoint, and by looking at AV perception through the lens of humans as Bayesian observers implementing a causal inference model. This allowed us to create a cohesive approach to look at differences and similarities of AV integration from infancy to older adulthood. Behavioral and neurophysiological evidence suggests that both prelinguistic and linguistic mechanisms exhibit distinct, yet mutually influential, effects across the lifespan within and between individuals.

Visuo-haptic processing of unfamiliar shapes: Comparing children and adults

The question of how our sensory perception abilities develop has been an active area of research, establishing trajectories of development from infancy that last well into late childhood and even adolescence. In this context, several studies have established changes in sensory processing of vision and touch around the age of 8 to 9 years. In this experiment, we explored the visual and haptic perceptual development of elementary school children of ages 6-11 in similarity-rating tasks of unfamiliar objects and compared their performance to adults. The participants were presented with parametrically-defined objects to be explored haptically and visually in separate groups for both children and adults. Our results showed that the raw similarity ratings of the children had more variability compared to adults. A detailed multidimensional scaling analysis revealed that the reconstructed perceptual space of the adult haptic group was significantly closer to the parameter space compared to the children group, whereas both groups' visual perceptual space was similarly well reconstructed. Beyond this, however, we found no clear evidence for an age effect in either modality within the children group. These results suggest that haptic processing of unfamiliar, abstract shapes may continue to develop beyond the age of 11 years later into adolescence.

The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy

In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.

Individual Differences in the Development of Gendered Speech in Preschool Children: Evidence From a Longitudinal Study

PURPOSE

We evaluated whether naive listeners' ratings of the gender typicality of the speech of children assigned male at birth (AMAB) and children assigned female at birth (AFAB) were different at two time points: one at which children were 2.5-3.5 years old and one when they were 4.5-5.5 years old. We also examined whether measures of speech, language, and inhibitory control predicted developmental changes in these ratings.

METHOD

A group of adults (N = 80) rated single-word productions of 55 AMAB and 55 AFAB children on a continuous scale from "definitely a boy" to "definitely a girl." Children's productions were taken from previous longitudinal study of phonological development and vocabulary growth. As part of that study, children completed a battery of standardized and nonstandardized tests at both time points.

RESULTS

Listener ratings for AMAB and AFAB children were significantly different at both time points. The difference was larger at the later time point, and this was due entirely to changes in the ratings of AMAB children's speech. A measure of language production and a measure of inhibitory control predicted developmental changes in these ratings, albeit only weakly, and not in a consistent direction.

CONCLUSIONS

The gender typicality of AMAB and AFAB children's speech is perceptibly different for children as young as 2.5 years old. Developmental changes in perceived gender typicality are driven by changes in the speech of AMAB children. The learning of gendered speech is not constrained or facilitated by overall speech and language skill.

Perceiving gender while perceiving language: Integrating psycholinguistics and gender theory

There is a substantial body of literature showing that men and women speak differently and that these differences are endemic to the speech signal. However, the psycholinguistic mechanisms underlying the integration of social category perception and language are still poorly understood. Speaker attributes such as emotional state, age, sex, and race have often been treated in the literature as dissociable, but perceptual systems for social categories demonstrably rely on interdependent cognitive processes. We introduce a diversity science framework for evaluating the existing literature on gender and speech perception, arguing that differences in beliefs about gender may be defined as differences in beliefs about differences. Treating individual, group, and societal level contrasts in ideological patterns as phenomenologically distinctive, we enumerate six ideological arenas which define claims about gender and examine the literature for treatment of these issues. We argue that both participants and investigators predictably show evidence of differences in ideological attitudes toward the normative definition of persons. The influence of social knowledge on linguistic perception therefore occurs in the context of predictable variation in both attention and inattention to people and the distinguishing features which mark them salient as kinds. We link experiences of visibility, invisibility, and hypervisibility with ideological variation regarding the significance of physiological, linguistic, and social features, concluding that gender ideologies are implicated both in linguistic processing and in social judgments of value between groups. We conclude with a summary of the key gaps in the current literature and recommendations for best practices studies that may use in future investigations of socially meaningful variation in speech perception. This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Language Linguistics > Language Acquisition Psychology > Perception and Psychophysics.

A Bayesian meta-analysis of infants' ability to perceive audio-visual congruence for speech

This paper quantifies the extent to which infants can perceive audio-visual congruence for speech information and assesses whether this ability changes with native language exposure over time. A hierarchical Bayesian robust regression model of 92 separate effect sizes extracted from 24 studies indicates a moderate effect size in a positive direction (0.35, CI [0.21: 0.50]). This result suggests that infants possess a robust ability to detect audio-visual congruence for speech. Moderator analyses, moreover, suggest that infants' audio-visual matching ability for speech emerges at an early point in the process of language acquisition and remains stable for both native and non-native speech throughout early development. A sensitivity analysis of the meta-analytic data, however, indicates that a moderate publication bias for significant results could shift the lower credible interval to include null effects. Based on these findings, we outline recommendations for new lines of enquiry and suggest ways to improve the replicability of results in future investigations.

The other-race effect on the McGurk effect in infancy

This study investigated the difference in the McGurk effect between own-race-face and other-race-face stimuli among Japanese infants from 5 to 9 months of age. The McGurk effect results from infants using information from a speaker’s face in audiovisual speech integration. We hypothesized that the McGurk effect varies with the speaker’s race because of the other-race effect, which indicates an advantage for own-race faces in our face processing system. Experiment 1 demonstrated the other-race effect on audiovisual speech integration such that the infants ages 5–6 months and 8–9 months are likely to perceive the McGurk effect when observing an own-race-face speaker, but not when observing an other-race-face speaker. Experiment 2 found the other-race effect on audiovisual speech integration regardless of irrelevant speech identity cues. Experiment 3 confirmed the infants’ ability to differentiate two auditory syllables. These results showed that infants are likely to integrate voice with an own-race-face, but not with an other-race-face. This implies the role of experiences with own-race-faces in the development of audiovisual speech integration. Our findings also contribute to the discussion of whether perceptual narrowing is a modality-general, pan-sensory process.

Infants' Mapping of New Faces to New Voices

The ability to identify individuals by voice is fundamental for communication. However, little is known about the expectations that infants hold when learning unfamiliar voices. Here, the voice-learning skills of 4- and 8-month-olds (N = 53; 29 girls, 14 boys of various ethnicities) were tested using a preferential-looking task that involved audiovisual stimuli of their mothers and other unfamiliar women. Findings reveal that the expectation that novel voices map on to novel faces emerges between 4 and 8 months of age, and that infants can retain learning of face-voice pairings via nonostensive cues by 8 months of age. This study provides new insights about infants' use of disambiguation and fast mapping in early voice learning.

The Role of Unimodal Feedback Pathways in Gender Perception During Activation of Voice and Face Areas

Cross-modal effects provide a model framework for investigating hierarchical inter-areal processing, particularly, under conditions where unimodal cortical areas receive contextual feedback from other modalities. Here, using complementary behavioral and brain imaging techniques, we investigated the functional networks participating in face and voice processing during gender perception, a high-level feature of voice and face perception. Within the framework of a signal detection decision model, Maximum likelihood conjoint measurement (MLCM) was used to estimate the contributions of the face and voice to gender comparisons between pairs of audio-visual stimuli in which the face and voice were independently modulated. Top–down contributions were varied by instructing participants to make judgments based on the gender of either the face, the voice or both modalities (N = 12 for each task). Estimated face and voice contributions to the judgments of the stimulus pairs were not independent; both contributed to all tasks, but their respective weights varied over a 40-fold range due to top–down influences. Models that best described the modal contributions required the inclusion of two different top–down interactions: (i) an interaction that depended on gender congruence across modalities (i.e., difference between face and voice modalities for each stimulus); (ii) an interaction that depended on the within modalities’ gender magnitude. The significance of these interactions was task dependent. Specifically, gender congruence interaction was significant for the face and voice tasks while the gender magnitude interaction was significant for the face and stimulus tasks. Subsequently, we used the same stimuli and related tasks in a functional magnetic resonance imaging (fMRI) paradigm (N = 12) to explore the neural correlates of these perceptual processes, analyzed with Dynamic Causal Modeling (DCM) and Bayesian Model Selection. Results revealed changes in effective connectivity between the unimodal Fusiform Face Area (FFA) and Temporal Voice Area (TVA) in a fashion that paralleled the face and voice behavioral interactions observed in the psychophysical data. These findings explore the role in perception of multiple unimodal parallel feedback pathways.

Infants' identification of gender in biological motion displays

Infants' knowledge of social categories, including gender-typed characteristics, is a vital aspect of social cognitive development. In the current study, we examined 9- to 12-month-old infants' understanding of the categories "male" and "female" by testing for gender matching in voices or faces with biological motion depicted in point light displays (PLDs). Infants did not show voice-PLD gender matching spontaneously (Experiment 1) or after "training" with gender-matching voice-PLD pairs (Experiment 2). In Experiment 3, however, infants were trained with gender-matching face-PLD pairs and we found that patterns of visual attention to top regions of PLD stimuli during training predicted gender matching of female faces and PLDs. Prior to the end of the first postnatal year, therefore, infants may begin to identify gender in human walk motions, and perhaps form social categories from biological motion.

Neural indicators of articulator-specific sensorimotor influences on infant speech perception

While there is increasing acceptance that even young infants detect correspondences between heard and seen speech, the common view is that oral-motor movements related to speech production cannot influence speech perception until infants begin to babble or speak. We investigated the extent of multimodal speech influences on auditory speech perception in prebabbling infants who have limited speech-like oral-motor repertoires. We used event-related potentials (ERPs) to examine how sensorimotor influences to the infant's own articulatory movements impact auditory speech perception in 3-mo-old infants. In experiment 1, there were ERP discriminative responses to phonetic category changes across two phonetic contrasts (bilabial-dental /ba/-/ɗa/; dental-retroflex /ɗa/-/ɖa/) in a mismatch paradigm, indicating that infants auditorily discriminated both contrasts. In experiment 2, inhibiting infants' own tongue-tip movements had a disruptive influence on the early ERP discriminative response to the /ɗa/-/ɖa/ contrast only. The same articulatory inhibition had contrasting effects on the perception of the /ba/-/ɗa/ contrast, which requires different articulators (the lips vs. the tongue) during production, and the /ɗa/-/ɖa/ contrast, whereby both phones require tongue-tip movement as a place of articulation. This articulatory distinction between the two contrasts plausibly accounts for the distinct influence of tongue-tip suppression on the neural responses to phonetic category change perception in definitively prebabbling, 3-mo-old, infants. The results showing a specificity in the relation between oral-motor inhibition and phonetic speech discrimination suggest a surprisingly early mapping between auditory and motor speech representation already in prebabbling infants.

Gender Encoding in Gender Diverse and Gender Conforming Children

Previous research suggests that people encode gender starting in childhood. The present research asked whether gender diverse children (i.e., children whose gender identity or expression differs from that expected based on assigned sex) encode gender. Results showed that 3- to 5-year-old gender diverse participants (N = 71), siblings of gender diverse children (N = 52), and gender conforming controls (N = 69) did not significantly differ in degree of gender encoding. These results converge with prior research to suggest that gender diverse children process gender in ways that do not differ from gender conforming children, and provide further evidence that gender encoding may be a common aspect of person perception in societies that support a binary view of gender.

The Other-Race-Effect on Audiovisual Speech Integration in Infants: A NIRS Study

Previous studies have revealed perceptual narrowing for the own-race-face in face discrimination, but this phenomenon is poorly understood in face and voice integration. We focused on infants' brain responses to the McGurk effect to examine whether the other-race effect occurs in the activation patterns. In Experiment 1, we conducted fNIRS measurements to find the presence of a mapping of the McGurk effect in Japanese 8- to 9-month-old infants and to examine the difference between the activation patterns in response to own-race-face and other-race-face stimuli. We used two race-face conditions, own-race-face (East Asian) and other-race-face (Caucasian), each of which contained audiovisual-matched and McGurk-type stimuli. While the infants (N = 34) were observing each speech stimulus for each race, we measured cerebral hemoglobin concentrations in bilateral temporal brain regions. The results showed that in the own-race-face condition, audiovisual-matched stimuli induced the activation of the left temporal region, and the McGurk stimuli induced the activation of the bilateral temporal regions. No significant activations were found in the other-race-face condition. These results mean that the McGurk effect occurred only in the own-race-face condition. In Experiment 2, we used a familiarization/novelty preference procedure to confirm that the infants (N = 28) could perceive the McGurk effect in the own-race-face condition but not that of the other-race-face. The behavioral data supported the results of the fNIRS data, implying the presence of narrowing for the own-race face in the McGurk effect. These results suggest that narrowing of the McGurk effect may be involved in the development of relatively high-order processing, such as face-to-face communication with people surrounding the infant. We discuss the hypothesis that perceptual narrowing is a modality-general, pan-sensory process.

Gender/Sex, Sexual Orientation, and Identity Are in the Body: How Did They Get There?

In this review, I explore theoretical and empirical approaches to the development of gender/sex and sexual orientation (SO). Leaving behind the nature versus nurture opposition, I look at both identities as deeply embodied. My approach intertwines sex, gender, orientation, bodies, and cultures without a demand to choose one over the other. First, I introduce basic definitions, focusing on how intertwined the concepts of sex and gender really are. I affirm recent trends to consider a new term-gender/sex-as the best way to think about these deeply interwoven bodily traits. I introduce several literatures, each of which considers the processes by which traits become embodied. These points of view offer a basis for future work on identity development. Specifically, and selectively, I provide insights from the fields of phenomenology, dyadic interaction and the formation of presymbolic representations in infancy, and dynamic systems in infant development. I consider how thinking about embodied cognition helps to address intersubjectivity and the emergence of subjective identity. Next, I review what we currently know about the development of complex sexual systems in infancy and toddlerhood. Finally, I discuss the few existing theories of SO development that consider the events of infancy and childhood.

The role of production abilities in the perception of consonant category in infants

The influence of motor knowledge on speech perception is well established, but the functional role of the motor system is still poorly understood. The present study explores the hypothesis that speech production abilities may help infants discover phonetic categories in the speech stream, in spite of coarticulation effects. To this aim, we examined the influence of babbling abilities on consonant categorization in 6- and 9-month-old infants. Using an intersensory matching procedure, we investigated the infants' capacity to associate auditory information about a consonant in various vowel contexts with visual information about the same consonant, and to map auditory and visual information onto a common phoneme representation. Moreover, a parental questionnaire evaluated the infants' consonantal repertoire. In a first experiment using /b/-/d/ consonants, we found that infants who displayed babbling abilities and produced the /b/ and/or the /d/ consonants in repetitive sequences were able to correctly perform intersensory matching, while non-babblers were not. In a second experiment using the /v/-/z/ pair, which is as visually contrasted as the /b/-/d/ pair but which is usually not produced at the tested ages, no significant matching was observed, for any group of infants, babbling or not. These results demonstrate, for the first time, that the emergence of babbling could play a role in the extraction of vowel-independent representations for consonant place of articulation. They have important implications for speech perception theories, as they highlight the role of sensorimotor interactions in the development of phoneme representations during the first year of life.

Auditory⁻Visual Matching of Conspecifics and Non-Conspecifics by Dogs and Human Infants

Simple Summary

Comparative investigations on infants’ and dogs’ social and communicative skills revealed striking similarity, which can be attributed to convergent evolutionary and domestication processes. Using a suitable experimental method that allows systematic and direct comparisons of dogs and humans is essential. In the current study, we used non-invasive eye-tracking technology in order to investigate looking behaviour of dogs and human infants in an auditory–visual matching task. We found a similar gazing pattern in the two species when they were presented with pictures and vocalisations of a dog and a female human, that is, both dogs and infants looked longer at the dog portrait during the dog’s bark, while matching human speech with the human face was less obvious. Our results suggested different mechanisms underlying this analogous behaviour and highlighted the importance of future investigations into cross-modal cognition in dogs and humans.

Abstract

We tested whether dogs and 14–16-month-old infants are able to integrate intersensory information when presented with conspecific and heterospecific faces and vocalisations. The looking behaviour of dogs and infants was recorded with a non-invasive eye-tracking technique while they were concurrently presented with a dog and a female human portrait accompanied with acoustic stimuli of female human speech and a dog’s bark. Dogs showed evidence of both con- and heterospecific intermodal matching, while infants’ looking preferences indicated effective auditory–visual matching only when presented with the audio and visual stimuli of the non-conspecifics. The results of the present study provided further evidence that domestic dogs and human infants have similar socio-cognitive skills and highlighted the importance of comparative examinations on intermodal perception.

Language experience influences audiovisual speech integration in unimodal and bimodal bilingual infants

Infants as young as 2 months can integrate audio and visual aspects of speech articulation. A shift of attention from the eyes towards the mouth of talking faces occurs around 6 months of age in monolingual infants. However, it is unknown whether this pattern of attention during audiovisual speech processing is influenced by speech and language experience in infancy. The present study investigated this question by analysing audiovisual speech processing in three groups of 4- to 8-month-old infants who differed in their language experience: monolinguals, unimodal bilinguals (infants exposed to two or more spoken languages) and bimodal bilinguals (hearing infants with Deaf mothers). Eye-tracking was used to study patterns of face scanning while infants were viewing faces articulating syllables with congruent, incongruent and silent auditory tracks. Monolinguals and unimodal bilinguals increased their attention to the mouth of talking faces between 4 and 8 months, while bimodal bilinguals did not show any age difference in their scanning patterns. Moreover, older (6.6 to 8 months), but not younger, monolinguals (4 to 6.5 months) showed increased visual attention to the mouth of faces articulating audiovisually incongruent rather than congruent faces, indicating surprise or novelty. In contrast, no audiovisual congruency effect was found in unimodal or bimodal bilinguals. Results suggest that speech and language experience influences audiovisual integration in infancy. Specifically, reduced or more variable experience of audiovisual speech from the primary caregiver may lead to less sensitivity to the integration of audio and visual cues of speech articulation.

Preverbal infants utilize cross-modal semantic congruency in artificial grammar acquisition

Learning in a multisensory world is challenging as the information from different sensory dimensions may be inconsistent and confusing. By adulthood, learners optimally integrate bimodal (e.g. audio-visual, AV) stimulation by both low-level (e.g. temporal synchrony) and high-level (e.g. semantic congruency) properties of the stimuli to boost learning outcomes. However, it is unclear how this capacity emerges and develops. To approach this question, we examined whether preverbal infants were capable of utilizing high-level properties with grammar-like rule acquisition. In three experiments, we habituated pre-linguistic infants with an audio-visual (AV) temporal sequence that resembled a grammar-like rule (A-A-B). We varied the cross-modal semantic congruence of the AV stimuli (Exp 1: congruent syllables/faces; Exp 2: incongruent syllables/shapes; Exp 3: incongruent beeps/faces) while all the other low-level properties (e.g. temporal synchrony, sensory energy) were constant. Eight- to ten-month-old infants only learned the grammar-like rule from AV congruent stimuli pairs (Exp 1), not from incongruent AV pairs (Exp 2, 3). Our results show that similar to adults, preverbal infants' learning is influenced by a high-level multisensory integration gating system, pointing to a perceptual origin of bimodal learning advantage that was not previously acknowledged.

Prosody in the Auditory and Visual Domains: A Developmental Perspective

The development of body movements such as hand or head gestures, or facial expressions, seems to go hand-in-hand with the development of speech abilities. We know that very young infants rely on the movements of their caregivers' mouth to segment the speech stream, that infants' canonical babbling is temporally related to rhythmic hand movements, that narrative abilities emerge at a similar time in speech and gestures, and that children make use of both modalities to access complex pragmatic intentions. Prosody has emerged as a key linguistic component in this speech-gesture relationship, yet its exact role in the development of multimodal communication is still not well understood. For example, it is not clear what the relative weights of speech prosody and body gestures are in language acquisition, or whether both modalities develop at the same time or whether one modality needs to be in place for the other to emerge. The present paper reviews existing literature on the interactions between speech prosody and body movements from a developmental perspective in order to shed some light on these issues.

Overcoming the other-race effect in infancy with multisensory redundancy: 10-12-month-olds discriminate dynamic other-race faces producing speech

We tested 4-6- and 10-12-month-old infants to investigate whether the often-reported decline in infant sensitivity to other-race faces may reflect responsiveness to static or dynamic/silent faces rather than a general process of perceptual narrowing. Across three experiments, we tested discrimination of either dynamic own-race or other-race faces which were either accompanied by a speech syllable, no sound, or a non-speech sound. Results indicated that 4-6- and 10-12-month-old infants discriminated own-race as well as other-race faces accompanied by a speech syllable, that only the 10-12-month-olds discriminated silent own-race faces, and that 4-6-month-old infants discriminated own-race and other-race faces accompanied by a non-speech sound but that 10-12-month-old infants only discriminated own-race faces accompanied by a non-speech sound. Overall, the results suggest that the ORE reported to date reflects infant responsiveness to static or dynamic/silent faces rather than a general process of perceptual narrowing.

Voice over: Audio-visual congruency and content recall in the gallery setting

Experimental research has shown that pairs of stimuli which are congruent and assumed to ‘go together’ are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues.

Selective attention to a talker's mouth in infancy: role of audiovisual temporal synchrony and linguistic experience

Previous studies have found that infants shift their attention from the eyes to the mouth of a talker when they enter the canonical babbling phase after 6 months of age. Here, we investigated whether this increased attentional focus on the mouth is mediated by audio-visual synchrony and linguistic experience. To do so, we tracked eye gaze in 4-, 6-, 8-, 10-, and 12-month-old infants while they were exposed either to desynchronized native or desynchronized non-native audiovisual fluent speech. Results indicated that, regardless of language, desynchronization disrupted the usual pattern of relative attention to the eyes and mouth found in response to synchronized speech at 10 months but not at any other age. These findings show that audio-visual synchrony mediates selective attention to a talker's mouth just prior to the emergence of initial language expertise and that it declines in importance once infants become native-language experts.

The organization and reorganization of audiovisual speech perception in the first year of life

The period between six and 12 months is a sensitive period for language learning during which infants undergo auditory perceptual attunement, and recent results indicate that this sensitive period may exist across sensory modalities. We tested infants at three stages of perceptual attunement (six, nine, and 11 months) to determine 1) whether they were sensitive to the congruence between heard and seen speech stimuli in an unfamiliar language, and 2) whether familiarization with congruent audiovisual speech could boost subsequent non-native auditory discrimination. Infants at six- and nine-, but not 11-months, detected audiovisual congruence of non-native syllables. Familiarization to incongruent, but not congruent, audiovisual speech changed auditory discrimination at test for six-month-olds but not nine- or 11-month-olds. These results advance the proposal that speech perception is audiovisual from early in ontogeny, and that the sensitive period for audiovisual speech perception may last somewhat longer than that for auditory perception alone.

Audio-Visual Perception of Gender by Infants Emerges Earlier for Adult-Directed Speech

Early multisensory perceptual experiences shape the abilities of infants to perform socially-relevant visual categorization, such as the extraction of gender, age, and emotion from faces. Here, we investigated whether multisensory perception of gender is influenced by infant-directed (IDS) or adult-directed (ADS) speech. Six-, 9-, and 12-month-old infants saw side-by-side silent video-clips of talking faces (a male and a female) and heard either a soundtrack of a female or a male voice telling a story in IDS or ADS. Infants participated in only one condition, either IDS or ADS. Consistent with earlier work, infants displayed advantages in matching female relative to male faces and voices. Moreover, the new finding that emerged in the current study was that extraction of gender from face and voice was stronger at 6 months with ADS than with IDS, whereas at 9 and 12 months, matching did not differ for IDS versus ADS. The results indicate that the ability to perceive gender in audiovisual speech is influenced by speech manner. Our data suggest that infants may extract multisensory gender information developmentally earlier when looking at adults engaged in conversation with other adults (i.e., ADS) than when adults are directly talking to them (i.e., IDS). Overall, our findings imply that the circumstances of social interaction may shape early multisensory abilities to perceive gender.

Impact of language on functional connectivity for audiovisual speech integration

Visual information about lip and facial movements plays a role in audiovisual (AV) speech perception. Although this has been widely confirmed, previous behavioural studies have shown interlanguage differences, that is, native Japanese speakers do not integrate auditory and visual speech as closely as native English speakers. To elucidate the neural basis of such interlanguage differences, 22 native English speakers and 24 native Japanese speakers were examined in behavioural or functional Magnetic Resonance Imaging (fMRI) experiments while mono-syllabic speech was presented under AV, auditory-only, or visual-only conditions for speech identification. Behavioural results indicated that the English speakers identified visual speech more quickly than the Japanese speakers, and that the temporal facilitation effect of congruent visual speech was significant in the English speakers but not in the Japanese speakers. Using fMRI data, we examined the functional connectivity among brain regions important for auditory-visual interplay. The results indicated that the English speakers had significantly stronger connectivity between the visual motion area MT and the Heschl’s gyrus compared with the Japanese speakers, which may subserve lower-level visual influences on speech perception in English speakers in a multisensory environment. These results suggested that linguistic experience strongly affects neural connectivity involved in AV speech integration.

The Dynamics of Infant Attention: Implications for Crossmodal Perception and Word-Mapping Research

The present review is a novel synthesis of research on infants' attention in two related domains-crossmodal perception and word mapping. The authors hypothesize that infant attention is malleable and shifts in real time. They review dynamic models of infant attention and provide empirical evidence for parallel trends in attention shifts from the two domains that support their hypothesis. When infants are exposed to competing auditory-visual stimuli in experiments, multiple factors cause attention to shift during infant-environment interactions. Additionally, attention shifts across nested timescales and individual variations in attention systematically explain development. They suggest future research to further elucidate the causal mechanisms that influence infants' attention dynamics, emphasizing the need to examine individual variations that index shifts over time.

Perception of Multisensory Gender Coherence in 6- and 9-month-old Infants

One of the most salient social categories conveyed by human faces and voices is gender. We investigated the developmental emergence of the ability to perceive the coherence of auditory and visual attributes of gender in 6- and 9-month-old infants. Infants viewed two side-by-side video clips of a man and a woman singing a nursery rhyme and heard a synchronous male or female soundtrack. Results showed that 6-month-old infants did not match the audible and visible attributes of gender, and 9-month-old infants matched only female faces and voices. These findings indicate that the ability to perceive the multisensory coherence of gender emerges relatively late in infancy and that it reflects the greater experience that most infants have with female faces and voices.

Neonatal Restriction of Tactile Inputs Leads to Long-Lasting Impairments of Cross-Modal Processing

Optimal behavior relies on the combination of inputs from multiple senses through complex interactions within neocortical networks. The ontogeny of this multisensory interplay is still unknown. Here, we identify critical factors that control the development of visual-tactile processing by combining in vivo electrophysiology with anatomical/functional assessment of cortico-cortical communication and behavioral investigation of pigmented rats. We demonstrate that the transient reduction of unimodal (tactile) inputs during a short period of neonatal development prior to the first cross-modal experience affects feed-forward subcortico-cortical interactions by attenuating the cross-modal enhancement of evoked responses in the adult primary somatosensory cortex. Moreover, the neonatal manipulation alters cortico-cortical interactions by decreasing the cross-modal synchrony and directionality in line with the sparsification of direct projections between primary somatosensory and visual cortices. At the behavioral level, these functional and structural deficits resulted in lower cross-modal matching abilities. Thus, neonatal unimodal experience during defined developmental stages is necessary for setting up the neuronal networks of multisensory processing.

Reducing unisensory experience during neonatal development causes permanent disruption of connectivity between primary sensory cortices, resulting in impaired multisensory abilities.

Our senses, working together, enable us to interact with the environment. To obtain a unified percept of the world, diverse sensory inputs need to be bound together within distributed but strongly interconnected neuronal networks. Many multisensory abilities emerge or mature late in life, long after the maturation of the individual senses, yet the factors and mechanisms controlling their development are largely unknown. Here, we provide evidence for the critical role of unisensory experience during early postnatal life for the development of multisensory integration. Focusing on visual-tactile interactions in pigmented rats with good visual acuity, we show that a transient reduction of tactile inputs during neonatal development leads to sparser direct connections between adult primary visual and somatosensory cortices. As a result, these animals showed reduced neuronal activation following co-occurring tactile and visual stimuli, as well as impaired communication within visual-somatosensory networks. The structural and functional deficits resulting from an early manipulation of tactile experience had major behavioral consequences, impairing the rats’ ability to transfer information about encountered objects between senses. Thus, unisensory experience during early development shapes the neuronal networks of multisensory processing and the ability to transfer cross-modal information.

Infants' preference for native audiovisual speech dissociated from congruency preference

Although infant speech perception in often studied in isolated modalities, infants' experience with speech is largely multimodal (i.e., speech sounds they hear are accompanied by articulating faces). Across two experiments, we tested infants’ sensitivity to the relationship between the auditory and visual components of audiovisual speech in their native (English) and non-native (Spanish) language. In Experiment 1, infants’ looking times were measured during a preferential looking task in which they saw two simultaneous visual speech streams articulating a story, one in English and the other in Spanish, while they heard either the English or the Spanish version of the story. In Experiment 2, looking times from another group of infants were measured as they watched single displays of congruent and incongruent combinations of English and Spanish audio and visual speech streams. Findings demonstrated an age-related increase in looking towards the native relative to non-native visual speech stream when accompanied by the corresponding (native) auditory speech. This increase in native language preference did not appear to be driven by a difference in preference for native vs. non-native audiovisual congruence as we observed no difference in looking times at the audiovisual streams in Experiment 2.

Perception of the multisensory coherence of fluent audiovisual speech in infancy: its emergence and the role of experience

To investigate the developmental emergence of the perception of the multisensory coherence of native and non-native audiovisual fluent speech, we tested 4-, 8- to 10-, and 12- to 14-month-old English-learning infants. Infants first viewed two identical female faces articulating two different monologues in silence and then in the presence of an audible monologue that matched the visible articulations of one of the faces. Neither the 4-month-old nor 8- to 10-month-old infants exhibited audiovisual matching in that they did not look longer at the matching monologue. In contrast, the 12- to 14-month-old infants exhibited matching and, consistent with the emergence of perceptual expertise for the native language, perceived the multisensory coherence of native-language monologues earlier in the test trials than that of non-native language monologues. Moreover, the matching of native audible and visible speech streams observed in the 12- to 14-month-olds did not depend on audiovisual synchrony, whereas the matching of non-native audible and visible speech streams did depend on synchrony. Overall, the current findings indicate that the perception of the multisensory coherence of fluent audiovisual speech emerges late in infancy, that audiovisual synchrony cues are more important in the perception of the multisensory coherence of non-native speech than that of native audiovisual speech, and that the emergence of this skill most likely is affected by perceptual narrowing.

Motherese by eye and ear: infants perceive visual prosody in point-line displays of talking heads

Infant-directed (ID) speech provides exaggerated auditory and visual prosodic cues. Here we investigated if infants were sensitive to the match between the auditory and visual correlates of ID speech prosody. We presented 8-month-old infants with two silent line-joined point-light displays of faces speaking different ID sentences, and a single vocal-only sentence matched to one of the displays. Infants looked longer to the matched than mismatched visual signal when full-spectrum speech was presented; and when the vocal signals contained speech low-pass filtered at 400 Hz. When the visual display was separated into rigid (head only) and non-rigid (face only) motion, the infants looked longer to the visual match in the rigid condition; and to the visual mismatch in the non-rigid condition. Overall, the results suggest 8-month-olds can extract information about the prosodic structure of speech from voice and head kinematics, and are sensitive to their match; and that they are less sensitive to the match between lip and voice information in connected speech.

The development of sensorimotor influences in the audiovisual speech domain: some critical questions

Speech researchers have long been interested in how auditory and visual speech signals are integrated, and the recent work has revived interest in the role of speech production with respect to this process. Here, we discuss these issues from a developmental perspective. Because speech perception abilities typically outstrip speech production abilities in infancy and childhood, it is unclear how speech-like movements could influence audiovisual speech perception in development. While work on this question is still in its preliminary stages, there is nevertheless increasing evidence that sensorimotor processes (defined here as any motor or proprioceptive process related to orofacial movements) affect developmental audiovisual speech processing. We suggest three areas on which to focus in future research: (i) the relation between audiovisual speech perception and sensorimotor processes at birth, (ii) the pathways through which sensorimotor processes interact with audiovisual speech processing in infancy, and (iii) developmental change in sensorimotor pathways as speech production emerges in childhood.

Early experience and multisensory perceptual narrowing

Perceptual narrowing reflects the effects of early experience and contributes in key ways to perceptual and cognitive development. Previous studies have found that unisensory perceptual sensitivity in young infants is broadly tuned such that they can discriminate native as well as non-native sensory inputs but that it is more narrowly tuned in older infants such that they only respond to native inputs. Recently, my coworkers and I discovered that multisensory perceptual sensitivity narrows as well. The present article reviews this new evidence in the general context of multisensory perceptual development and the effects of early experience. Together, the evidence on unisensory and multisensory narrowing shows that early experience shapes the emergence of perceptual specialization and expertise.

Infant-Directed Visual Prosody: Mothers' Head Movements and Speech Acoustics

Acoustical changes in the prosody of mothers' speech to infants are distinct and near universal. However, less is known about the visible properties mothers' infant-directed (ID) speech, and their relation to speech acoustics. Mothers' head movements were tracked as they interacted with their infants using ID speech, and compared to movements accompanying their adult-directed (AD) speech. Movement measures along three dimensions of head translation, and three axes of head rotation were calculated. Overall, more head movement was found for ID than AD speech, suggesting that mothers exaggerate their visual prosody in a manner analogous to the acoustical exaggerations in their speech. Regression analyses examined the relation between changing head position and changing acoustical pitch (F₀ ) over time. Head movements and voice pitch were more strongly related in ID speech than in AD speech. When these relations were examined across time windows of different durations, stronger relations were observed for shorter time windows (< 5 sec). However, the particular form of these more local relations did not extend or generalize to longer time windows. This suggests that the multimodal correspondences in speech prosody are variable in form, and occur within limited time spans.

Age-related sensitive periods influence visual language discrimination in adults

Adults as well as infants have the capacity to discriminate languages based on visual speech alone. Here, we investigated whether adults' ability to discriminate languages based on visual speech cues is influenced by the age of language acquisition. Adult participants who had all learned English (as a first or second language) but did not speak French were shown faces of bilingual (French/English) speakers silently reciting sentences in either language. Using only visual speech information, adults who had learned English from birth or as a second language before the age of 6 could discriminate between French and English significantly better than chance. However, adults who had learned English as a second language after age 6 failed to discriminate these two languages, suggesting that early childhood exposure is crucial for using relevant visual speech information to separate languages visually. These findings raise the possibility that lowered sensitivity to non-native visual speech cues may contribute to the difficulties encountered when learning a new language in adulthood.

Cross-modal signatures in maternal speech and singing

We explored the possibility of a unique cross-modal signature in maternal speech and singing that enables adults and infants to link unfamiliar speaking or singing voices with subsequently viewed silent videos of the talkers or singers. In Experiment 1, adults listened to 30-s excerpts of speech followed by successively presented 7-s silent video clips, one from the previously heard speaker (different speech content) and the other from a different speaker. They successfully identified the previously heard speaker. In Experiment 2, adults heard comparable excerpts of singing followed by silent video clips from the previously heard singer (different song) and another singer. They failed to identify the previously heard singer. In Experiment 3, the videos of talkers and singers were blurred to obscure mouth movements. Adults successfully identified the talkers and they also identified the singers from videos of different portions of the song previously heard. In Experiment 4, 6- to 8-month-old infants listened to 30-s excerpts of the same maternal speech or singing followed by exposure to the silent videos on alternating trials. They looked longer at the silent videos of previously heard talkers and singers. The findings confirm the individuality of maternal speech and singing performance as well as adults' and infants' ability to discern the unique cross-modal signatures. The cues that enable cross-modal matching of talker and singer identity remain to be determined.

Recognition of Amodal Language Identity Emerges in Infancy

Audiovisual speech consists of overlapping and invariant patterns of dynamic acoustic and optic articulatory information. Research has shown that infants can perceive a variety of basic audio-visual (A-V) relations but no studies have investigated whether and when infants begin to perceive higher order A-V relations inherent in speech. Here, we asked whether and when infants become capable of recognizing amodal language identity, a critical perceptual skill that is necessary for the development of multisensory communication. Because, at a minimum, such a skill requires the ability to perceive suprasegmental auditory and visual linguistic information, we predicted that this skill would not emerge before higher-level speech processing and multisensory integration skills emerge. Consistent with this prediction, we found that recognition of the amodal identity of language emerges at 10-12 months of age but that when it emerges it is restricted to infants' native language.

Six-month-old infants match other-race faces with a non-native language

Early in life, infants possess an effective face-processing system which becomes specialized according to the faces present in the environment. Infants are also exposed to the voices and sounds of caregivers. Previous studies have found that face–voice associations become progressively more tuned to the types of association most prevalent in the environment. The present study investigated whether 6-month-old infants associate own-race faces with their native language and faces from a different race with a non-native language. Infants were presented with pictures of own- and other-race faces simultaneously, with a native or non-native language in a habituation paradigm. Results indicate that 6-month-olds are able to match other-race faces to a non-native language.

Neural correlates of perceptual narrowing in cross-species face-voice matching

Integrating the multisensory features of talking faces is critical to learning and extracting coherent meaning from social signals. While we know much about the development of these capacities at the behavioral level, we know very little about the underlying neural processes. One prominent behavioral milestone of these capacities is the perceptual narrowing of face-voice matching, whereby young infants match faces and voices across species, but older infants do not. In the present study, we provide neurophysiological evidence for developmental decline in cross-species face-voice matching. We measured event-related brain potentials (ERPs) while 4- and 8-month-old infants watched and listened to congruent and incongruent audio-visual presentations of monkey vocalizations and humans mimicking monkey vocalizations. The ERP results indicated that younger infants distinguished between the congruent and the incongruent faces and voices regardless of species, whereas in older infants, the sensitivity to multisensory congruency was limited to the human face and voice. Furthermore, with development, visual and frontal brain processes and their functional connectivity became more sensitive to the congruence of human faces and voices relative to monkey faces and voices. Our data show the neural correlates of perceptual narrowing in face-voice matching and support the notion that postnatal experience with species identity is associated with neural changes in multisensory processing (Lewkowicz & Ghazanfar, 2009).

The joint development of hemispheric lateralization for words and faces

Consistent with long-standing findings from behavioral studies, neuroimaging investigations have identified a region of the inferior temporal cortex that, in adults, shows greater face selectivity in the right than left hemisphere and, conversely, a region that shows greater word selectivity in the left than right hemisphere. What has not been determined is how this pattern of mature hemispheric specialization emerges over the course of development. The present study examines the hemispheric superiority for faces and words in children, young adolescents and adults in a discrimination task in which stimuli are presented briefly in either hemifield. Whereas adults showed the expected left and right visual field superiority for face and word discrimination, respectively, the young adolescents demonstrated only the right-field superiority for words and no field superiority for faces. Although the children's overall accuracy was lower than that of the older groups, like the young adolescents, they exhibited a right visual field superiority for words but no field superiority for faces. Interestingly, the emergence of face lateralization was correlated with reading competence, measured on an independent standardized test, after regressing out age, quantitative reasoning scores, and face discrimination accuracy. Taken together, these findings suggest that the hemispheric organization of face and word recognition do not develop independently and that word lateralization, which emerges earlier, may drive later face lateralization. A theoretical account in which competition for visual representations unfolds over the course of development is proposed to account for the findings.

The development of the uncanny valley in infants

When adults view very realistic humanoid robots or computer avatars they often exhibit an aversion to them. This phenomenon, known as the "uncanny valley," is assumed to be evolutionary in origin, perhaps tapping into modules for disgust or attractiveness that detect violations of our normal expectations regarding social signals. Here, we test an alternative hypothesis that the uncanny valley is developmental in origin and, thus, that specific early experience with real human faces leads to its eventual emergence. To test this idea, we measured visual preferences in response to all possible pairs of a human face, realistic avatar face, and an unrealistic avatar face in groups of 6-, 8-, 10-, and 12-month-old infants. Consistent with the developmental hypothesis, we found that the uncanny valley effect emerges at 12 months of age suggesting that perceptual experience with real human faces is critical to its emergence.

Monkeys and humans share a common computation for face/voice integration

Speech production involves the movement of the mouth and other regions of the face resulting in visual motion cues. These visual cues enhance intelligibility and detection of auditory speech. As such, face-to-face speech is fundamentally a multisensory phenomenon. If speech is fundamentally multisensory, it should be reflected in the evolution of vocal communication: similar behavioral effects should be observed in other primates. Old World monkeys share with humans vocal production biomechanics and communicate face-to-face with vocalizations. It is unknown, however, if they, too, combine faces and voices to enhance their perception of vocalizations. We show that they do: monkeys combine faces and voices in noisy environments to enhance their detection of vocalizations. Their behavior parallels that of humans performing an identical task. We explored what common computational mechanism(s) could explain the pattern of results we observed across species. Standard explanations or models such as the principle of inverse effectiveness and a "race" model failed to account for their behavior patterns. Conversely, a "superposition model", positing the linear summation of activity patterns in response to visual and auditory components of vocalizations, served as a straightforward but powerful explanatory mechanism for the observed behaviors in both species. As such, it represents a putative homologous mechanism for integrating faces and voices across primates.