Effects of formant proximity and stimulus prototypicality on the neural discrimination of vowels: Evidence from the auditory frequency-following response

Child-Directed Speech in Noise: Testing Signal- and Code-Based Phonetic Enhancement

PURPOSE

Talkers adapt their speech according to the demands of their listeners and the communicative context, enhancing the properties of the signal (pitch, intensity) and/or properties of the code (enhancement of phonemic contrasts). This study asked how mothers adapt their child-directed speech (CDS) in ways that might serve the immediate goals of increasing intelligibility, as well as long-term goals of supporting speech and language development in their children.

METHOD

Mothers (N = 28) participated in a real-time interactive speech production/perception paradigm, in which mothers instructed their young (3- to 5-year-old) children, or an adult listener, to select the picture corresponding to a target word. The task was performed at low and high levels (56 vs. 75 dB SPL) of background noise to examine the Lombard effects of decreased audibility on speech production.

RESULTS

Acoustic-phonetic analyses of CDS and adult-directed speech (ADS) productions of target words and carrier phrase (e.g., "Find pig") revealed that mothers significantly enhanced the mean pitch, pitch variability, and intensity of target words in CDS, particularly at higher background noise levels and for younger children. Mothers produce CDS with a higher signal-to-noise ratio than ADS. However, limited evidence was found for phonetic enhancement of the segmental properties of speech. Although increased category separation was found in the voice onset time of stop consonants, decreased vowel category separation (an anti-enhancement effect) was observed in CDS.

CONCLUSIONS

Mothers readily enhance the suprasegmental signal properties of their speech in CDS, but not the acoustic-phonetic properties of phonemes. This study fails to provide evidence of phonetic enhancement in preschool children in a dyadic communication task under noisy listening conditions.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.24645423.

Reaction time for correct identification of vowels in consonant-vowel syllables and of vowel segments

Reaction times for correct vowel identification were measured to determine the effects of intertrial intervals, vowel, and cue type. Thirteen adults with normal hearing, aged 20-38 years old, participated. Stimuli included three naturally produced syllables (/ba/ /bi/ /bu/) presented whole or segmented to isolate the formant transition or static formant center. Participants identified the vowel presented via loudspeaker by mouse click. Results showed a significant effect of intertrial intervals, no significant effect of cue type, and a significant vowel effect-suggesting that feedback occurs, vowel identification may depend on cue duration, and vowel bias may stem from focal structure.

Encoding of speech in convolutional layers and the brain stem based on language experience

Comparing artificial neural networks with outputs of neuroimaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. The proposed technique is based on a similar principle that underlies electroencephalography (EEG): averaging of neural (artificial or biological) activity across neurons in the time domain, and allows to compare encoding of any acoustic property in the brain and in intermediate convolutional layers of an artificial neural network. Our approach allows a direct comparison of responses to a phonetic property in the brain and in deep neural networks that requires no linear transformations between the signals. We argue that the brain stem response (cABR) and the response in intermediate convolutional layers to the exact same stimulus are highly similar without applying any transformations, and we quantify this observation. The proposed technique not only reveals similarities, but also allows for analysis of the encoding of actual acoustic properties in the two signals: we compare peak latency (i) in cABR relative to the stimulus in the brain stem and in (ii) intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examine and compare the effect of prior language exposure on the peak latency in cABR and in intermediate convolutional layers. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property and for other neuroimaging techniques.

Perceptual warping exposes categorical representations for speech in human brainstem responses

The brain transforms continuous acoustic events into discrete category representations to downsample the speech signal for our perceptual-cognitive systems. Such phonetic categories are highly malleable, and their percepts can change depending on surrounding stimulus context. Previous work suggests these acoustic-phonetic mapping and perceptual warping of speech emerge in the brain no earlier than auditory cortex. Here, we examined whether these auditory-category phenomena inherent to speech perception occur even earlier in the human brain, at the level of auditory brainstem. We recorded speech-evoked frequency following responses (FFRs) during a task designed to induce more/less warping of listeners' perceptual categories depending on stimulus presentation order of a speech continuum (random, forward, backward directions). We used a novel clustered stimulus paradigm to rapidly record the high trial counts needed for FFRs concurrent with active behavioral tasks. We found serial stimulus order caused perceptual shifts (hysteresis) near listeners' category boundary confirming identical speech tokens are perceived differentially depending on stimulus context. Critically, we further show neural FFRs during active (but not passive) listening are enhanced for prototypical vs. category-ambiguous tokens and are biased in the direction of listeners' phonetic label even for acoustically-identical speech stimuli. These findings were not observed in the stimulus acoustics nor model FFR responses generated via a computational model of cochlear and auditory nerve transduction, confirming a central origin to the effects. Our data reveal FFRs carry category-level information and suggest top-down processing actively shapes the neural encoding and categorization of speech at subcortical levels. These findings suggest the acoustic-phonetic mapping and perceptual warping in speech perception occur surprisingly early along the auditory neuroaxis, which might aid understanding by reducing ambiguity inherent to the speech signal.

Neuronal and behavioral affective perceptions of human and naturalness-reduced emotional prosodies

Artificial voices are nowadays embedded into our daily lives with latest neural voices approaching human voice consistency (naturalness). Nevertheless, behavioral, and neuronal correlates of the perception of less naturalistic emotional prosodies are still misunderstood. In this study, we explored the acoustic tendencies that define naturalness from human to synthesized voices. Then, we created naturalness-reduced emotional utterances by acoustic editions of human voices. Finally, we used Event-Related Potentials (ERP) to assess the time dynamics of emotional integration when listening to both human and synthesized voices in a healthy adult sample. Additionally, listeners rated their perceptions for valence, arousal, discrete emotions, naturalness, and intelligibility. Synthesized voices were characterized by less lexical stress (i.e., reduced difference between stressed and unstressed syllables within words) as regards duration and median pitch modulations. Besides, spectral content was attenuated toward lower F2 and F3 frequencies and lower intensities for harmonics 1 and 4. Both psychometric and neuronal correlates were sensitive to naturalness reduction. (1) Naturalness and intelligibility ratings dropped with emotional utterances synthetization, (2) Discrete emotion recognition was impaired as naturalness declined, consistent with P200 and Late Positive Potentials (LPP) being less sensitive to emotional differentiation at lower naturalness, and (3) Relative P200 and LPP amplitudes between prosodies were modulated by synthetization. Nevertheless, (4) Valence and arousal perceptions were preserved at lower naturalness, (5) Valence (arousal) ratings correlated negatively (positively) with Higuchi’s fractal dimension extracted on neuronal data under all naturalness perturbations, (6) Inter-Trial Phase Coherence (ITPC) and standard deviation measurements revealed high inter-individual heterogeneity for emotion perception that is still preserved as naturalness reduces. Notably, partial between-participant synchrony (low ITPC), along with high amplitude dispersion on ERPs at both early and late stages emphasized miscellaneous emotional responses among subjects. In this study, we highlighted for the first time both behavioral and neuronal basis of emotional perception under acoustic naturalness alterations. Partial dependencies between ecological relevance and emotion understanding outlined the modulation but not the annihilation of emotional integration by synthetization.

Development of infants' neural speech processing and its relation to later language skills: A MEG study

The 'sensitive period' for phonetic learning (∼6-12 months) is one of the earliest milestones in language acquisition where infants start to become specialized in processing speech sounds in their native language. In the last decade, advancements in neuroimaging technologies for infants are starting to shed light on the underlying neural mechanisms supporting this important learning period. The current study reports on a large longitudinal dataset with the aim to replicate and extend on two important questions: 1) what are the developmental changes during the 'sensitive period' for native and nonnative speech processing? 2) how does native and nonnative speech processing in infants predict later language outcomes? Fifty-four infants were recruited at 7 months of age and their neural processing of speech was measured using Magnetoencephalography (MEG). Specifically, the neural sensitivity to a native and a nonnative speech contrast was indexed by the mismatch response (MMR). They repeated the measurement again at 11 months of age and their language development was further tracked from 12 months to 30 months of age using the MacArthur-Bates Communicative Development Inventory (CDI). Using an a priori Region-of-Interest (ROI) approach, we observed significant increases for the Native MMR in the left inferior frontal region (IF) and superior temporal region (ST) from 7 to 11 months, but not for the Nonnative MMR. Complementary whole brain comparison revealed more widespread developmental changes for both contrasts. However, only individual differences in the left IF and ST for the Nonnative MMR at 11 months of age were significant predictors of individual vocabulary growth up to 30 months of age. An exploratory machine-learning based analysis further revealed that whole brain time series for both Native and Nonnative contrasts can robustly predict later outcomes, but with very different underlying spatial-temporal patterns. The current study extends our current knowledge and suggests that native and nonnative speech processing may follow different developmental trajectories and utilize different mechanisms that are relevant for later language skills.

The tone atlas of perceptual discriminability and perceptual distance: Four tone languages and five language groups

Some prior investigations suggest that tone perception is flexible, reasonably independent of native phonology, whereas others suggest it is constrained by native phonology. We address this issue in a systematic and comprehensive investigation of adult tone perception. Sampling from diverse tone and non-tone speaking communities, we tested discrimination of the three major tone systems (Cantonese, Thai, Mandarin) that dominate the tone perception literature, in relation to native language and language experience as well as stimulus variation (tone properties, presentation order, pitch cues) using linear mixed effect modelling and multidimensional scaling. There was an overall discrimination advantage for tone language speakers and for native tones. However, language- and tone-specific effects, and presentation order effects also emerged. Thus, over and above native phonology, stimulus variation exerts a powerful influence on tone discrimination. This study provides a tone atlas, a reference guide to inform empirical studies of tone sensitivity, both retrospectively and prospectively.

Lexical and Prosodic Pitch Modifications in Cantonese Infant-directed Speech

The functions of acoustic-phonetic modifications in infant-directed speech (IDS) remain a question: do they specifically serve to facilitate language learning via enhanced phonemic contrasts (the hyperarticulation hypothesis) or primarily to improve communication via prosodic exaggeration (the prosodic hypothesis)? The study of lexical tones provides a unique opportunity to shed light on this, as lexical tones are phonemically contrastive, yet their primary cue, pitch, is also a prosodic cue. This study investigated Cantonese IDS and found increased intra-talker variation of lexical tones, which more likely posed a challenge to rather than facilitated phonetic learning. Although tonal space was expanded which could facilitate phonetic learning, its expansion was a function of overall intonational modifications. Similar findings were observed in speech to pets who should not benefit from larger phonemic distinction. We conclude that lexical-tone adjustments in IDS mainly serve to broadly enhance communication rather than specifically increase phonemic contrast for learners.

Neurophysiological Correlates of Asymmetries in Vowel Perception: An English-French Cross-Linguistic Event-Related Potential Study

Behavioral studies examining vowel perception in infancy indicate that, for many vowel contrasts, the ease of discrimination changes depending on the order of stimulus presentation, regardless of the language from which the contrast is drawn and the ambient language that infants have experienced. By adulthood, linguistic experience has altered vowel perception; analogous asymmetries are observed for non-native contrasts but are mitigated for native contrasts. Although these directional effects are well documented behaviorally, the brain mechanisms underlying them are poorly understood. In the present study we begin to address this gap. We first review recent behavioral work which shows that vowel perception asymmetries derive from phonetic encoding strategies, rather than general auditory processes. Two existing theoretical models-the Natural Referent Vowel framework and the Native Language Magnet model-are invoked as a means of interpreting these findings. Then we present the results of a neurophysiological study which builds on this prior work. Using event-related brain potentials, we first measured and assessed the mismatch negativity response (MMN, a passive neurophysiological index of auditory change detection) in English and French native-speaking adults to synthetic vowels that either spanned two different phonetic categories (/y/vs./u/) or fell within the same category (/u/). Stimulus presentation was organized such that each vowel was presented as standard and as deviant in different blocks. The vowels were presented with a long (1,600-ms) inter-stimulus interval to restrict access to short-term memory traces and tap into a "phonetic mode" of processing. MMN analyses revealed weak asymmetry effects regardless of the (i) vowel contrast, (ii) language group, and (iii) MMN time window. Then, we conducted time-frequency analyses of the standard epochs for each vowel. In contrast to the MMN analysis, time-frequency analysis revealed significant differences in brain oscillations in the theta band (4-8 Hz), which have been linked to attention and processing efficiency. Collectively, these findings suggest that early-latency (pre-attentive) mismatch responses may not be a strong neurophysiological correlate of asymmetric behavioral vowel discrimination. Rather, asymmetries may reflect differences in neural processing efficiency for vowels with certain inherent acoustic-phonetic properties, as revealed by theta oscillatory activity.

Asymmetries in Accessing Vowel Representations Are Driven by Phonological and Acoustic Properties: Neural and Behavioral Evidence From Natural German Minimal Pairs

In vowel discrimination, commonly found discrimination patterns are directional asymmetries where discrimination is faster (or easier) if differing vowels are presented in a certain sequence compared to the reversed sequence. Different models of speech sound processing try to account for these asymmetries based on either phonetic or phonological properties. In this study, we tested and compared two of those often-discussed models, namely the Featurally Underspecified Lexicon (FUL) model (Lahiri and Reetz, 2002) and the Natural Referent Vowel (NRV) framework (Polka and Bohn, 2011). While most studies presented isolated vowels, we investigated a large stimulus set of German vowels in a more naturalistic setting within minimal pairs. We conducted an mismatch negativity (MMN) study in a passive and a reaction time study in an active oddball paradigm. In both data sets, we found directional asymmetries that can be explained by either phonological or phonetic theories. While behaviorally, the vowel discrimination was based on phonological properties, both tested models failed to explain the found neural patterns comprehensively. Therefore, we additionally examined the influence of a variety of articulatory, acoustical, and lexical factors (e.g., formant structure, intensity, duration, and frequency of occurrence) but also the influence of factors beyond the well-known (perceived loudness of vowels, degree of openness) in depth via multiple regression analyses. The analyses revealed that the perceptual factor of perceived loudness has a greater impact than considered in the literature and should be taken stronger into consideration when analyzing preattentive natural vowel processing.

Analyzing the FFR: A tutorial for decoding the richness of auditory function

The frequency-following response, or FFR, is a neurophysiological response to sound that precisely reflects the ongoing dynamics of sound. It can be used to study the integrity and malleability of neural encoding of sound across the lifespan. Sound processing in the brain can be impaired with pathology and enhanced through expertise. The FFR can index linguistic deprivation, autism, concussion, and reading impairment, and can reflect the impact of enrichment with short-term training, bilingualism, and musicianship. Because of this vast potential, interest in the FFR has grown considerably in the decade since our first tutorial. Despite its widespread adoption, there remains a gap in the current knowledge of its analytical potential. This tutorial aims to bridge this gap. Using recording methods we have employed for the last 20 + years, we have explored many analysis strategies. In this tutorial, we review what we have learned and what we think constitutes the most effective ways of capturing what the FFR can tell us. The tutorial covers FFR components (timing, fundamental frequency, harmonics) and factors that influence FFR (stimulus polarity, response averaging, and stimulus presentation/recording jitter). The spotlight is on FFR analyses, including ways to analyze FFR timing (peaks, autocorrelation, phase consistency, cross-phaseogram), magnitude (RMS, SNR, FFT), and fidelity (stimulus-response correlations, response-to-response correlations and response consistency). The wealth of information contained within an FFR recording brings us closer to understanding how the brain reconstructs our sonic world.