A multimodal spectral approach to characterize rhythm in natural speech

Modulation transfer functions for audiovisual speech

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

Characterizing first and second language rhythm in English using spectral coherence between temporal envelope and mouth opening-closing movements

This study investigated the rhythmic differences between first and second language English from 19 native speakers of American English and an equal number of native speakers of Mandarin. Speech rhythm was viewed from MacNeilage's frame/content theory. The spectral coherence between the temporal envelope and the mouth opening and closing kinematics was computed to operationalize the rhythmic frame. The spectral centroid, spread, rolloff, flatness, and entropy were calculated to reveal the frequency distribution patterns in the coherence. Using a binary logistic regression model, these measures were collectively found to be effective in characterizing rhythmic differences between native and non-native groups (A' = 0.71 and B″_D = -0.06). Specifically, the native group was significantly higher than the non-native group in terms of spectral centroid and spread, whereas the native group was significantly lower than its non-native counterpart in terms of spectral flatness and entropy. Both groups were not significantly different in spectral rolloff. Possible explanations for the result as well as the efficacy of employing the aforesaid coherence in speech rhythm research in general were discussed.

Neural oscillations track natural but not artificial fast speech: Novel insights from speech-brain coupling using MEG

Neural oscillations contribute to speech parsing via cortical tracking of hierarchical linguistic structures, including syllable rate. While the properties of neural entrainment have been largely probed with speech stimuli at either normal or artificially accelerated rates, the important case of natural fast speech has been largely overlooked. Using magnetoencephalography, we found that listening to naturally-produced speech was associated with cortico-acoustic coupling, both at normal (∼6 syllables/s) and fast (∼9 syllables/s) rates, with a corresponding shift in peak entrainment frequency. Interestingly, time-compressed sentences did not yield such coupling, despite being generated at the same rate as the natural fast sentences. Additionally, neural activity in right motor cortex exhibited stronger tuning to natural fast rather than to artificially accelerated speech, and showed evidence for stronger phase-coupling with left temporo-parietal and motor areas. These findings are highly relevant for our understanding of the role played by auditory and motor cortex oscillations in the perception of naturally produced speech.

Correcting MEG Artifacts Caused by Overt Speech

Recording brain activity during speech production using magnetoencephalography (MEG) can help us to understand the dynamics of speech production. However, these measurements are challenging due to the induced artifacts coming from several sources such as facial muscle activity, lower jaw and head movements. Here, we aimed to characterize speech-related artifacts, focusing on head movements, and subsequently present an approach to remove these artifacts from MEG data. We recorded MEG from 11 healthy participants while they pronounced various syllables in different loudness. Head positions/orientations were extracted during speech production to investigate its role in MEG distortions. Finally, we present an artifact rejection approach using the combination of regression analysis and signal space projection (SSP) in order to correct the induced artifact from MEG data. Our results show that louder speech leads to stronger head movements and stronger MEG distortions. Our proposed artifact rejection approach could successfully remove the speech-related artifact and retrieve the underlying neurophysiological signals. As the presented artifact rejection approach was shown to remove artifacts arising from head movements, induced by overt speech in the MEG, it will facilitate research addressing the neural basis of speech production with MEG.

Mapping the neuroanatomical impact of very preterm birth across childhood

Those born very preterm (VPT; <32 weeks gestational age) have an increased risk in developing a wide range of cognitive deficits. In early-to-late childhood, brain structure has been shown to be altered in VPT compared to full-term (FT) children; however, the results are inconsistent. The current study examined subcortical volumes, cortical thickness, and surface area in a large cohort of VPT and FT children aged 4-12 years. Structural magnetic resonance imaging (MRI) was obtained on 120 VPT and 146 FT children who returned up to three times, resulting in 176 VPT and 173 FT unique data points. For each participant, Corticometric Iterative Vertex-based Estimation of Thickness was used to obtain global measurements of total brain, cortical grey and cortical white matter volumes, along with surface-based measurements of cortical thickness and surface area, and Multiple Automatically Generated Templates (MAGeT) brain segmentation tool was used to segment the subcortical structures. To examine group differences and group-age interactions, mixed-effects models were used (controlling for whole-brain volume). We found few differences between the two groups in subcortical volumes. The VPT children showed increased cortical thickness in frontal, occipital and fusiform gyri and inferior pre-post-central areas, while thinning occurred in the midcingulate. Cortical thickness in occipital regions showed more rapid decreases with age in the VPT compared to the FT children. VPT children also showed both regional increases, particularly in the temporal lobe, and decreases in surface area. Our results indicate a delayed maturational trajectory in those born VPT.

Experimental and theoretical study of velocity fluctuations during slow movements in humans

Moving smoothly is generally considered as a higher-order goal of motor control and moving jerkily as a witness of clumsiness or pathology, yet many common and well-controlled movements (e.g., tracking movements) have irregular velocity profiles with widespread fluctuations. The origin and nature of these fluctuations have been associated with the operation of an intermittent process but in fact remain poorly understood. Here we studied velocity fluctuations during slow movements, using combined experimental and theoretical tools. We recorded arm movement trajectories in a group of healthy participants performing back-and-forth movements at different speeds, and we analyzed velocity profiles in terms of series of segments (portions of velocity between 2 minima). We found that most of the segments were smooth (i.e., corresponding to a biphasic acceleration) and had constant duration irrespective of movement speed and linearly increasing amplitude with movement speed. We accounted for these observations with an optimal feedback control model driven by a staircase goal position signal in the presence of sensory noise. Our study suggests that one and the same control process can explain the production of fast and slow movements, i.e., fast movements emerge from the immediate tracking of a global goal position and slow movements from the successive tracking of intermittently updated intermediate goal positions. NEW & NOTEWORTHY We show in experiments and modeling that slow movements could result from the brain tracking a sequence of via points regularly distributed in time and space. Accordingly, slow movements would differ from fast movement by the nature of the guidance and not by the nature of control. This result could help in understanding the origin and nature of slow and segmented movements frequently observed in brain disorders.

Predictive entrainment of natural speech through two fronto-motor top-down channels

Natural communication between interlocutors is enabled by the ability to predict upcoming speech in a given context. Previously we showed that these predictions rely on a fronto-motor top-down control of low-frequency oscillations in auditory-temporal brain areas that track intelligible speech. However, a comprehensive spatio-temporal characterisation of this effect is still missing. Here, we applied transfer entropy to source-localised MEG data during continuous speech perception. First, at low frequencies (1-4 Hz, brain delta phase to speech delta phase), predictive effects start in left fronto-motor regions and progress to right temporal regions. Second, at higher frequencies (14-18 Hz, brain beta power to speech delta phase), predictive patterns show a transition from left inferior frontal gyrus via left precentral gyrus to left primary auditory areas. Our results suggest a progression of prediction processes from higher-order to early sensory areas in at least two different frequency channels.

Cortical Tracking of Global and Local Variations of Speech Rhythm during Connected Natural Speech Perception

During natural speech perception, listeners must track the global speaking rate, that is, the overall rate of incoming linguistic information, as well as transient, local speaking rate variations occurring within the global speaking rate. Here, we address the hypothesis that this tracking mechanism is achieved through coupling of cortical signals to the amplitude envelope of the perceived acoustic speech signals. Cortical signals were recorded with magnetoencephalography (MEG) while participants perceived spontaneously produced speech stimuli at three global speaking rates (slow, normal/habitual, and fast). Inherently to spontaneously produced speech, these stimuli also featured local variations in speaking rate. The coupling between cortical and acoustic speech signals was evaluated using audio-MEG coherence. Modulations in audio-MEG coherence spatially differentiated between tracking of global speaking rate, highlighting the temporal cortex bilaterally and the right parietal cortex, and sensitivity to local speaking rate variations, emphasizing the left parietal cortex. Cortical tuning to the temporal structure of natural connected speech thus seems to require the joint contribution of both auditory and parietal regions. These findings suggest that cortical tuning to speech rhythm operates on two functionally distinct levels: one encoding the global rhythmic structure of speech and the other associated with online, rapidly evolving temporal predictions. Thus, it may be proposed that speech perception is shaped by evolutionary tuning, a preference for certain speaking rates, and predictive tuning, associated with cortical tracking of the constantly changing-rate of linguistic information in a speech stream.

Development of speech rhythm in first language: The role of syllable intensity variability

The opening-closing alternations of the mouth were viewed as the articulatory basis of speech rhythm. Such articulatory cycles have been observed to highly correlate with the intensity curve of the speech signal. Analysis of the intensity variability in English monolingual children and adults revealed that (1) adults showed significantly smaller intensity variability than children, and (2) intensity variability decreased from intermediate-aged children to older children. Maturation of articulatory motor control is likely to be the main reason for the reduced variability in articulatory cycles, and hence smaller intensity variability in adults and older children.

The Effect of Speech Repetition Rate on Neural Activation in Healthy Adults: Implications for Treatment of Aphasia and Other Fluency Disorders

Functional imaging studies have provided insight into the effect of rate on production of syllables, pseudowords, and naturalistic speech, but the influence of rate on repetition of commonly-used words/phrases suitable for therapeutic use merits closer examination. Aim: To identify speech-motor regions responsive to rate and test the hypothesis that those regions would provide greater support as rates increase, we used an overt speech repetition task and functional magnetic resonance imaging (fMRI) to capture rate-modulated activation within speech-motor regions and determine whether modulations occur linearly and/or show hemispheric preference. Methods: Twelve healthy, right-handed adults participated in an fMRI task requiring overt repetition of commonly-used words/phrases at rates of 1, 2, and 3 syllables/second (syll./sec.). Results: Across all rates, bilateral activation was found both in ventral portions of primary sensorimotor cortex and middle and superior temporal regions. A repeated measures analysis of variance with pairwise comparisons revealed an overall difference between rates in temporal lobe regions of interest (ROIs) bilaterally (p < 0.001); all six comparisons reached significance (p < 0.05). Five of the six were highly significant (p < 0.008), while the left-hemisphere 2- vs. 3-syll./sec. comparison, though still significant, was less robust (p = 0.037). Temporal ROI mean beta-values increased linearly across the three rates bilaterally. Significant rate effects observed in the temporal lobes were slightly more pronounced in the right-hemisphere. No significant overall rate differences were seen in sensorimotor ROIs, nor was there a clear hemispheric effect. Conclusion: Linear effects in superior temporal ROIs suggest that sensory feedback corresponds directly to task demands. The lesser degree of significance in left-hemisphere activation at the faster, closer-to-normal rate may represent an increase in neural efficiency (and therefore, decreased demand) when the task so closely approximates a highly-practiced function. The presence of significant bilateral activation during overt repetition of words/phrases at all three rates suggests that repetition-based speech production may draw support from either or both hemispheres. This bihemispheric redundancy in regions associated with speech-motor control and their sensitivity to changes in rate may play an important role in interventions for nonfluent aphasia and other fluency disorders, particularly when right-hemisphere structures are the sole remaining pathway for production of meaningful speech.

Rhythmic Continuous-Time Coding in the Songbird Analog of Vocal Motor Cortex

Songbirds learn and produce complex sequences of vocal gestures. Adult birdsong requires premotor nucleus HVC, in which projection neurons (PNs) burst sparsely at stereotyped times in the song. It has been hypothesized that PN bursts, as a population, form a continuous sequence, while a different model of HVC function proposes that both HVC PN and interneuron activity is tightly organized around motor gestures. Using a large dataset of PNs and interneurons recorded in singing birds, we test several predictions of these models. We find that PN bursts in adult birds are continuously and nearly uniformly distributed throughout song. However, we also find that PN and interneuron firing rates exhibit significant 10-Hz rhythmicity locked to song syllables, peaking prior to syllable onsets and suppressed prior to offsets-a pattern that predominates PN and interneuron activity in HVC during early stages of vocal learning.