Decoding speech sounds from neurophysiological data: Practical considerations and theoretical implications

Machine learning techniques have proven to be a useful tool in cognitive neuroscience. However, their implementation in scalp-recorded electroencephalography (EEG) is relatively limited. To address this, we present three analyses using data from a previous study that examined event-related potential (ERP) responses to a wide range of naturally-produced speech sounds. First, we explore which features of the EEG signal best maximize machine learning accuracy for a voicing distinction, using a support vector machine (SVM). We manipulate three dimensions of the EEG signal as input to the SVM: number of trials averaged, number of time points averaged, and polynomial fit. We discuss the trade-offs in using different feature sets and offer some recommendations for researchers using machine learning. Next, we use SVMs to classify specific pairs of phonemes, finding that we can detect differences in the EEG signal that are not otherwise detectable using conventional ERP analyses. Finally, we characterize the timecourse of phonetic feature decoding across three phonological dimensions (voicing, manner of articulation, and place of articulation), and find that voicing and manner are decodable from neural activity, whereas place of articulation is not. This set of analyses addresses both practical considerations in the application of machine learning to EEG, particularly for speech studies, and also sheds light on current issues regarding the nature of perceptual representations of speech.

Immediate neural impact and incomplete compensation after semantic hub disconnection

The human brain extracts meaning using an extensive neural system for semantic knowledge. Whether broadly distributed systems depend on or can compensate after losing a highly interconnected hub is controversial. We report intracranial recordings from two patients during a speech prediction task, obtained minutes before and after neurosurgical treatment requiring disconnection of the left anterior temporal lobe (ATL), a candidate semantic knowledge hub. Informed by modern diaschisis and predictive coding frameworks, we tested hypotheses ranging from solely neural network disruption to complete compensation by the indirectly affected language-related and speech-processing sites. Immediately after ATL disconnection, we observed neurophysiological alterations in the recorded frontal and auditory sites, providing direct evidence for the importance of the ATL as a semantic hub. We also obtained evidence for rapid, albeit incomplete, attempts at neural network compensation, with neural impact largely in the forms stipulated by the predictive coding framework, in specificity, and the modern diaschisis framework, more generally. The overall results validate these frameworks and reveal an immediate impact and capability of the human brain to adjust after losing a brain hub.

Supra-second interval timing in bipolar disorder: examining the role of disorder sub-type, mood, and medication status

BACKGROUND

Widely reported by bipolar disorder (BD) patients, cognitive symptoms, including deficits in executive function, memory, attention, and timing are under-studied. Work suggests that individuals with BD show impairments in interval timing tasks, including supra-second, sub-second, and implicit motor timing compared to the neuronormative population. However, how time perception differs within individuals with BD based on disorder sub-type (BDI vs II), depressed mood, or antipsychotic medication-use has not been thoroughly investigated. The present work administered a supra-second interval timing task concurrent with electroencephalography (EEG) to patients with BD and a neuronormative comparison group. As this task is known to elicit frontal theta oscillations, signal from the frontal (Fz) lead was analyzed at rest and during the task.

RESULTS

Results suggest that individuals with BD show impairments in supra-second interval timing and reduced frontal theta power during the task compared to neuronormative controls. However, within BD sub-groups, neither time perception nor frontal theta differed in accordance with BD sub-type, depressed mood, or antipsychotic medication use.

CONCLUSIONS

This work suggests that BD sub-type, depressed mood status or antipsychotic medication use does not alter timing profile or frontal theta activity. Together with previous work, these findings point to timing impairments in BD patients across a wide range of modalities and durations indicating that an altered ability to assess the passage of time may be a fundamental cognitive abnormality in BD.

Supra-second interval timing in bipolar disorder: examining the role of disorder sub-type, mood, and medication status

Background : Widely reported by bipolar disorder (BD) patients, cognitive symptoms, including deficits in executive function, memory, attention, and timing are under-studied. Work suggests that individuals with BD show impairments in interval timing tasks, including supra-second, sub-second, and implicit motor timing compared to the neuronormative population. However, how time perception differs within individuals with BD based on BD sub-type (BDI vs II), mood, or antipsychotic medication-use has not been thoroughly investigated. The present work administered a supra-second interval timing task concurrent with electroencephalography (EEG) to patients with BD and a neuronormative comparison group. As this task is known to elicit frontal theta oscillations, signal from the frontal (Fz) lead was analyzed at rest and during the task. Results : Results suggest that individuals with BD show impairments in supra-second interval timing and reduced frontal theta power compared during the task to neuronormative controls. However, within BD sub-groups, neither time perception nor frontal theta differed in accordance with BD sub-type, mood, or antipsychotic medication use. Conclusions : his work suggests that BD sub-type, mood status or antipsychotic medication use does not alter timing profile or frontal theta activity. Together with previous work, these findings point to timing impairments in BD patients across a wide range of modalities and durations indicating that an altered ability to assess the passage of time may be a fundamental cognitive abnormality in BD.

Decoding the temporal dynamics of spoken word and nonword processing from EEG

The efficiency of spoken word recognition is essential for real-time communication. There is consensus that this efficiency relies on an implicit process of activating multiple word candidates that compete for recognition as the acoustic signal unfolds in real-time. However, few methods capture the neural basis of this dynamic competition on a msec-by-msec basis. This is crucial for understanding the neuroscience of language, and for understanding hearing, language and cognitive disorders in people for whom current behavioral methods are not suitable. We applied machine-learning techniques to standard EEG signals to decode which word was heard on each trial and analyzed the patterns of confusion over time. Results mirrored psycholinguistic findings: Early on, the decoder was equally likely to report the target (e.g., baggage) or a similar sounding competitor (badger), but by around 500 msec, competitors were suppressed. Follow up analyses show that this is robust across EEG systems (gel and saline), with fewer channels, and with fewer trials. Results are robust within individuals and show high reliability. This suggests a powerful and simple paradigm that can assess the neural dynamics of speech decoding, with potential applications for understanding lexical development in a variety of clinical disorders.

Dynamic EEG analysis during language comprehension reveals interactive cascades between perceptual processing and sentential expectations

Understanding spoken language requires analysis of the rapidly unfolding speech signal at multiple levels: acoustic, phonological, and semantic. However, there is not yet a comprehensive picture of how these levels relate. We recorded electroencephalography (EEG) while listeners (N = 31) heard sentences in which we manipulated acoustic ambiguity (e.g., a bees/peas continuum) and sentential expectations (e.g., Honey is made by bees). EEG was analyzed with a mixed effects model over time to quantify how language processing cascades proceed on a millisecond-by-millisecond basis. Our results indicate: (1) perceptual processing and memory for fine-grained acoustics is preserved in brain activity for up to 900 msec; (2) contextual analysis begins early and is graded with respect to the acoustic signal; and (3) top-down predictions influence perceptual processing in some cases, however, these predictions are available simultaneously with the veridical signal. These mechanistic insights provide a basis for a better understanding of the cortical language network.