Cortical tracking of voice pitch in the presence of multiple speakers depends on selective attention

EEG-based cross-subject passive music pitch perception using deep learning models

Pitch plays an essential role in music perception and forms the fundamental component of melodic interpretation. However, objectively detecting and decoding brain responses to musical pitch perception across subjects remains to be explored. In this study, we employed electroencephalography (EEG) as an objective measure to obtain the neural responses of musical pitch perception. The EEG signals from 34 subjects under hearing violin sounds at pitches G3 and B6 were collected with an efficient passive Go/No-Go paradigm. The lightweight modified EEGNet model was proposed for EEG-based pitch classification. Specifically, within-subject modeling with the modified EEGNet model was performed to construct individually optimized models. Subsequently, based on the within-subject model pool, a classifier ensemble (CE) method was adopted to construct the cross-subject model. Additionally, we analyzed the optimal time window of brain decoding for pitch perception in the EEG data and discussed the interpretability of these models. The experiment results show that the modified EEGNet model achieved an average classification accuracy of 77% for within-subject modeling, significantly outperforming other compared methods. Meanwhile, the proposed CE method achieved an average accuracy of 74% for cross-subject modeling, significantly exceeding the chance-level accuracy of 50%. Furthermore, we found that the optimal EEG data window for the pitch perception lies 0.4 to 0.9 s onset. These promising results demonstrate that the proposed methods can be effectively used in the objective assessment of pitch perception and have generalization ability in cross-subject modeling.

Neural entrainment to pitch changes of auditory targets in noise

Neural entrainment to certain acoustic features can predict speech-in-noise perception, but these features are difficult to separate. We measured neural responses to both natural speech-in-noise and stimuli (auditory figure-ground) that simulate speech-in-noise without any acoustic or linguistic confounds such as stress contour and semantics. The figure-ground stimulus is formed by multiple temporally coherent pure-tone components embedded in a random tone cloud. Previous work has shown that discrimination of dynamic figure-ground based on the fundamental frequency (F0) of natural speech predicts speech-in-noise recognition independent of hearing and age. In this study, we compared the brain substrate for the figure-ground analysis based on the F0 contour and a statistically similar '1/f' contour with speech-in-noise. We used the temporal response function to predict the electroencephalography responses to the frequency trajectories of the auditory targets. We demonstrate that the brain significantly tracked the pitch changes in both AFG conditions (F0 and 1/F tracking) and a sentence-in-noise condition (F0 tracking) at similar latencies, but at similar magnitudes only when tracking the F0 contour. The pitch-tracking accuracy was consistently high across the delta and theta bands for the AFG condition but not for speech. Sensor-space analysis revealed that speech-in-noise performance correlated with the positive peak amplitude of the F0 figure-ground at 100 ms. Source-space analysis revealed bilateral temporal lobe and hippocampal generators, and strong tracking in the superior parietal lobe for auditory figures and natural speech. In conclusion, our findings demonstrate that the human brain reliably tracks the F0 trajectory of both speech and a non-linguistic figure in noise, with speech tracking showing reduced accuracy in the theta band compared to figure-ground tracking. Despite the difference in prediction accuracy, we reveal striking similarities in neural entrainment patterns and source locations between the two paradigms. These results suggest that neural entrainment engages high-level cortical mechanisms independent of linguistic content. Furthermore, we show that TRF peak amplitude serves as a potential biomarker for speech-in-noise ability, highlighting possible clinical applications.

Neural speech tracking in a virtual acoustic environment: audio-visual benefit for unscripted continuous speech

The audio-visual benefit in speech perception-where congruent visual input enhances auditory processing-is well-documented across age groups, particularly in challenging listening conditions and among individuals with varying hearing abilities. However, most studies rely on highly controlled laboratory environments with scripted stimuli. Here, we examine the audio-visual benefit using unscripted, natural speech from untrained speakers within a virtual acoustic environment. Using electroencephalography (EEG) and cortical speech tracking, we assessed neural responses across audio-visual, audio-only, visual-only, and masked-lip conditions to isolate the role of lip movements. Additionally, we analysed individual differences in acoustic and visual features of the speakers, including pitch, jitter, and lip-openness, to explore their influence on the audio-visual speech tracking benefit. Results showed a significant audio-visual enhancement in speech tracking with background noise, with the masked-lip condition performing similarly to the audio-only condition, emphasizing the importance of lip movements in adverse listening situations. Our findings reveal the feasibility of cortical speech tracking with naturalistic stimuli and underscore the impact of individual speaker characteristics on audio-visual integration in real-world listening contexts.

Dynamics of Pitch Perception in the Auditory Cortex

The ability to perceive pitch allows human listeners to experience music, recognize the identity and emotion conveyed by conversational partners, and make sense of their auditory environment. A pitch percept is formed by weighting different acoustic cues (e.g., signal fundamental frequency and interharmonic spacing) and contextual cues (expectation). How and when such cues are neurally encoded and integrated remains debated. In this study, 28 participants (16 female) listened to tone sequences with different acoustic cues (pure tones, complex missing fundamental tones, and tones with an ambiguous mixture), placed in predictable and less predictable sequences, while magnetoencephalography was recorded. Decoding analyses revealed that pitch was encoded in neural responses to all three tone types in the low-to-mid auditory cortex and sensorimotor cortex bilaterally, with right-hemisphere dominance. The pattern of activity generalized across cue types, offset in time: pitch was neurally encoded earlier for harmonic tones (∼85 ms) than pure tones (∼95 ms). For ambiguous tones, pitch emerged significantly earlier in predictable contexts than in unpredictable. The results suggest that a unified neural representation of pitch emerges by integrating independent pitch cues and that context alters the dynamics of pitch generation when acoustic cues are ambiguous.

Cortical tracking of speakers' spectral changes predicts selective listening

A social scene is particularly informative when people are distinguishable. To understand somebody amid a "cocktail party" chatter, we automatically index their voice. This ability is underpinned by parallel processing of vocal spectral contours from speech sounds, but it has not yet been established how this occurs in the brain's cortex. We investigate single-trial neural tracking of slow frequency modulations in speech using electroencephalography. Participants briefly listened to unfamiliar single speakers, and in addition, they performed a cocktail party comprehension task. Quantified through stimulus reconstruction methods, robust tracking was found in neural responses to slow (delta-theta range) modulations of frequency contours in the fourth and fifth formant band, equivalent to the 3.5-5 KHz audible range. The spectral spacing between neighboring instantaneous frequency contours (ΔF), which also yields indexical information from the vocal tract, was similarly decodable. Moreover, EEG evidence of listeners' spectral tracking abilities predicted their chances of succeeding at selective listening when faced with two-speaker speech mixtures. In summary, the results indicate that the communicating brain can rely on locking of cortical rhythms to major changes led by upper resonances of the vocal tract. Their corresponding articulatory mechanics hence continuously issue a fundamental credential for listeners to target in real time.

Neural encoding of melodic expectations in music across EEG frequency bands

The human brain tracks regularities in the environment and extrapolates these to predict future events. Prior work on music cognition suggests that low-frequency (1-8 Hz) brain activity encodes melodic predictions beyond the stimulus acoustics. Building on this work, we aimed to disentangle the frequency-specific neural dynamics linked to melodic prediction uncertainty (modelled as entropy) and prediction error (modelled as surprisal) for temporal (note onset) and content (note pitch) information. By using multivariate temporal response function (TRF) models, we re-analysed the electroencephalogram (EEG) from 20 subjects (10 musicians) who listened to Western tonal music. Our results show that melodic expectation metrics improve the EEG reconstruction accuracy in all frequency bands below the gamma range (< 30 Hz). Crucially, we found that entropy contributed more strongly to the reconstruction accuracy enhancement compared to surprisal in all frequency bands. Additionally, we found that the encoding of temporal, but not content, information metrics was not limited to low frequencies, rather it extended to higher frequencies (> 8 Hz). An analysis of the TRF weights revealed that the temporal predictability of a note (entropy of note onset) may be encoded in the delta- (1-4 Hz) and beta-band (12-30 Hz) brain activity prior to the stimulus, suggesting that these frequency bands associate with temporal predictions. Strikingly, we also revealed that melodic expectations selectively enhanced EEG reconstruction accuracy in the beta band for musicians, and in the alpha band (8-12 Hz) for non-musicians, suggesting that musical expertise influences the neural dynamics underlying predictive processing in music cognition.

Episodic long-term memory formation during slow-wave sleep

We are unresponsive during slow-wave sleep but continue monitoring external events for survival. Our brain wakens us when danger is imminent. If events are non-threatening, our brain might store them for later consideration to improve decision-making. To test this hypothesis, we examined whether novel vocabulary consisting of simultaneously played pseudowords and translation words are encoded/stored during sleep, and which neural-electrical events facilitate encoding/storage. An algorithm for brain-state-dependent stimulation selectively targeted word pairs to slow-wave peaks or troughs. Retrieval tests were given 12 and 36 hr later. These tests required decisions regarding the semantic category of previously sleep-played pseudowords. The sleep-played vocabulary influenced awake decision-making 36 hr later, if targeted to troughs. The words' linguistic processing raised neural complexity. The words' semantic-associative encoding was supported by increased theta power during the ensuing peak. Fast-spindle power ramped up during a second peak likely aiding consolidation. Hence, new vocabulary played during slow-wave sleep was stored and influenced decision-making days later.

Attentional Modulation of the Cortical Contribution to the Frequency-Following Response Evoked by Continuous Speech

Selective attention to one of several competing speakers is required for comprehending a target speaker among other voices and for successful communication with them. It moreover has been found to involve the neural tracking of low-frequency speech rhythms in the auditory cortex. Effects of selective attention have also been found in subcortical neural activities, in particular regarding the frequency-following response related to the fundamental frequency of speech (speech-FFR). Recent investigations have, however, shown that the speech-FFR contains cortical contributions as well. It remains unclear whether these are also modulated by selective attention. Here we used magnetoencephalography to assess the attentional modulation of the cortical contributions to the speech-FFR. We presented both male and female participants with two competing speech signals and analyzed the cortical responses during attentional switching between the two speakers. Our findings revealed robust attentional modulation of the cortical contribution to the speech-FFR: the neural responses were higher when the speaker was attended than when they were ignored. We also found that, regardless of attention, a voice with a lower fundamental frequency elicited a larger cortical contribution to the speech-FFR than a voice with a higher fundamental frequency. Our results show that the attentional modulation of the speech-FFR does not only occur subcortically but extends to the auditory cortex as well.SIGNIFICANCE STATEMENT Understanding speech in noise requires attention to a target speaker. One of the speech features that a listener can use to identify a target voice among others and attend it is the fundamental frequency, together with its higher harmonics. The fundamental frequency arises from the opening and closing of the vocal folds and is tracked by high-frequency neural activity in the auditory brainstem and in the cortex. Previous investigations showed that the subcortical neural tracking is modulated by selective attention. Here we show that attention affects the cortical tracking of the fundamental frequency as well: it is stronger when a particular voice is attended than when it is ignored.

Hemispheric asymmetries for music and speech: Spectrotemporal modulations and top-down influences

Hemispheric asymmetries in auditory cognition have been recognized for a long time, but their neural basis is still debated. Here I focus on specialization for processing of speech and music, the two most important auditory communication systems that humans possess. A great deal of evidence from lesion studies and functional imaging suggests that aspects of music linked to the processing of pitch patterns depend more on right than left auditory networks. A complementary specialization for temporal resolution has been suggested for left auditory networks. These diverse findings can be integrated within the context of the spectrotemporal modulation framework, which has been developed as a way to characterize efficient neuronal encoding of complex sounds. Recent studies show that degradation of spectral modulation impairs melody perception but not speech content, whereas degradation of temporal modulation has the opposite effect. Neural responses in the right and left auditory cortex in those studies are linked to processing of spectral and temporal modulations, respectively. These findings provide a unifying model to understand asymmetries in terms of sensitivity to acoustical features of communication sounds in humans. However, this explanation does not account for evidence that asymmetries can shift as a function of learning, attention, or other top-down factors. Therefore, it seems likely that asymmetries arise both from bottom-up specialization for acoustical modulations and top-down influences coming from hierarchically higher components of the system. Such interactions can be understood in terms of predictive coding mechanisms for perception.

Cocktail party training induces increased speech intelligibility and decreased cortical activity in bilateral inferior frontal gyri. A functional near-infrared study

The human brain networks responsible for selectively listening to a voice amid other talkers remain to be clarified. The present study aimed to investigate relationships between cortical activity and performance in a speech-in-speech task, before (Experiment I) and after training-induced improvements (Experiment II). In Experiment I, 74 participants performed a speech-in-speech task while their cortical activity was measured using a functional near infrared spectroscopy (fNIRS) device. One target talker and one masker talker were simultaneously presented at three different target-to-masker ratios (TMRs): adverse, intermediate and favorable. Behavioral results show that performance may increase monotonically with TMR in some participants and failed to decrease, or even improved, in the adverse-TMR condition for others. On the neural level, an extensive brain network including the frontal (left prefrontal cortex, right dorsolateral prefrontal cortex and bilateral inferior frontal gyri) and temporal (bilateral auditory cortex) regions was more solicited by the intermediate condition than the two others. Additionally, bilateral frontal gyri and left auditory cortex activities were found to be positively correlated with behavioral performance in the adverse-TMR condition. In Experiment II, 27 participants, whose performance was the poorest in the adverse-TMR condition of Experiment I, were trained to improve performance in that condition. Results show significant performance improvements along with decreased activity in bilateral inferior frontal gyri, the right dorsolateral prefrontal cortex, the left inferior parietal cortex and the right auditory cortex in the adverse-TMR condition after training. Arguably, lower neural activity reflects higher efficiency in processing masker inhibition after speech-in-speech training. As speech-in-noise tasks also imply frontal and temporal regions, we suggest that regardless of the type of masking (speech or noise) the complexity of the task will prompt the implication of a similar brain network. Furthermore, the initial significant cognitive recruitment will be reduced following a training leading to an economy of cognitive resources.