Time course of word identification and semantic integration in spoken language

Neurophysiological signatures of prediction in language: A critical review of anticipatory negativities

Recent event-related potential (ERP) studies in language comprehension converge in finding anticipatory negativities preceding words or word segments that can be pre-activated based on either sentence contexts or phonological cues. We review these findings from different paradigms in the light of evidence from other cognitive domains in which slow negative potentials have long been associated with anticipatory processes and discuss their potential underlying mechanisms. We propose that this family of anticipatory negativities captures common mechanisms associated with the pre-activation of linguistic information both within words and within sentences. Future studies could utilize these anticipatory negativities in combination with other, well-established ERPs, to simultaneously track prediction-related processes emerging at different time intervals (before and after the perception of pre-activated input) and with distinct time courses (shorter-lived and longer-lived cognitive operations).

Incompleteness features in the descriptive discourse of Chinese elders with and without Alzheimer's disease

Alzheimer's disease (AD) can manifest itself with prominent language dysfunction. Incompleteness in discourse refers to the lack of indispensable sentence-constructing elements that hinder communication fluency and accuracy. The current study investigates how the pattern of incompleteness is associated with the descriptive discourse produced by elders withoutAD and those with different stages ofAD. The Chinese discourse samples were collected from the picture description of 40 elders with mild probableAD (Mini-Mental State Examination (MMSE) 21-26, Montreal Cognitive Assessment Scale-Basic (MoCA-B) 15-19), 40 elders with moderate probableAD (MMSE 11-20, MoCA-B 10-14), and 40 controls (MMSE 26-29, MoCA-B 24-29). The total production of incomplete sentences and six incompleteness features were examined. The MildAD, ModerateAD, and Control groups differed in the total output of the incomplete sentence. Group differences also emerged in four incompleteness features: missing subject, missing predicate, missing object, and missing functional word. The ModerateAD group differed from the MildAD group with respect to most significant features, while MildAD and Control groups were very similar. The results suggested thatAD impairs the sentence construction ability of Chinese elders, especially at the later stage. These statistically significant differences between the groups might provide some references when diagnosing the risk and possibility of cognitive impairment of Chinese elders, facilitating the design of clinical evaluation or screening for probableAD.

Relative Contributions of Social, Contextual, and Lexical Factors in Speech Processing

This exploratory study examined the simultaneous interactions and relative contributions of bottom-up social information (regional dialect, speaking style), top-down contextual information (semantic predictability), and the internal dynamics of the lexicon (neighborhood density, lexical frequency) to lexical access and word recognition. Cross-modal matching and intelligibility in noise tasks were conducted with a community sample of adults at a local science museum. Each task featured one condition in which keywords were presented in isolation and one condition in which they were presented within a multiword phrase. Lexical processing was slower and more accurate when keywords were presented in their phrasal context, and was both faster and more accurate for auditory stimuli produced in the local Midland dialect. In both tasks, interactions were observed among stimulus dialect, speaking style, semantic predictability, phonological neighborhood density, and lexical frequency. These interactions revealed that bottom-up social information and top-down contextual information contribute more to speech processing than the internal dynamics of the lexicon. Moreover, the relatively stronger bottom-up social effects were observed in both the isolated word and multiword phrase conditions, suggesting that social variation is central to speech processing, even in non-interactive laboratory tasks. At the same time, the specific interactions observed differed between the two experiments, reflecting task-specific demands related to processing time constraints and signal degradation.

Speech comprehension in noisy environments: Evidence from the predictability effects on the N400 and LPC

Introduction

Speech comprehension involves context-based lexical predictions for efficient semantic integration. This study investigated how noise affects the predictability effect on event-related potentials (ERPs) such as the N400 and late positive component (LPC) in speech comprehension.

Methods

Twenty-seven listeners were asked to comprehend sentences in clear and noisy conditions (hereinafter referred to as "clear speech" and "noisy speech," respectively) that ended with a high-or low-predictability word during electroencephalogram (EEG) recordings.

Results

The study results regarding clear speech showed the predictability effect on the N400, wherein low-predictability words elicited a larger N400 amplitude than did high-predictability words in the centroparietal and frontocentral regions. Noisy speech showed a reduced and delayed predictability effect on the N400 in the centroparietal regions. Additionally, noisy speech showed a predictability effect on the LPC in the centroparietal regions.

Discussion

These findings suggest that listeners achieve comprehension outcomes through different neural mechanisms according to listening conditions. Noisy speech may be comprehended with a second-pass process that possibly functions to recover the phonological form of degraded speech through phonetic reanalysis or repair, thus compensating for decreased predictive efficiency.

Neurocognitive Dynamics of Prosodic Salience over Semantics during Explicit and Implicit Processing of Basic Emotions in Spoken Words

How language mediates emotional perception and experience is poorly understood. The present event-related potential (ERP) study examined the explicit and implicit processing of emotional speech to differentiate the relative influences of communication channel, emotion category and task type in the prosodic salience effect. Thirty participants (15 women) were presented with spoken words denoting happiness, sadness and neutrality in either the prosodic or semantic channel. They were asked to judge the emotional content (explicit task) and speakers' gender (implicit task) of the stimuli. Results indicated that emotional prosody (relative to semantics) triggered larger N100, P200 and N400 amplitudes with greater delta, theta and alpha inter-trial phase coherence (ITPC) and event-related spectral perturbation (ERSP) values in the corresponding early time windows, and continued to produce larger LPC amplitudes and faster responses during late stages of higher-order cognitive processing. The relative salience of prosodic and semantics was modulated by emotion and task, though such modulatory effects varied across different processing stages. The prosodic salience effect was reduced for sadness processing and in the implicit task during early auditory processing and decision-making but reduced for happiness processing in the explicit task during conscious emotion processing. Additionally, across-trial synchronization of delta, theta and alpha bands predicted the ERP components with higher ITPC and ERSP values significantly associated with stronger N100, P200, N400 and LPC enhancement. These findings reveal the neurocognitive dynamics of emotional speech processing with prosodic salience tied to stage-dependent emotion- and task-specific effects, which can reveal insights into understanding language and emotion processing from cross-linguistic/cultural and clinical perspectives.

The electrophysiological correlates of word pre-activation during associative word learning

Human beings continuously make use of learned associations to generate predictions about future occurrences in the environment. Such memory-related predictive processes provide a scaffold for learning in that mental representations of foreseeable events can be adjusted or strengthened based on a specific outcome. Learning the meaning of novel words through picture-word associations constitutes a prime example of associative learning because pictures preceding words can trigger word prediction through the pre-activation of the related mnemonic representations. In the present electroencephalography (EEG) study, we used event-related potentials (ERPs) to compare neural indices of word pre-activation between a word learning condition with maximal prediction likelihood and a non-learning control condition with low prediction. Results revealed that prediction-related N400 amplitudes in response to pictures decreased over time at central electrodes as a function of word learning, whereas late positive component (LPC) amplitudes increased. Notably, N400 but not LPC changes were also predictive of word learning performance, suggesting that the N400 component constitutes a sensitive marker of word pre-activation during associative word learning.

Can You Hear What's Coming? Failure to Replicate ERP Evidence for Phonological Prediction

Prediction-based theories of language comprehension assume that listeners predict both the meaning and phonological form of likely upcoming words. In alleged event-related potential (ERP) demonstrations of phonological prediction, prediction-mismatching words elicit a phonological mismatch negativity (PMN), a frontocentral negativity that precedes the centroparietal N400 component. However, classification and replicability of the PMN has proven controversial, with ongoing debate on whether the PMN is a distinct component or merely an early part of the N400. In this electroencephalography (EEG) study, we therefore attempted to replicate the PMN effect and its separability from the N400, using a participant sample size (N = 48) that was more than double that of previous studies. Participants listened to sentences containing either a predictable word or an unpredictable word with/without phonological overlap with the predictable word. Preregistered analyses revealed a widely distributed negative-going ERP in response to unpredictable words in both the early (150-250 ms) and the N400 (300-500 ms) time windows. Bayes factor analysis yielded moderate evidence against a different scalp distribution of the effects in the two time windows. Although our findings do not speak against phonological prediction during sentence comprehension, they do speak against the PMN effect specifically as a marker of phonological prediction mismatch. Instead of an PMN effect, our results demonstrate the early onset of the auditory N400 effect associated with unpredictable words. Our failure to replicate further highlights the risk associated with commonly employed data-contingent analyses (e.g., analyses involving time windows or electrodes that were selected based on visual inspection) and small sample sizes in the cognitive neuroscience of language.

Context Matters for Tone and Intonation Processing in Mandarin

In tonal languages such as Mandarin, both lexical tone and sentence intonation are primarily signaled by F0. Their F0 encodings are sometimes in conflict and sometimes in congruency. The present study investigated how tone and intonation, with F0 encodings in conflict or in congruency, are processed and how semantic context may affect their processing. To this end, tone and intonation identification experiments were conducted in both semantically neutral and constraining contexts. Results showed that the overall performance of tone identification was better than that of intonation. Specifically, tone identification was seldom affected by intonation information irrespective of semantic contexts. However, intonation identification, particularly question intonation, was susceptible to the final lexical tone identity and affected by the semantic context. In the semantically neutral context, questions ending with a rising tone and a falling tone were equally difficult to identify. In the semantically constraining context, questions ending with a falling tone were much better identified than those ending with a rising tone. This perceptual asymmetry suggests that top-down information provided by the semantically constraining context can play a facilitating role for listeners to disentangle intonational information from tonal information, but mainly in sentences with the lexical falling tone in the final position.

Parallel processing in speech perception with local and global representations of linguistic context

Speech processing is highly incremental. It is widely accepted that human listeners continuously use the linguistic context to anticipate upcoming concepts, words, and phonemes. However, previous evidence supports two seemingly contradictory models of how a predictive context is integrated with the bottom-up sensory input: Classic psycholinguistic paradigms suggest a two-stage process, in which acoustic input initially leads to local, context-independent representations, which are then quickly integrated with contextual constraints. This contrasts with the view that the brain constructs a single coherent, unified interpretation of the input, which fully integrates available information across representational hierarchies, and thus uses contextual constraints to modulate even the earliest sensory representations. To distinguish these hypotheses, we tested magnetoencephalography responses to continuous narrative speech for signatures of local and unified predictive models. Results provide evidence that listeners employ both types of models in parallel. Two local context models uniquely predict some part of early neural responses, one based on sublexical phoneme sequences, and one based on the phonemes in the current word alone; at the same time, even early responses to phonemes also reflect a unified model that incorporates sentence-level constraints to predict upcoming phonemes. Neural source localization places the anatomical origins of the different predictive models in nonidentical parts of the superior temporal lobes bilaterally, with the right hemisphere showing a relative preference for more local models. These results suggest that speech processing recruits both local and unified predictive models in parallel, reconciling previous disparate findings. Parallel models might make the perceptual system more robust, facilitate processing of unexpected inputs, and serve a function in language acquisition.

Inhibition Effect of Audio-Visual Semantic Interference in Chinese Interface: An ERP Study of Concrete Icons and Chinese Characters

In this study, event-related potential (ERP) was used to examine whether the brain has an inhibition effect on the interference of audio-visual information in the Chinese interface. Concrete icons (flame and snowflake) or Chinese characters (暖 and 凉) with opposite semantics were used as target carriers, and colors (red and blue) and speeches (热 and 冷) were used as audio-visual intervention stimuli. In the experiment, target carrier and audio-visual intervention were presented in a random combination, and the subjects needed to determine whether the semantics of the two matched quickly. By comparing the overall cognitive performance of two carriers, it was found that the brain had a more significant inhibition effect on audio-visual intervention stimuli with different semantics (SBH/LBH and SRC/LRC) relative to the same semantics (SRH/LRH). The semantic mismatch caused significant N400, indicating that semantic interference in the interface information would trigger the brain’s inhibition effect. Therefore, the more complex the semantic matching of interface information was, the higher the amplitude of N400 became. The results confirmed that the semantic relationship between target carrier and audio-visual intervention was the key factor affecting the cognitive inhibition effect. Moreover, under different intervention stimuli, the ERP’s negative activity caused by Chinese characters in frontal and parietal-occipital regions was more evident than that by concrete icons, indicating that concrete icons had a lower inhibition effect than Chinese characters. Therefore, we considered that this inhibition effect was based on the semantic constraints of the target carrier itself, which might come from the knowledge learning and intuitive experience stored in the human brain.

Activation of Literal Word Meanings in Idioms: Evidence from Eye-tracking and ERP Experiments

How the language processing system handles formulaic language such as idioms is a matter of debate. We investigated the activation of constituent meanings by means of predictive processing in an eye-tracking experiment and in two ERP experiments (auditory and visual). In the eye-tracking experiment, German-speaking participants listened to idioms in which the final word was excised (Hannes let the cat out of the . . .). Well before the offset of these idiom fragments, participants fixated on the correct idiom completion (bag) more often than on unrelated distractors (stomach). Moreover, there was an early fixation bias towards semantic associates (basket) of the correct completion, which ended shortly after the offset of the fragment. In the ERP experiments, sentences (spoken or written) either contained complete idioms, or the final word of the idiom was replaced with a semantic associate or with an unrelated word. Across both modalities, ERPs reflected facilitated processing of correct completions across several regions of interest (ROIs) and time windows. Facilitation of semantic associates was only reliably evident in early components for auditory idiom processing. The ERP findings for spoken idioms compliment the eye-tracking data by pointing to early decompositional processing of idioms. It seems that in spoken idiom processing, holistic representations do not solely determine lexical processing.

The Same Ultra-Rapid Parallel Brain Dynamics Underpin the Production and Perception of Speech

The temporal dynamics by which linguistic information becomes available is one of the key properties to understand how language is organized in the brain. An unresolved debate between different brain language models is whether words, the building blocks of language, are activated in a sequential or parallel manner. In this study, we approached this issue from a novel perspective by directly comparing the time course of word component activation in speech production versus perception. In an overt object naming task and a passive listening task, we analyzed with mixed linear models at the single-trial level the event-related brain potentials elicited by the same lexico-semantic and phonological word knowledge in the two language modalities. Results revealed that both word components manifested simultaneously as early as 75 ms after stimulus onset in production and perception; differences between the language modalities only became apparent after 300 ms of processing. The data provide evidence for ultra-rapid parallel dynamics of language processing and are interpreted within a neural assembly framework where words recruit the same integrated cell assemblies across production and perception. These word assemblies ignite early on in parallel and only later on reverberate in a behavior-specific manner.

Dissociable electrophysiological measures of natural language processing reveal differences in speech comprehension strategy in healthy ageing

Healthy ageing leads to changes in the brain that impact upon sensory and cognitive processing. It is not fully clear how these changes affect the processing of everyday spoken language. Prediction is thought to play an important role in language comprehension, where information about upcoming words is pre-activated across multiple representational levels. However, evidence from electrophysiology suggests differences in how older and younger adults use context-based predictions, particularly at the level of semantic representation. We investigate these differences during natural speech comprehension by presenting older and younger subjects with continuous, narrative speech while recording their electroencephalogram. We use time-lagged linear regression to test how distinct computational measures of (1) semantic dissimilarity and (2) lexical surprisal are processed in the brains of both groups. Our results reveal dissociable neural correlates of these two measures that suggest differences in how younger and older adults successfully comprehend speech. Specifically, our results suggest that, while younger and older subjects both employ context-based lexical predictions, older subjects are significantly less likely to pre-activate the semantic features relating to upcoming words. Furthermore, across our group of older adults, we show that the weaker the neural signature of this semantic pre-activation mechanism, the lower a subject’s semantic verbal fluency score. We interpret these findings as prediction playing a generally reduced role at a semantic level in the brains of older listeners during speech comprehension and that these changes may be part of an overall strategy to successfully comprehend speech with reduced cognitive resources.

Active listening

This paper introduces active listening, as a unified framework for synthesising and recognising speech. The notion of active listening inherits from active inference, which considers perception and action under one universal imperative: to maximise the evidence for our (generative) models of the world. First, we describe a generative model of spoken words that simulates (i) how discrete lexical, prosodic, and speaker attributes give rise to continuous acoustic signals; and conversely (ii) how continuous acoustic signals are recognised as words. The 'active' aspect involves (covertly) segmenting spoken sentences and borrows ideas from active vision. It casts speech segmentation as the selection of internal actions, corresponding to the placement of word boundaries. Practically, word boundaries are selected that maximise the evidence for an internal model of how individual words are generated. We establish face validity by simulating speech recognition and showing how the inferred content of a sentence depends on prior beliefs and background noise. Finally, we consider predictive validity by associating neuronal or physiological responses, such as the mismatch negativity and P300, with belief updating under active listening, which is greatest in the absence of accurate prior beliefs about what will be heard next.

Generative models, linguistic communication and active inference

This paper presents a biologically plausible generative model and inference scheme that is capable of simulating communication between synthetic subjects who talk to each other. Building on active inference formulations of dyadic interactions, we simulate linguistic exchange to explore generative models that support dialogues. These models employ high-order interactions among abstract (discrete) states in deep (hierarchical) models. The sequential nature of language processing mandates generative models with a particular factorial structure-necessary to accommodate the rich combinatorics of language. We illustrate linguistic communication by simulating a synthetic subject who can play the 'Twenty Questions' game. In this game, synthetic subjects take the role of the questioner or answerer, using the same generative model. This simulation setup is used to illustrate some key architectural points and demonstrate that many behavioural and neurophysiological correlates of linguistic communication emerge under variational (marginal) message passing, given the right kind of generative model. For example, we show that theta-gamma coupling is an emergent property of belief updating, when listening to another.

Combined Brain-Perfusion SPECT and EEG Measurements Suggest Distinct Strategies for Speech Comprehension in CI Users With Higher and Lower Performance

Cochlear implantation constitutes a successful therapy of inner ear deafness, with the majority of patients showing good outcomes. There is, however, still some unexplained variability in outcomes with a number of cochlear-implant (CI) users, showing major limitations in speech comprehension. The current study used a multimodal diagnostic approach combining single-photon emission computed tomography (SPECT) and electroencephalography (EEG) to examine the mechanisms underlying speech processing in postlingually deafened CI users (N = 21). In one session, the participants performed a speech discrimination task, during which a 96-channel EEG was recorded and the perfusions marker 99mTc-HMPAO was injected intravenously. The SPECT scan was acquired 1.5 h after injection to measure the cortical activity during the speech task. The second session included a SPECT scan after injection without stimulation at rest. Analysis of EEG and SPECT data showed N400 and P600 event-related potentials (ERPs) particularly evoked by semantic violations in the sentences, and enhanced perfusion in a temporo-frontal network during task compared to rest, involving the auditory cortex bilaterally and Broca's area. Moreover, higher performance in testing for word recognition and verbal intelligence strongly correlated to the activation in this network during the speech task. However, comparing CI users with lower and higher speech intelligibility [median split with cutoff + 7.6 dB signal-to-noise ratio (SNR) in the Göttinger sentence test] revealed for CI users with higher performance additional activations of parietal and occipital regions and for those with lower performance stronger activation of superior frontal areas. Furthermore, SPECT activity was tightly coupled with EEG and cognitive abilities, as indicated by correlations between (1) cortical activation and the amplitudes in EEG, N400 (temporal and occipital areas)/P600 (parietal and occipital areas) and (2) between cortical activation in left-sided temporal and bilateral occipital/parietal areas and working memory capacity. These results suggest the recruitment of a temporo-frontal network in CI users during speech processing and a close connection between ERP effects and cortical activation in CI users. The observed differences in speech-evoked cortical activation patterns for CI users with higher and lower speech intelligibility suggest distinct processing strategies during speech rehabilitation with CI.

Neural Processes Underlying Nonword Rhyme Differentiate Eventual Stuttering Persistence and Recovery

Purpose Phonological skills have been associated with developmental stuttering. The current study aimed to determine whether the neural processes underlying phonology, specifically for nonword rhyming, differentiated stuttering persistence and recovery. Method Twenty-six children who stutter (CWS) and 18 children who do not stutter, aged 5 years, completed an auditory nonword rhyming task. Event-related brain potentials were elicited by prime, rhyming, and nonrhyming targets. CWS were followed longitudinally to determine eventual persistence (n = 14) or recovery (n = 12). This is a retrospective analysis of data acquired when all CWS presented as stuttering. Results CWS who eventually recovered and children who do not stutter exhibited the expected rhyme effect, with larger event-related brain potential amplitudes elicited by nonrhyme targets compared to rhyme targets. In contrast, CWS who eventually persisted exhibited a reverse rhyme effect, with larger responses to rhyme than nonrhyme targets. Conclusions These findings suggest that CWS who eventually persisted are not receiving the same benefit of phonological priming as CWS who eventually recovered for complex nonword rhyming tasks. These results indicate divergent patterns of phonological processing in young CWS who eventually persisted, especially for difficult tasks with limited semantic context, and suggest that the age of 5 years may be an important developmental period for phonology in CWS. Supplemental Material https://doi.org/10.23641/asha.12682874.

The temporal dynamics of first and second language processing: ERPs to spoken words in Mandarin-English bilinguals

The dynamics of bilingual spoken word recognition remain poorly characterized, especially for individuals who speak two languages that are highly dissimilar in their phonological and morphological structure. The present study compared first language (L1) and second language (L2) spoken word processing within a group of adult Mandarin-English bilinguals (N = 34; ages 18-25). Event-related potentials (ERPs) were recorded while participants completed the same cross-modal matching task separately in their L1 Mandarin and L2 English. This task consisted of deciding whether spoken words matched pictures of items. Pictures and spoken words either matched (e.g., Mandarin: TANG2-tang2; English: BELL-bell), or differed in word-initial phonemes (e.g., Mandarin: TANG2-lang2; English: BELL-shell), word-final phonemes (e.g., Mandarin: TANG2-tao2; English: BELL-bed), or whole words (e.g., Mandarin: TANG2-xia1: English: BELL-ham). Each mismatch type was associated with a pattern of modulation of the Phonological Mapping Negativity, the N400, and the Late N400 that was distinct from those of the other mismatch types yet similar between the two languages. This was interpreted as evidence of incremental processing with similar temporal dynamics in both languages. These findings support models of spoken word recognition in bilingual individuals that adopt an interactive-activation framework for both L1 and L2 processing.

Neurophysiological foundations of loss and failure sadness differently modulate emotional conceptual processing

Sadness is divided into two subtypes, namely loss and failure sadness, which are encoded by different concepts of one's mind. However, it is unclear how such a conceptual difference is supported by neurophysiological foundations. In the present study, we conducted an electroencephalogram experiment for processing congruency between loss- and failure-sadness contexts and emotional words. Electroencephalogram recordings were performed for 23 participants, using a picture-word priming paradigm without explicit congruency judgment. One of the three types of emotional pictures (loss, failure, or neutral picture as the baseline) preceded emotional target words with high, middle, or low fitting properties for sadness contexts in each trial. No significant word-onset event-related potential effects were observed. Upon word-offset event-related potential effects, middle-phase negative potentials around 400 ms for high-fitting words, increased in the failure prime-target context but not in the loss context, compared to the neutral context. Additionally, the negative potentials increased as the failure-sadness intensity decreased, which indicated contextual conflict between prime pictures and target words. In contrast, the corresponding negative potentials for the loss context increased as the loss-sadness intensity increased, which indicated congruency effects under sadness bias. In later latency, after around 400 ms, the slow negative event-related potential effects appeared similar for both the loss and failure contexts. These results suggest that loss and failure sadness are differently represented in the mind, and are founded on the middle-phase neurophysiological processing.

Code-Switching Does Not Equal Code-Switching. An Event-Related Potentials Study on Switching From L2 German to L1 Russian at Prepositions and Nouns

Studies on event-related potentials (ERP) in code-switching (CS) have concentrated on single-word insertions, usually nouns. However, CS ranges from inserting single words into the main language of discourse to alternating languages for larger segments of a discourse, and can occur at various syntactic positions and with various word classes. This ERP study examined native speakers of Russian who had learned German as a second language; they were asked to listen to sentences with CS from their second language, German, to their first language, Russian. CS included either a whole prepositional phrase or only the lexical head noun of a prepositional phrase. CS at nouns resulted in a late positive complex (LPC), whereas CS at prepositions resulted in a broad early negativity, which was followed by an anterior negativity with a posterior positivity. Only in the last time window (800–1000 ms) did CS at prepositions result in a broad positivity similar to CS at nouns. The differences between both types of CS indicate that they relate to different psycholinguistic processes.

Expectations about word stress modulate neural activity in speech-sensitive cortical areas

A recent dual-stream model of language processing proposed that the postero-dorsal stream performs predictive sequential processing of linguistic information via hierarchically organized internal models. However, it remains unexplored whether the prosodic segmentation of linguistic information involves predictive processes. Here, we addressed this question by investigating the processing of word stress, a major component of speech segmentation, using probabilistic repetition suppression (RS) modulation as a marker of predictive processing. In an event-related acoustic fMRI RS paradigm, we presented pairs of pseudowords having the same (Rep) or different (Alt) stress patterns, in blocks with varying Rep and Alt trial probabilities. We found that the BOLD signal was significantly lower for Rep than for Alt trials, indicating RS in the posterior and middle superior temporal gyrus (STG) bilaterally, and in the anterior STG in the left hemisphere. Importantly, the magnitude of RS was modulated by repetition probability in the posterior and middle STG. These results reveal the predictive processing of word stress in the STG areas and raise the possibility that words stress processing is related to the dorsal "where" auditory stream.

Phonological P2 or PMN During Spoken Word Recognition in Mandarin Chinese

Abstract. Word-initial phonological mismatches during spoken word recognition often elicit an event-related potential (ERP) component, namely, the phonological mapping negativity (PMN) in cross-modal priming studies or studies using sentence as context. However, recent studies also reported that a phonological P2 but not PMN has been observed for Mandarin Chinese spoken word recognition in unimodal word-matching and meaning-matching experiments, that is, both the prime and target words were presented auditorily. In the present study, the same pairs of disyllabic Mandarin Chinese words as in the prior unimodal studies were used as stimuli to investigate whether or not the phonological P2 effect is modulated by prime modality and can be replicated in a cross-modal design (i.e., written primes followed by spoken targets). Both the phonological and semantic relations between primes and targets were manipulated. Participants were instructed to judge whether the meaning of the two words were same or not. An enhanced PMN between 250 and 320 ms was elicited by word-initial phonological mismatches. In the later time window, centro-parietally distributed early N400 and late N400 were elicited in semantically unrelated conditions. The presence of PMN instead of P2 in the current study implies that ERP markers of word-initial phonological mismatches during spoken word recognition are modulated by the modality of primes at the level of phonological analysis.

Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (n = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain's electrophysiological index of semantic processing. A spatio-temporally fine-grained mixed-effect multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatio-temporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Speech predictability can hinder communication in difficult listening conditions

In difficult listening situations, such as in noisy environments, one would expect speech intelligibility to improve over time thanks to noise adaptation and/or to speech predictability facilitating the recognition of upcoming words. We tested this possibility by presenting normal-hearing human listeners (N = 100; 70 women) with sentences and measuring word recognition as a function of word position in a sentence. Sentences were presented in quiet and in competition with various masker sounds at individualized levels where listeners had 50% probability of recognizing a full sentence. Contrary to expectations, recognition was best for the first word and gradually deteriorated with increasing word position along the sentence. The worsening in recognition was unlikely due to differences in word audibility or word type and was uncorrelated with age or working memory capacity. Using a probabilistic model of word recognition, we show that the worsening effect probably occurs because misunderstandings generate inaccurate predictions that outweigh the benefits from accurate predictions. Analyses also revealed that predictions overruled the potential benefits from noise adaptation. We conclude that although speech predictability can facilitate sentence recognition, it can also result in declines in word recognition as the sentence unfolds because of inaccuracies in prediction.

The parallel mechanism of semantic context influences and parafoveal word identification

Previous event-related potential (ERP) studies have shown that the N400 effect tracks with the isolation point of a word in spoken language comprehension. These results suggested the semantic context of a sentence influences word processing before word identification. However, this delayed effect of N400 latency is not often the case in written language comprehension. The present study combined ERP with a rapid serial visual presentation with flanker (RSVP-flanker) method to examine this issue when words were presented in the parafovea. We manipulated three kinds of critical words: expected (EXP), semantically violated (VIO) and semantically cohort violated (COH-VIO). The ERP results showed that the parafoveal N400 effect was delayed in the contrast of COH-VIO vs. EXP than the contrast of VIO vs. EXP. These results suggested that the semantic context influences parafoveal word processing before word identification in written language comprehension.

A Developmental Perspective on Processing Semantic Context: Preliminary Evidence from Sentential Auditory Word Repetition in School-Aged Children

The current investigation examined the developmental changes involved in processing semantic context in auditorily presented sentences, as well as underlying attentional and suppression mechanisms. Thirty-nine typically developing school-aged children aged 6;0-14;0 years participated in the current cross-sectional sentential auditory word repetition study. Component processes involved in auditory word recognition were examined and their respective developmental trajectories systematically delineated. Experimental manipulations included semantic congruity (congruous, incongruous), sentence constraint (high, low), cloze probability (high, low), and processing mode. High sentence constraints elicited top-down pre-potency type effects, which resulted in active suppression of anticipated cloze words and longer naming latencies of perceived cloze words when violated with conflicting bottom-up information. In addition, developmental changes in component processes reflected underlying changes in attention, with evidence that suppression mechanisms remained relatively constant with age. Findings are interpreted in line with the Trace (McClelland and Elman in Cogn Psychol 18(1):1-86, 1986) model of auditory word recognition.

Oscillatory dynamics of cortical functional connections in semantic prediction

An event related potential, known as the N400, has been particularly useful in investigating language processing as it serves as a neural index for semantic prediction. There are numerous studies on the functional segregation of N400 neural sources; however, the oscillatory dynamics of functional connections among the relevant sources has remained elusive. In this study we acquired magnetoencephalography data during a classic N400 paradigm, where the semantic predictability of a fixed target noun was manipulated in simple German sentences. We conducted inter-regional functional connectivity (FC) and time-frequency analysis on known regions of the semantic network, encompassing bilateral temporal, and prefrontal cortices. Increased FC was found in less predicted (LP) nouns compared with highly predicted (HP) nouns in three connections: (a) right inferior frontal gyrus (IFG) and right middle temporal gyrus (MTG) from 0 to 300 ms mainly within the alpha band, (b) left lateral orbitofrontal (LOF) and right IFG around 400 ms within the beta band, and (c) left superior temporal gyrus (STG) and left LOF from 300 to 700 ms in the beta and low gamma bands. Furthermore, gamma spectral power (31-70 Hz) was stronger in HP nouns than in LP nouns in left anterior temporal cortices in earlier time windows (0-200 ms). Our findings support recent theories in language comprehension, suggesting fronto-temporal top-down connections are mainly mediated through beta oscillations while gamma band frequencies are involved in matching between prediction and input.

The Role of Context in Processing Chinese Three-Character Verb-Object Metaphors: An Event-Related Potential Study

Whether and how context plays a role in metaphor processing remains a controversial issue. One major theory on metaphor comprehension, the graded salience hypothesis (GSH) model, emphasizes salience as the key factor determining the precedence of semantic access. Using event-related potential technique, the present study examined Chinese metaphors to investigate whether the salient meaning is always processed first regardless of context. The experiment employed a Prime-Target-Probe paradigm. Three-character Chinese verb-object metaphors were used as the Target proceeded by one of the three contexts (the Prime): (1) metaphorical context priming the Target's metaphorical meaning, (2) literal context priming the Target's literal meaning, and (3) irrelevant context as the control condition. The Target was then followed by the Probe, which was always related to the Target (except in the filler condition). Forty participants were asked to judge whether the Target and the following Probe were semantically related. The N400 elicited by the Target showed no contextual effect. The N400 amplitude elicited by the Probe was smaller in the metaphorical priming condition compared with the literal priming condition, while the N400 in the irrelevant control condition was between the other two conditions, demonstrating a clear context effect. In addition, an unexpected P240 component also showed the similar graded pattern. Our results mostly support the GSH model, indicating that the salient meaning invariantly gets activated first before the activation of the nonsalient meaning at the lexical access stage. However, context does play a role in a parallel way either facilitating or suppressing this interpretation in the latter meaning integration stage.

Large-scale replication study reveals a limit on probabilistic prediction in language comprehension

Do people routinely pre-activate the meaning and even the phonological form of upcoming words? The most acclaimed evidence for phonological prediction comes from a 2005 Nature Neuroscience publication by DeLong, Urbach and Kutas, who observed a graded modulation of electrical brain potentials (N400) to nouns and preceding articles by the probability that people use a word to continue the sentence fragment (‘cloze’). In our direct replication study spanning 9 laboratories (N=334), pre-registered replication-analyses and exploratory Bayes factor analyses successfully replicated the noun-results but, crucially, not the article-results. Pre-registered single-trial analyses also yielded a statistically significant effect for the nouns but not the articles. Exploratory Bayesian single-trial analyses showed that the article-effect may be non-zero but is likely far smaller than originally reported and too small to observe without very large sample sizes. Our results do not support the view that readers routinely pre-activate the phonological form of predictable words.

Mixed metaphors: Electrophysiological brain responses to (un)expected concrete and abstract prepositional phrases

Languages around the world use spatial terminology, like prepositions, to describe non-spatial, abstract concepts, including time (e.g., in the moment). The Metaphoric Mapping Theory explains this pattern by positing that a universal human cognitive process underlies it, whereby abstract concepts are conceptualized via the application of concrete, three-dimensional space onto abstract domains. The alternative view is that the use of spatial propositions in abstract phrases is idiomatic, and thus does not trigger metaphoric mapping. In the current study, event-related potentials (ERPs) were used to examine the time-course of neural processing of concrete and abstract phrases consisting of the prepositions in or on followed by congruent and incongruent nouns (e.g., in the bowl/plate and in the moment/mend). ERPs were recorded from the onset of reference nouns in 28 adult participants using a 128-channel electrode net. Results show that congruency has differential effects on neural measures, depending on whether the noun is concrete or abstract. Incongruent reference nouns in concrete phrases (e.g., on the bowl) elicited a significant central negativity (an N400 effect), while incongruent reference nouns in abstract phrases (e.g., on the moment) did not. These results suggest that spatially incongruent concrete nouns are semantically unexpected (N400 effect). A P600 effect, which might indicate rechecking, reanalysis and/or reconstruction, was predicted for incongruent abstract nouns, but was not observed, possibly due to the variability in abstract stimuli. Findings cast doubt on accounts claiming that abstract uses of prepositions are cognitively and metaphorically linked to their spatial sense during natural, on-line processing.

The costs of being certain: Brain potential evidence for linguistic preactivation in sentence processing

Prediction in sentence comprehension is often investigated by measuring the amplitude of the N400 ERP component to words that are more or less predictable from their preceding context. The N400-linked to the activation of word-associated semantic information-is reduced for words that are predictable, indicating that preactivation can lead to facilitated processing. We addressed the question whether there is measurable neural activity related to the preactivation of linguistic information before input confirms or disconfirms this prediction. We therefore measured ERPs not only to moderately to highly predictable target words, but also to preceding adverbs. Based on two separate cloze pretests, we quantified the impact of the adverb upon the predictability of the subsequent target word. Using linear mixed-effects analyses, we could show that the N400 amplitude at the target word was inversely related to target cloze value, thus replicating the finding that prediction has a facilitative effect on semantic processing. Crucially, the N400 amplitude at the pretarget adverb was modulated by adverb impact: When adverbs increased the predictability of the following word, the N400 was more negative going. We argue that this effect is related to the preactivation of linguistic information. Our findings indicate that the specification of predictions can lead to additional processes before these predictions are confirmed or disconfirmed and that activation of word-associated information through prediction is highly comparable to activation through actual input.

Prediction Signatures in the Brain: Semantic Pre-Activation during Language Comprehension

There is broad agreement that context-based predictions facilitate lexical-semantic processing. A robust index of semantic prediction during language comprehension is an evoked response, known as the N400, whose amplitude is modulated as a function of semantic context. However, the underlying neural mechanisms that utilize relations of the prior context and the embedded word within it are largely unknown. We measured magnetoencephalography (MEG) data while participants were listening to simple German sentences in which the verbs were either highly predictive for the occurrence of a particular noun (i.e., provided context) or not. The identical set of nouns was presented in both conditions. Hence, differences for the evoked responses of the nouns can only be due to differences in the earlier context. We observed a reduction of the N400 response for highly predicted nouns. Interestingly, the opposite pattern was observed for the preceding verbs: highly predictive (that is more informative) verbs yielded stronger neural magnitude compared to less predictive verbs. A negative correlation between the N400 effect of the verb and that of the noun was found in a distributed brain network, indicating an integral relation between the predictive power of the verb and the processing of the subsequent noun. This network consisted of left hemispheric superior and middle temporal areas and a subcortical area; the parahippocampus. Enhanced activity for highly predictive relative to less predictive verbs, likely reflects establishing semantic features associated with the expected nouns, that is a pre-activation of the expected nouns.

Atypical audiovisual word processing in school-age children with a history of specific language impairment: an event-related potential study

Background

Visual speech cues influence different aspects of language acquisition. However, whether developmental language disorders may be associated with atypical processing of visual speech is unknown. In this study, we used behavioral and ERP measures to determine whether children with a history of SLI (H-SLI) differ from their age-matched typically developing (TD) peers in the ability to match auditory words with corresponding silent visual articulations.

Methods

Nineteen 7–13-year-old H-SLI children and 19 age-matched TD children participated in the study. Children first heard a word and then saw a speaker silently articulating a word. In half of trials, the articulated word matched the auditory word (congruent trials), while in another half, it did not (incongruent trials). Children specified whether the auditory and the articulated words matched. We examined ERPs elicited by the onset of visual stimuli (visual P1, N1, and P2) as well as ERPs elicited by the articulatory movements themselves—namely, N400 to incongruent articulations and late positive complex (LPC) to congruent articulations. We also examined whether ERP measures of visual speech processing could predict (1) children’s linguistic skills and (2) the use of visual speech cues when listening to speech-in-noise (SIN).

Results

H-SLI children were less accurate in matching auditory words with visual articulations. They had a significantly reduced P1 to the talker’s face and a smaller N400 to incongruent articulations. In contrast, congruent articulations elicited LPCs of similar amplitude in both groups of children. The P1 and N400 amplitude was significantly correlated with accuracy enhancement on the SIN task when seeing the talker’s face.

Conclusions

H-SLI children have poorly defined correspondences between speech sounds and visually observed articulatory movements that produce them.

Task modulation of disyllabic spoken word recognition in Mandarin Chinese: a unimodal ERP study

Using unimodal auditory tasks of word-matching and meaning-matching, this study investigated how the phonological and semantic processes in Chinese disyllabic spoken word recognition are modulated by top-down mechanism induced by experimental tasks. Both semantic similarity and word-initial phonological similarity between the primes and targets were manipulated. Results showed that at early stage of recognition (~150-250 ms), an enhanced P2 was elicited by the word-initial phonological mismatch in both tasks. In ~300-500 ms, a fronto-central negative component was elicited by word-initial phonological similarities in the word-matching task, while a parietal negativity was elicited by semantically unrelated primes in the meaning-matching task, indicating that both the semantic and phonological processes can be involved in this time window, depending on the task requirements. In the late stage (~500-700 ms), a centro-parietal Late N400 was elicited in both tasks, but with a larger effect in the meaning-matching task than in the word-matching task. This finding suggests that the semantic representation of the spoken words can be activated automatically in the late stage of recognition, even when semantic processing is not required. However, the magnitude of the semantic activation is modulated by task requirements.

Cross-modal symbolic processing can elicit either an N2 or a protracted N2/N400 response

A cross-modal symbolic paradigm was used to elicit EEG activity related to semantic incongruence. Twenty-five undergraduate students viewed pairings of visual lexical cues (e.g., DOG) with congruent (50% of trials) or incongruent (50%) auditory nonlexical stimuli (animal vocalizations; e.g., sound of a dog woofing or a cat meowing). In one condition, many different pairs of congruent/incongruent stimuli were shown, whereas in a second condition only two pairs of stimuli were repeatedly shown. A typical N400-like pattern of incongruence-related activity (including activity in the N2 time window) was evident in the condition using many stimuli, whereas the incongruence-related activity in the two-stimuli condition was confined to differential N2-like activity. A supplementary analysis excluded stimulus characteristics as the source of this differential activity between conditions. We found that a single individual performing a fixed task can demonstrate either a protracted N400-like pattern of activity or a more temporally focused N2-like pattern of activity in response to the same stimulus, which suggests that the N2 may be a precursor to the protracted N400 response.

Did you or I say pretty, rude or brief? An ERP study of the effects of speaker's identity on emotional word processing

During speech comprehension, multiple cues need to be integrated at a millisecond speed, including semantic information, as well as voice identity and affect cues. A processing advantage has been demonstrated for self-related stimuli when compared with non-self stimuli, and for emotional relative to neutral stimuli. However, very few studies investigated self-other speech discrimination and, in particular, how emotional valence and voice identity interactively modulate speech processing. In the present study we probed how the processing of words' semantic valence is modulated by speaker's identity (self vs. non-self voice). Sixteen healthy subjects listened to 420 prerecorded adjectives differing in voice identity (self vs. non-self) and semantic valence (neutral, positive and negative), while electroencephalographic data were recorded. Participants were instructed to decide whether the speech they heard was their own (self-speech condition), someone else's (non-self speech), or if they were unsure. The ERP results demonstrated interactive effects of speaker's identity and emotional valence on both early (N1, P2) and late (Late Positive Potential - LPP) processing stages: compared with non-self speech, self-speech with neutral valence elicited more negative N1 amplitude, self-speech with positive valence elicited more positive P2 amplitude, and self-speech with both positive and negative valence elicited more positive LPP. ERP differences between self and non-self speech occurred in spite of similar accuracy in the recognition of both types of stimuli. Together, these findings suggest that emotion and speaker's identity interact during speech processing, in line with observations of partially dependent processing of speech and speaker information.