Does Discourse Congruence Influence Spoken Language Comprehension before Lexical Association? Evidence from Event-Related Potentials

The Interaction of Context Constraints and Predictive Validity during Sentence Reading

Words are not processed in isolation; instead, they are commonly embedded in phrases and sentences. The sentential context influences the perception and processing of a word. However, how this is achieved by brain processes and whether predictive mechanisms underlie this process remain a debated topic. Here, we employed an experimental paradigm in which we orthogonalized sentence context constraints and predictive validity, which was defined as the ratio of congruent to incongruent sentence endings within the experiment. While recording electroencephalography, participants read sentences with three levels of sentential context constraints (high, medium, and low). Participants were also separated into two groups that differed in their ratio of valid congruent to incongruent target words that could be predicted from the sentential context. For both groups, we investigated modulations of alpha power before, and N400 amplitude modulations after target word onset. The results reveal that the N400 amplitude gradually decreased with higher context constraints and cloze probability. In contrast, alpha power was not significantly affected by context constraint. Neither the N400 nor alpha power were significantly affected by changes in predictive validity.

Cortical networks for recognition of speech with simultaneous talkers

The relative contributions of superior temporal vs. inferior frontal and parietal networks to recognition of speech in a background of competing speech remain unclear, although the contributions themselves are well established. Here, we use fMRI with spectrotemporal modulation transfer function (ST-MTF) modeling to examine the speech information represented in temporal vs. frontoparietal networks for two speech recognition tasks with and without a competing talker. Specifically, 31 listeners completed two versions of a three-alternative forced choice competing speech task: "Unison" and "Competing", in which a female (target) and a male (competing) talker uttered identical or different phrases, respectively. Spectrotemporal modulation filtering (i.e., acoustic distortion) was applied to the two-talker mixtures and ST-MTF models were generated to predict brain activation from differences in spectrotemporal-modulation distortion on each trial. Three cortical networks were identified based on differential patterns of ST-MTF predictions and the resultant ST-MTF weights across conditions (Unison, Competing): a bilateral superior temporal (S-T) network, a frontoparietal (F-P) network, and a network distributed across cortical midline regions and the angular gyrus (M-AG). The S-T network and the M-AG network responded primarily to spectrotemporal cues associated with speech intelligibility, regardless of condition, but the S-T network responded to a greater range of temporal modulations suggesting a more acoustically driven response. The F-P network responded to the absence of intelligibility-related cues in both conditions, but also to the absence (presence) of target-talker (competing-talker) vocal pitch in the Competing condition, suggesting a generalized response to signal degradation. Task performance was best predicted by activation in the S-T and F-P networks, but in opposite directions (S-T: more activation = better performance; F-P: vice versa). Moreover, S-T network predictions were entirely ST-MTF mediated while F-P network predictions were ST-MTF mediated only in the Unison condition, suggesting an influence from non-acoustic sources (e.g., informational masking) in the Competing condition. Activation in the M-AG network was weakly positively correlated with performance and this relation was entirely superseded by those in the S-T and F-P networks. Regarding contributions to speech recognition, we conclude: (a) superior temporal regions play a bottom-up, perceptual role that is not qualitatively dependent on the presence of competing speech; (b) frontoparietal regions play a top-down role that is modulated by competing speech and scales with listening effort; and (c) performance ultimately relies on dynamic interactions between these networks, with ancillary contributions from networks not involved in speech processing per se (e.g., the M-AG network).

Cognitive control mediates age-related changes in flexible anticipatory processing during listening comprehension

Effective listening comprehension not only requires processing local linguistic input, but also necessitates incorporating contextual cues available in the global communicative environment. Local sentence processing can be facilitated by pre-activation of likely upcoming input, or predictive processing. Recent evidence suggests that young adults can flexibly adapt local predictive processes based on cues provided by the global communicative environment, such as the reliability of specific speakers. Whether older comprehenders can also flexibly adapt to global contextual cues is currently unknown. Moreover, it is unclear whether the underlying mechanisms supporting local predictive processing differ from those supporting adaptation to global contextual cues. Critically, it is unclear whether these mechanisms change as a function of typical aging. We examined the flexibility of prediction in young and older adults by presenting sentences from speakers whose utterances were typically more or less predictable (i.e., reliable speakers who produced expected words 80% of the time, versus unreliable speakers who produced expected words 20% of the time). For young listeners, global speaker reliability cues modulated neural effects of local predictability on the N400. In contrast, older adults, on average, did not show global modulation of local processing. Importantly, however, cognitive control (i.e., Stroop interference effects) mediated age-related reductions in sensitivity to the reliability of the speaker. Both young and older adults with high cognitive control showed greater N400 effects of predictability during sentences produced by a reliable speaker, suggesting that cognitive control is required to regulate the strength of top-down predictions based on global contextual information. Critically, cognitive control predicted sensitivity to global speaker-specific information but not local predictability cues, suggesting that predictive processing in local sentence contexts may be supported by separable neural mechanisms from adaptation of prediction as a function of global context. These results have important implications for interpreting age-related change in predictive processing, and for drawing more generalized conclusions regarding domain-general versus language-specific accounts of prediction.

Retrieval (N400) and integration (P600) in expectation-based comprehension

Expectation-based theories of language processing, such as Surprisal theory, are supported by evidence of anticipation effects in both behavioural and neurophysiological measures. Online measures of language processing, however, are known to be influenced by factors such as lexical association that are distinct from—but often confounded with—expectancy. An open question therefore is whether a specific locus of expectancy related effects can be established in neural and behavioral processing correlates. We address this question in an event-related potential experiment and a self-paced reading experiment that independently cross expectancy and lexical association in a context manipulation design. We find that event-related potentials reveal that the N400 is sensitive to both expectancy and lexical association, while the P600 is modulated only by expectancy. Reading times, in turn, reveal effects of both association and expectancy in the first spillover region, followed by effects of expectancy alone in the second spillover region. These findings are consistent with the Retrieval-Integration account of language comprehension, according to which lexical retrieval (N400) is facilitated for words that are both expected and associated, whereas integration difficulty (P600) will be greater for unexpected words alone. Further, an exploratory analysis suggests that the P600 is not merely sensitive to expectancy violations, but rather, that there is a continuous relation. Taken together, these results suggest that the P600, like reading times, may reflect a meaning-centric notion of Surprisal in language comprehension.

Design and evaluation of the effectiveness of a corpus of congruent and incongruent English sentences for the study of event related potentials

OBJECTIVE

To design and evaluate the effectiveness of a stimulus material in eliciting the N400 event related potential (ERP).

DESIGN

A set of 700 semantically congruent and incongruent sentences was developed in accordance with current linguistic norms, and validated with an electroencephalography (EEG) study, in which the influence of age and gender on the N400 ERP magnitude was analysed.

STUDY SAMPLE

Forty-five normal-hearing subjects (19-57 years, 21 females) participated in the EEG study.

RESULTS

The stimulus material used in the EEG study elicited a robust N400 ERP, with a morphology consistent with the literature. Results also showed no statistically significant effect of age or gender on the N400 magnitude.

CONCLUSIONS

The material presented in this paper constitutes the largest complete stimulus set suitable for both auditory and text-based N400 experiments. This material may help facilitate the efficient implementation of future N400 ERP studies, as well as promote standardisation and consistency across studies.

How working memory capacity modulates the time course of semantic integration at sentence and discourse level

During comprehension, language users not only immediately integrate information from local sentence context, but also information from global discourse context for full understanding. In the present study, we examined whether the time course of these integration processes is constrained by language users' working memory capacity. Sentence and discourse stimuli were constructed. For the sentence stimuli, each sentence contained a critical word that was either congruent or incongruent with its preceding sentence context. For the discourse stimuli, each discourse contained four sentences with a target word embedded at the final sentence and the target word was either congruent or incongruent with the information provided at the first sentence of the discourse. Participants of high and low working memory span were instructed to read for comprehension. Our results showed that while the high span readers showed the N400 and P600 effects to semantically incongruent words, the low span readers only showed the P600 effect. This pattern was found regardless of whether the incongruent words were placed at sentence or discourse context. These results suggest that the low span readers are relatively delayed than the high span ones at both sentence- and discourse-level semantic integration and indicate that working memory functions have greater influence than context scope on the time course of semantic integration.

Adaptation to Animacy Violations during Listening Comprehension

The goal of this study was to examine adaptation to various types of animacy violations in cartoon-like stories. We measured the event-related potentials (ERPs) elicited by words at the beginning, middle, and end of four-sentence stories in order to examine adaptation over time to conflicts between stored word knowledge and context-derived meaning (specifically, to inanimate objects serving as main characters, as they might in a cartoon). The fourth and final sentence of each story contained a predicate that required either an animate or an inanimate subject. The results showed that listeners quickly adapted to stories in the Inanimate Noun conditions, consistent with previous research (Filik & Leuthold, 2008; Nieuwland & Van Berkum, 2006). They showed evidence of processing difficulty for animacy-requiring predicates in the Inanimate Noun conditions in Sentence 1, but the effect dissipated in Sentences 2 and 3. In Sentences 2 and 4, we measured ERPs at three critical points where it was possible to observe the influence of both context-based expectations and expectations from prior knowledge on processing. Overall, the pattern of results demonstrates how listeners flexibly adapt to unusual, conflict-ridden input, using previous context to generate expectations about upcoming input, but that current context is weighted appropriately in combination with expectations from background knowledge and prior language experience.

Cognitive control ability mediates prediction costs in monolinguals and bilinguals

In this study, we examined the role that cognitive control and language regulation ability play in mediating readers' susceptibility to prediction error costs when reading in the native language (L1) or a second language (L2). Twenty-four English monolinguals (Experiment 1) and 28 Chinese-English bilinguals (Experiment 2) read sentences in English while their EEG was recorded. The sentences varied in the predictability of an upcoming expected word and in whether that prediction was confirmed. Monolinguals showed sensitivity to sentence contexts in which expectations were not met (i.e., when unexpected words were encountered) in the form of a late, frontally-distributed positivity, but for bilinguals this effect was more complex. For both groups, performance on the prediction task was modulated by individual differences on the AX-CPT, a measure of inhibitory control. However, the bilinguals' reading performance in the L2 was affected not only by inhibitory control, but also by their performance on an L1 verbal fluency task that indexed language regulation and production capability, related to their language dominance and immersion context. Bilinguals with better regulation of the L1 generated a larger frontal positivity in response to unexpected words in the L2, an effect that was attenuated by inhibitory control ability. In contrast, bilinguals with lower regulatory ability generated a larger, late negativity, which was also mediated by control. These findings suggest that the ability to regulate the native language when immersed in a second language environment can influence mechanisms underlying the prediction process when reading in the L2. In addition, cognitive control ability, specifically inhibitory control, appears to mediate the difficulty readers incur when predictions are disconfirmed, not only in the native language, but also for proficient bilinguals reading in the L2. We argue that the mechanisms engaged during prediction in the L1 and L2 are fundamentally the same, and that what differs for bilinguals are the additional demands imposed by their language experience and language use.

Abnormal neural hierarchy in processing of verbal information in patients with schizophrenia

Previous research indicates abnormal comprehension of verbal information in patients with schizophrenia. Yet the neural mechanism underlying the breakdown of verbal information processing in schizophrenia is poorly understood. Imaging studies in healthy populations have shown a network of brain areas involved in hierarchical processing of verbal information over time. Here, we identified critical aspects of this hierarchy, examining patients with schizophrenia. Using functional magnetic resonance imaging, we examined various levels of information comprehension elicited by naturally presented verbal stimuli; from a set of randomly shuffled words to an intact story. Specifically, patients with first episode schizophrenia (N = 15), their non-manifesting siblings (N = 14) and healthy controls (N = 15) listened to a narrated story and randomly scrambled versions of it. To quantify the degree of dissimilarity between the groups, we adopted an inter-subject correlation (inter-SC) approach, which estimates differences in synchronization of neural responses within and between groups. The temporal topography found in healthy and siblings groups were consistent with our previous findings - high synchronization in responses from early sensory toward high order perceptual and cognitive areas. In patients with schizophrenia, stimuli with short and intermediate temporal scales evoked a typical pattern of reliable responses, whereas story condition (long temporal scale) revealed robust and widespread disruption of the inter-SCs. In addition, the more similar the neural activity of patients with schizophrenia was to the average response in the healthy group, the less severe the positive symptoms of the patients. Our findings suggest that system-level neural indication of abnormal verbal information processing in schizophrenia reflects disease manifestations.

The N400 in processing repeated name and pronoun anaphors in sentences and discourse

We report two EEG/ERP experiments that examined processing of repeated name (e.g., Bill; Experiment 1) and pronoun (e.g., he; Experiment 2) subject anaphors to single antecedents (e.g., Bill) and to antecedents embedded in a conjunction (e.g., Bill and Mary) within sentences and discourses. Experiment 1 replicated previous reports of repeated references to single antecedents eliciting greater N400 negativity than repeated references to conjoined antecedents within sentences, and extended these results to cross-sentence (discourse) references. Experiment 2 found that pronouns also elicited greater N400 negativity following single than conjoined antecedents. In both experiments, references to conjoined antecedents elicited greater frontal negativity than references to single antecedents in both sentences and discourse. Our results indicate that, in processing subject anaphors, the N400 is an index of reference predictability rather than a marker of the fit between antecedent salience and reference form, and that frontal negativity marks referential ambiguity elicited by conjoined phrases.

Language context processing deficits in schizophrenia: The role of attentional engagement

Individuals with schizophrenia exhibit problems in language comprehension that are most evident during discourse processing. We hypothesized that deficits in cognitive control contribute to these comprehension deficits during discourse processing, and investigated the underlying cognitive-neural mechanisms using EEG (alpha power) and ERPs (N400). N400 amplitudes to globally supported or unsupported target words near the end of stories were used to index sensitivity to previous context. ERPs showed reduced sensitivity to context in patients versus controls. EEG alpha-band activity was used to index attentional engagement while participants listened to the stories. We found that context effects varied with attentional engagement in both groups, as well as with negative symptom severity in patients. Both groups demonstrated trial-to-trial fluctuations in alpha. Relatively high alpha power was associated with compromised discourse processing in participants with schizophrenia when it occurred during any early portion of the story. In contrast, discourse processing was only compromised in controls when alpha was relatively high for longer segments of the stories. Our results indicate that shifts in attention from the story context may be more detrimental to discourse processing for participants with schizophrenia than for controls, most likely due to an impaired ability to benefit from global context.

Graded expectations: Predictive processing and the adjustment of expectations during spoken language comprehension

The goal of this study was to investigate the use of the local and global contexts for incoming words during listening comprehension. Local context was manipulated by presenting a target noun (e.g., "cake," "veggies") that was preceded by a word that described a prototypical or atypical feature of the noun (e.g., "sweet," "healthy"). Global context was manipulated by presenting the noun in a scenario that was consistent or inconsistent with the critical noun (e.g., a birthday party). Event-related potentials (ERPs) were examined at the feature word and at the critical noun. An N400 effect was found at the feature word, reflecting the effect of compatibility with the global context. Global predictability and the local feature word consistency interacted at the critical noun: A larger N200 was found to nouns that mismatched predictions when the context was maximally constraining, relative to nouns in the other conditions. A graded N400 response was observed at the critical noun, modulated by global predictability and feature consistency. Finally, post-N400 positivity effects of context updating were observed to nouns that were supported by one contextual cue (global/local) but were unsupported by the other. These results indicate that (1) incoming words that are compatible with context-based expectations receive a processing benefit; (2) when the context is sufficiently constraining, specific lexical items may be activated; and (3) listeners dynamically adjust their expectations when input is inconsistent with their predictions, provided that the inconsistency has some level of support from either the global or the local context.

It takes time to prime: semantic priming in the ocular lexical decision task

Two eye-tracking experiments were conducted in which the manual response mode typically used in lexical decision tasks (LDTs) was replaced with an eye-movement response through a sequence of 3 words. This ocular LDT combines the explicit control of task goals found in LDTs with the highly practiced ocular response used in reading text. In Experiment 1, forward saccades indicated an affirmative lexical decision (LD) on each word in the triplet. In Experiment 2, LD responses were delayed until all 3 letter strings had been read. The goal of the study was to evaluate the contribution of task goals and response mode to semantic priming. Semantic priming is very robust in tasks that involve recognition of words in isolation, such as LDT, but limited during text reading, as measured using eye movements. Gaze durations in both experiments showed robust semantic priming even though ocular response times were much shorter than manual LDs for the same words in the English Lexicon Project. Ex-Gaussian distribution fits revealed that the priming effect was concentrated in estimates of tau (τ), meaning that priming was most pronounced in the slow tail of the distribution. This pattern shows differential use of the prime information, which may be more heavily recruited in cases in which the LD is difficult, as indicated by longer response times. Compared with the manual LD responses, ocular LDs provide a more sensitive measure of this task-related influence on word recognition as measured by the LDT.

Spared and impaired spoken discourse processing in schizophrenia: effects of local and global language context

Individuals with schizophrenia are impaired in a broad range of cognitive functions, including impairments in the controlled maintenance of context-relevant information. In this study, we used ERPs in human subjects to examine whether impairments in the controlled maintenance of spoken discourse context in schizophrenia lead to overreliance on local associations among the meanings of individual words. Healthy controls (n = 22) and patients (n = 22) listened to short stories in which we manipulated global discourse congruence and local priming. The target word in the last sentence of each story was globally congruent or incongruent and locally associated or unassociated. ERP local association effects did not significantly differ between control participants and schizophrenia patients. However, in contrast to controls, patients only showed effects of discourse congruence when targets were primed by a word in the local context. When patients had to use discourse context in the absence of local priming, they showed impaired brain responses to the target. Our findings indicate that schizophrenia patients are impaired during discourse comprehension when demands on controlled maintenance of context are high. We further found that ERP measures of increased reliance on local priming predicted reduced social functioning, suggesting that alterations in the neural mechanisms underlying discourse comprehension have functional consequences in the illness.

Effects of working memory span on processing of lexical associations and congruence in spoken discourse

The goal of this study was to determine whether variability in working memory (WM) capacity and cognitive control affects the processing of global discourse congruence and local associations among words when participants listened to short discourse passages. The final, critical word of each passage was either associated or unassociated with a preceding prime word (e.g., “He was not prepared for the fame and fortune/praise”). These critical words were also either congruent or incongruent with respect to the preceding discourse context [e.g., a context in which a prestigious prize was won (congruent) or in which the protagonist had been arrested (incongruent)]. We used multiple regression to assess the unique contribution of suppression ability (our measure of cognitive control) and WM capacity on the amplitude of individual N400 effects of congruence and association. Our measure of suppression ability did not predict the size of the N400 effects of association or congruence. However, as expected, the results showed that high WM capacity individuals were less sensitive to the presence of lexical associations (showed smaller N400 association effects). Furthermore, differences in WM capacity were related to differences in the topographic distribution of the N400 effects of discourse congruence. The topographic differences in the global congruence effects indicate differences in the underlying neural generators of the N400 effects, as a function of WM. This suggests additional, or at a minimum, distinct, processing on the part of higher capacity individuals when tasked with integrating incoming words into the developing discourse representation.

Dissociating N400 effects of prediction from association in single-word contexts

When a word is preceded by a supportive context such as a semantically associated word or a strongly constraining sentence frame, the N400 component of the ERP is reduced in amplitude. An ongoing debate is the degree to which this reduction reflects a passive spread of activation across long-term semantic memory representations as opposed to specific predictions about upcoming input. We addressed this question by embedding semantically associated prime-target pairs within an experimental context that encouraged prediction to a greater or lesser degree. The proportion of related items was used to manipulate the predictive validity of the prime for the target while holding semantic association constant. A semantic category probe detection task was used to encourage semantic processing and to preclude the need for a motor response on the trials of interest. A larger N400 reduction to associated targets was observed in the high than the low relatedness proportion condition, consistent with the hypothesis that predictions about upcoming stimuli make a substantial contribution to the N400 effect. We also observed an earlier priming effect (205-240 msec) in the high-proportion condition, which may reflect facilitation because of form-based prediction. In summary, the results suggest that predictability modulates N400 amplitude to a greater degree than the semantic content of the context.

Cognitive control influences the use of meaning relations during spoken sentence comprehension

The aim of this study was to investigate individual differences in the influence of lexical association on word recognition during auditory sentence processing. Lexical associations among individual words (e.g. salt and pepper) represent one type of semantic information that is available during the processing of words in context. We predicted that individuals would vary in their sensitivity to this type of local context as a function of suppression ability and working-memory capacity. Lexical association was manipulated in auditory sentence contexts, and multiple regression analyses were employed to examine the relation between individuals' brain responses to meaning relations in sentences and measures of working-memory capacity, cognitive control and vocabulary. Lexical association influenced the processing of words that were embedded in sentences and also showed a great deal of individual variability. Specifically, suppression ability emerged as a significant predictor of sensitivity to lexical association, such that individuals who performed poorly on our measure of suppression ability (the Stroop task), compared to those who performed well, showed larger N400 effects of lexical association.