The fate of the unexpected: Consequences of misprediction assessed using ERP repetition effects

Prediction in reading: A review of predictability effects, their theoretical implications, and beyond

Historically, prediction during reading has been considered an inefficient and cognitively expensive processing mechanism given the inherently generative nature of language, which allows upcoming text to unfold in an infinite number of possible ways. This article provides an accessible and comprehensive review of the psycholinguistic research that, over the past 40 or so years, has investigated whether readers are capable of generating predictions during reading, typically via experiments on the effects of predictability (i.e., how well a word can be predicted from its prior context). Five theoretically important issues are addressed: What is the best measure of predictability? What is the functional relationship between predictability and processing difficulty? What stage(s) of processing does predictability affect? Are predictability effects ubiquitous? What processes do predictability effects actually reflect? Insights from computational models of reading about how predictability manifests itself to facilitate the reading of text are also discussed. This review concludes by arguing that effects of predictability can, to a certain extent, be taken as demonstrating evidence that prediction is an important but flexible component of real-time language comprehension, in line with broader predictive accounts of cognitive functioning. However, converging evidence, especially from concurrent eye-tracking and brain-imaging methods, is necessary to refine theories of prediction.

Looking for immediate and downstream evidence of lexical prediction in eye movements during reading

Previous investigations of whether readers make predictions about the full identity of upcoming words have focused on the extent to which there are processing consequences when readers encounter linguistic input that is incompatible with their expectations. To date, eye-movement studies have revealed inconsistent evidence of the processing costs that would be expected to accompany lexical prediction. This study investigated whether readers' lexical predictions were observable during or downstream from their initial point of activation. Three experiments assessed readers' eye movements to predictable and unpredictable words, and then to subsequent downstream words, which probed the lingering activation of previously expected words. The results showed novel evidence of processing costs for unexpected input but only when supported by a plausible linguistic environment, suggesting that readers could strategically modulate their predictive processing. However, there was limited evidence that their lexical predictions affected downstream processing. The implications of these findings for understanding the role of prediction in language processing are discussed.

The Costs (and Benefits?) of Effortful Listening for Older Adults: Insights from Simultaneous Electrophysiology, Pupillometry, and Memory

Although the impact of acoustic challenge on speech processing and memory increases as a person ages, older adults may engage in strategies that help them compensate for these demands. In the current preregistered study, older adults (n = 48) listened to sentences-presented in quiet or in noise-that were high constraint with either expected or unexpected endings or were low constraint with unexpected endings. Pupillometry and EEG were simultaneously recorded, and subsequent sentence recognition and word recall were measured. Like young adults in prior work, we found that noise led to increases in pupil size, delayed and reduced ERP responses, and decreased recall for unexpected words. However, in contrast to prior work in young adults where a larger pupillary response predicted a recovery of the N400 at the cost of poorer memory performance in noise, older adults did not show an associated recovery of the N400 despite decreased memory performance. Instead, we found that in quiet, increases in pupil size were associated with delays in N400 onset latencies and increased recognition memory performance. In conclusion, we found that transient variation in pupil-linked arousal predicted trade-offs between real-time lexical processing and memory that emerged at lower levels of task demand in aging. Moreover, with increased acoustic challenge, older adults still exhibited costs associated with transient increases in arousal without the corresponding benefits.

Error-based Implicit Learning in Language: The Effect of Sentence Context and Constraint in a Repetition Paradigm

Prediction errors drive implicit learning in language, but the specific mechanisms underlying these effects remain debated. This issue was addressed in an EEG study manipulating the context of a repeated unpredictable word (repetition of the complete sentence or repetition of the word in a new sentence context) and sentence constraint. For the manipulation of sentence constraint, unexpected words were presented either in high-constraint (eliciting a precise prediction) or low-constraint sentences (not eliciting any specific prediction). Repetition-induced reduction of N400 amplitudes and of power in the alpha/beta frequency band was larger for words repeated with their sentence context as compared with words repeated in a new low-constraint context, suggesting that implicit learning happens not only at the level of individual items but additionally improves sentence-based predictions. These processing benefits for repeated sentences did not differ between constraint conditions, suggesting that sentence-based prediction update might be proportional to the amount of unpredicted semantic information, rather than to the precision of the prediction that was violated. In addition, the consequences of high-constraint prediction violations, as reflected in a frontal positivity and increased theta band power, were reduced with repetition. Overall, our findings suggest a powerful and specific adaptation mechanism that allows the language system to quickly adapt its predictions when unexpected semantic information is processed, irrespective of sentence constraint, and to reduce potential costs of strong predictions that were violated.

The Impact of Linguistic Prediction Violations on Downstream Recognition Memory and Sentence Recall

Predicting upcoming words during language comprehension not only affects processing in the moment but also has consequences for memory, although the source of these memory effects (e.g., whether driven by lingering pre-activations, re-analysis following prediction violations, or other mechanisms) remains underspecified. Here, we investigated downstream impacts of prediction on memory in two experiments. First, we recorded EEG as participants read strongly and weakly constraining sentences with expected, unexpected but plausible, or semantically anomalous endings ("He made a holster for his gun / father / train") and were tested on their recognition memory for the sentence endings. Participants showed similar rates of false alarms for predicted but never presented sentence endings whether the prediction violation was plausible or anomalous, suggesting that these arise from pre-activation of the expected words during reading. During sentence reading, especially in strongly constraining sentences, plausible prediction violations elicited an anterior positivity; anomalous endings instead elicited a posterior positivity, whose amplitude was predictive of later memory for those anomalous words. ERP patterns at the time of recognition differentiated plausible and anomalous sentence endings: Words that had been plausible prediction violations elicited enhanced late positive complex amplitudes, suggesting greater episodic recollection, whereas anomalous sentence endings elicited greater N1 amplitudes, suggesting attentional tagging. In a follow-up behavioral study, a separate group of participants read the same sentence stimuli and were tested for sentence-level recall. We found that recall of full sentences was impaired when sentences ended with a prediction violation. Taken together, the results suggest that prediction violations draw attention and affect encoding of the violating word, in a manner that depends on plausibility, and that this, in turn, may impair future memory of the gist of the sentence.

Graded and ungraded expectation patterns: Prediction dynamics during active comprehension

Language comprehension can be facilitated by the accurate prediction of upcoming words, but prediction effects are not ubiquitous, and comprehenders likely use predictive processing to varying degrees depending on task demands. To ascertain the processing consequences of prioritizing prediction, we here compared ERPs elicited when young adult participants simply read for comprehension with those collected in a subsequent block that required active prediction. We were particularly interested in frontally-distributed post-N400 effects for expected and unexpected words in strongly constraining contexts, which have previously been documented as two distinct patterns: an enhanced positivity ("anterior positivity") observed for prediction violations compared to words that are merely unpredictable (because they occur in weakly constraining sentences) and a distinction between expected endings in more constraining contexts and those same weakly constrained words ("frontal negativity" to the strongly predicted words). We found that the size of the anterior positivity effect was unchanged between passive comprehension and active prediction, suggesting that some processes related to prediction may engage state-like networks. On the other hand, the frontal negativity showed graded patterns from the interaction of task and sentence type. These differing patterns support the hypothesis that there are two separate effects with frontal scalp distributions that occur after the N400 and further suggest that the impact of violating predictions (as long as prediction is engaged at all) is largely stable across varying levels of effort/attention directed toward prediction, whereas other comprehension processes can be modulated by task demands.

Context-specific effects of violated expectations: ERP evidence

A complete understanding of the predictive processing effect in sentence comprehension needs to understand both the facilitation effect of successful prediction and the cost associated with disconfirmed predictions. The current study compares the predictive processing effect across two types of contexts in Mandarin Chinese: the classifier-noun vs. verb-noun phrases, when controlling for the degree of contextual constraints and cloze probability of the target nouns across the two contexts. The two contexts showed similar N400 patterns for expected target nouns, indicative of an identical facilitation effect of confirmed contextual expectation. But in the post-N400 time window, the processing cost associated with the unexpected words differed between the two contexts. Additional differences between the two contexts were also revealed by the neural oscillation patterns obtained prior to the target noun. The differences between the classifier vs. verb contexts shed new light on the revision mechanism that deals with disconfirmed expectations.

Multiple predictions during language comprehension: Friends, foes, or indifferent companions?

To comprehend language, we continually use prior context to pre-activate expected upcoming information, resulting in facilitated processing of incoming words that confirm these predictions. But what are the consequences of disconfirming prior predictions? To address this question, most previous studies have examined unpredictable words appearing in contexts that constrain strongly for a single continuation. However, during natural language processing, it is far more common to encounter contexts that constrain for multiple potential continuations, each with some probability. Here, we ask whether and how pre-activating both higher and lower probability alternatives influences the processing of the lower probability incoming word. One possibility is that, similar to language production, there is continuous pressure to select the higher-probability pre-activated alternative through competitive inhibition. During comprehension, this would result in relative costs in processing the lower probability target. A second possibility is that if the two pre-activated alternatives share semantic features, they mutually enhance each other's pre-activation. This would result in greater facilitation in processing the lower probability target. To distinguish between these accounts, we recorded ERPs as participants read three-sentence scenarios that constrained either for a single word or for two potential continuations - a higher probability expected candidate and a lower probability second-best candidate. We found no evidence that competitive pre-activation between the expected and second-best candidates resulted in costs in processing the second-best target, either during lexico-semantic processing (indexed by the N400) or at later stages of processing (indexed by a later frontal positivity). Instead, we found only benefits of pre-activating multiple alternatives, with evidence of enhanced graded facilitation on lower-probability targets that were semantically related to a higher-probability pre-activated alternative. These findings are consistent with a previous eye-tracking study by Luke and Christianson (2016, Cogn Psychol) using corpus-based materials. They have significant theoretical implications for models of predictive language processing, indicating that routine graded prediction in language comprehension does not operate through the same competitive mechanisms that are engaged in language production. Instead, our results align more closely with hierarchical probabilistic accounts of language comprehension, such as predictive coding.

Prediction during language comprehension: what is next?

Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it. We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input. The nature of these predictions may vary between individuals owing to differences in language experience, among other factors. We then turn to unresolved questions which may guide the search for the underlying mechanisms. (i) Is prediction essential to language processing or an optional strategy? (ii) Are predictions generated from within the language system or by domain-general processes? (iii) What is the relationship between prediction and memory? (iv) Does prediction in comprehension require simulation via the production system? We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.

Understanding the Effects of Constraint and Predictability in ERP

Intuitively, strongly constraining contexts should lead to stronger probabilistic representations of sentences in memory. Encountering unexpected words could therefore be expected to trigger costlier shifts in these representations than expected words. However, psycholinguistic measures commonly used to study probabilistic processing, such as the N400 event-related potential (ERP) component, are sensitive to word predictability but not to contextual constraint. Some research suggests that constraint-related processing cost may be measurable via an ERP positivity following the N400, known as the anterior post-N400 positivity (PNP). The PNP is argued to reflect update of a sentence representation and to be distinct from the posterior P600, which reflects conflict detection and reanalysis. However, constraint-related PNP findings are inconsistent. We sought to conceptually replicate Federmeier et al. (2007) and Kuperberg et al. (2020), who observed that the PNP, but not the N400 or the P600, was affected by constraint at unexpected but plausible words. Using a pre-registered design and statistical approach maximising power, we demonstrated a dissociated effect of predictability and constraint: strong evidence for predictability but not constraint in the N400 window, and strong evidence for constraint but not predictability in the later window. However, the constraint effect was consistent with a P600 and not a PNP, suggesting increased conflict between a strong representation and unexpected input rather than greater update of the representation. We conclude that either a simple strong/weak constraint design is not always sufficient to elicit the PNP, or that previous PNP constraint findings could be an artifact of smaller sample size.

Parafoveal preview benefit in a conflicting sentential context: Evidence from ERPs

In natural reading, the reader's processing of a word starts when the word is located in parafoveal vision. Relative to a situation with an invalid preview, fixations are significantly shorter after a fully valid, identical preview. Although research on the identity preview benefit has been ongoing for more than 40 years, the mechanism of this preview benefit, the level of lexical processing it occurs, and its relationship to the sentential context remain unclear. This study employed EEG brain component analysis technology to address these questions in Chinese sentence reading. We manipulated the sentential context to make the target word plausible or not plausible with the sentence and manipulated the target word present or not present in preview. EEG results showed that the identity preview benefit can affect not only the early preview positivity, reflecting the early orthographic processing of words, but also the N400 and LPC components, reflecting the late and in-depth semantic processing of words. Conflicting sentential context, in which the target word is implausible and cannot be integrated into the sentence, can interfere temporarily with these processes. These findings suggest that in the process of sentence reading, an identical preview word can promote the subsequent reading process at multiple levels, and its role is modulated by contextual information.

Benefits and costs of predictive processing: How sentential constraint and word expectedness affect memory formation

This study investigated how the strength of schema support provided by strongly (SC) and weakly constraining (WC) sentences affects the encoding of expected and unexpected words, and how this is reflected in event-related potentials (ERPs). In a surprise recognition memory test, words studied on the previous day were presented together with new words and lures that were expected but not presented in the study phase. ERPs recorded in the study phase were compared for subsequently remembered and forgotten words. Better memory performance for expected over unexpected words was electrophysiologically supported by a parietal subsequent memory effect (SME) reflecting enhanced item-specific encoding of contextually expected words. SC sentences not only facilitated the semantic integration of sentence-ending words, as reflected in reduced N400 amplitudes, but also enabled the rapid successful encoding of these words into memory, which is evidenced by an SC > WC pattern in memory performance and correlations between pre- and post-stimulus SMEs for SC sentences. In contrast, words processed in WC sentence contexts necessitated sustained elaborative encoding processes as reflected in a late frontal slow wave SME. Expected but not presented words were associated with high rates of false positive memory decisions, indicating that these words remained in a state of high accessibility in memory even one day after the study phase. These mnemonic costs of predictive processing were more pronounced for expected words from SC sentences than from WC sentences and could reflect the lingering of strong semantic predictions which were associated with the pre-updating of sentence representations.

Connecting and considering: Electrophysiology provides insights into comprehension

The ability to rapidly and systematically access knowledge stored in long-term memory in response to incoming sensory information-that is, to derive meaning from the world-lies at the core of human cognition. Research using methods that can precisely track brain activity over time has begun to reveal the multiple cognitive and neural mechanisms that make this possible. In this article, I delineate how a process of connecting affords an effortless, continuous infusion of meaning into human perception. In a relatively invariant time window, uncovered through studies using the N400 component of the event-related potential, incoming sensory information naturally induces a graded landscape of activation across long-term semantic memory, creating what might be called "proto-concepts". Connecting can be (but is not always) followed by a process of further considering those activations, wherein a set of more attentionally demanding "active comprehension" mechanisms mediate the selection, augmentation, and transformation of the initial semantic representations. The result is a limited set of more stable bindings that can be arranged in time or space, revised as needed, and brought to awareness. With this research, we are coming closer to understanding how the human brain is able to fluidly link sensation to experience, to appreciate language sequences and event structures, and, sometimes, to even predict what might be coming up next.

The N400 ERP component reflects an error-based implicit learning signal during language comprehension

The functional significance of the N400 evoked-response component is still actively debated. An increasing amount of theoretical and computational modelling work is built on the interpretation of the N400 as a prediction error. In neural network modelling work, it was proposed that the N400 component can be interpreted as the change in a probabilistic representation of meaning that drives the continuous adaptation of an internal model of the statistics of the environment. These results imply that increased N400 amplitudes should correspond to greater adaptation, which can be measured via implicit memory. To investigate this model derived hypothesis, the current study manipulated expectancy in a sentence reading task to influence N400 amplitudes and subsequently presented the previously expected vs. unexpected words in a perceptual identification task to measure implicit memory. As predicted, reaction times in the perceptual identification task were significantly faster for previously unexpected words that induced larger N400 amplitudes in the previous sentence reading task. Additionally, it could be demonstrated that this adaptation seems to specifically depend on the process underlying N400 amplitudes, as participants with larger N400 differences during sentence reading also exhibited a larger implicit memory benefit in the perceptual identification task. These findings support the interpretation of the N400 as an implicit learning signal driving adaptation in language processing.