Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation

Sorry, you make less sense to me: The effect of non-native speaker status on metaphor processing

Preconceived assumptions about the speaker have been shown to strongly and automatically influence speech interpretation. This study contributes to previous research by investigating the impact of non-nativeness on perceived metaphor sensibility. To eliminate the effects of speech disfluency, we used exclusively written sentences but introduced their "authors" as having a strong native or non-native accent through a written vignette. The author's language proficiency was never mentioned. Metaphorical sentences featured familiar ("The pictures streamed through her head") and unfamiliar ("The textbooks snored on the desk") verbal metaphors and closely matched literal expressions from a pre-tested database. We also administered a battery of psychological tests to assess whether ratings could be predicted by individual differences. The results revealed that all sentences attributed to the non-native speaker were perceived as less sensical. Incorporating the identity of the non-native speaker also took more effort, as indicated by longer processing and evaluation times. Additionally, while a general bias against non-native speakers emerged even without oral speech, person-based factors played a significant role. Lower ratings of non-native compared to native speakers were largely driven by individuals from less linguistically diverse backgrounds and those with less cognitive reflection. Extraversion and political ideology also modulated ratings in a unique way. The study highlights the impact of preconceived notions about the speaker on sentence processing and the importance of taking interpersonal variation into account.

Register and morphosyntactic congruence during sentence processing in German: An eye-tracking study

In the present study, we used eye-tracking to investigate formality-register and morphosyntactic congruence during sentence reading. While research frequently covers participants' processing of lexical, (morpho-)syntactic, or semantic knowledge (e.g., operationalized by means of violations to which we can measure responses relative to felicitous stimuli), less attention has been devoted to the full breadth of pragmatic and context-related aspects. One such aspect is sensitivity to formality-register congruence, i.e., the match or mismatch between the register of a target word and the formality conveyed by the (linguistic) context. In particular, we investigated how congruence of linguistic register with context formality, as well as its interplay with morphosyntactic knowledge, may unfold during reading and be reflected in eye movements. In our study, 40 native German speakers read context sentences conveying a formal or informal situation, and a target sentence containing a high- or low-register verb (e.g., Engl. transl. The policeman detained the activist vs. The policeman nabbed the activist) which matched or mismatched the formality of the preceding context sentences. We additionally manipulated subject-verb agreement, with either a match (see examples above) or a mismatch thereof (e.g., Engl. transl. *The policeman detain the activist; *The policeman nab the activist). We predicted that a violation of formality-register congruence would be reflected in longer reading times at the verb and post-verbal object region, as this would be in line with previous research on context violations (e.g., Lüdtke & Kaup, 2006; Reali et al., 2015; Traxler & Pickering, 1996). We found effects of morphosyntactic congruence on late processing stages at the verb and on earlier processing stages at the post-verbal object region. As far as formality-register congruence is concerned, only late (in total reading time analysis, in the post-verbal object region) and subtle effects emerged. The results suggest that, compared to morphosyntactic violations, formality-register congruence effects emerge quite subtly and slowly during reading.

The Self-reference Effect Can Modulate Language Syntactic Processing Even Without Explicit Awareness: An Electroencephalography Study

Although it is well established that self-related information can rapidly capture our attention and bias cognitive functioning, whether this self-bias can affect language processing remains largely unknown. In addition, there is an ongoing debate as to the functional independence of language processes, notably regarding the syntactic domain. Hence, this study investigated the influence of self-related content on syntactic speech processing. Participants listened to sentences that could contain morphosyntactic anomalies while the masked face identity (self, friend, or unknown faces) was presented for 16 msec preceding the critical word. The language-related ERP components (left anterior negativity [LAN] and P600) appeared for all identity conditions. However, the largest LAN effect followed by a reduced P600 effect was observed for self-faces, whereas a larger LAN with no reduction of the P600 was found for friend faces compared with unknown faces. These data suggest that both early and late syntactic processes can be modulated by self-related content. In addition, alpha power was more suppressed over the left inferior frontal gyrus only when self-faces appeared before the critical word. This may reflect higher semantic demands concomitant to early syntactic operations (around 150-550 msec). Our data also provide further evidence of self-specific response, as reflected by the N250 component. Collectively, our results suggest that identity-related information is rapidly decoded from facial stimuli and may impact core linguistic processes, supporting an interactive view of syntactic processing. This study provides evidence that the self-reference effect can be extended to syntactic processing.

Situating language register across the ages, languages, modalities, and cultural aspects: Evidence from complementary methods

In the present review paper by members of the collaborative research center "Register: Language Users' Knowledge of Situational-Functional Variation" (CRC 1412), we assess the pervasiveness of register phenomena across different time periods, languages, modalities, and cultures. We define "register" as recurring variation in language use depending on the function of language and on the social situation. Informed by rich data, we aim to better understand and model the knowledge involved in situation- and function-based use of language register. In order to achieve this goal, we are using complementary methods and measures. In the review, we start by clarifying the concept of "register", by reviewing the state of the art, and by setting out our methods and modeling goals. Against this background, we discuss three key challenges, two at the methodological level and one at the theoretical level: (1) To better uncover registers in text and spoken corpora, we propose changes to established analytical approaches. (2) To tease apart between-subject variability from the linguistic variability at issue (intra-individual situation-based register variability), we use within-subject designs and the modeling of individuals' social, language, and educational background. (3) We highlight a gap in cognitive modeling, viz. modeling the mental representations of register (processing), and present our first attempts at filling this gap. We argue that the targeted use of multiple complementary methods and measures supports investigating the pervasiveness of register phenomena and yields comprehensive insights into the cross-methodological robustness of register-related language variability. These comprehensive insights in turn provide a solid foundation for associated cognitive modeling.

Causal Contributions of the Domain-General (Multiple Demand) and the Language-Selective Brain Networks to Perceptual and Semantic Challenges in Speech Comprehension

Listening to spoken language engages domain-general multiple demand (MD; frontoparietal) regions of the human brain, in addition to domain-selective (frontotemporal) language regions, particularly when comprehension is challenging. However, there is limited evidence that the MD network makes a functional contribution to core aspects of understanding language. In a behavioural study of volunteers (n = 19) with chronic brain lesions, but without aphasia, we assessed the causal role of these networks in perceiving, comprehending, and adapting to spoken sentences made more challenging by acoustic-degradation or lexico-semantic ambiguity. We measured perception of and adaptation to acoustically degraded (noise-vocoded) sentences with a word report task before and after training. Participants with greater damage to MD but not language regions required more vocoder channels to achieve 50% word report, indicating impaired perception. Perception improved following training, reflecting adaptation to acoustic degradation, but adaptation was unrelated to lesion location or extent. Comprehension of spoken sentences with semantically ambiguous words was measured with a sentence coherence judgement task. Accuracy was high and unaffected by lesion location or extent. Adaptation to semantic ambiguity was measured in a subsequent word association task, which showed that availability of lower-frequency meanings of ambiguous words increased following their comprehension (word-meaning priming). Word-meaning priming was reduced for participants with greater damage to language but not MD regions. Language and MD networks make dissociable contributions to challenging speech comprehension: Using recent experience to update word meaning preferences depends on language-selective regions, whereas the domain-general MD network plays a causal role in reporting words from degraded speech.

Integration of Social Context vs. Linguistic Reference During Situated Language Processing

Research findings on language comprehension suggest that many kinds of non-linguistic cues can rapidly affect language processing. Extant processing accounts of situated language comprehension model these rapid effects and are only beginning to accommodate the role of non-linguistic emotional, cues. To begin with a detailed characterization of distinct cues and their relative effects, three visual-world eye-tracking experiments assessed the relative importance of two cue types (action depictions vs. emotional facial expressions) as well as the effects of the degree of naturalness of social (facial) cues (smileys vs. natural faces). We predicted to replicate previously reported rapid effects of referentially mediated actions. In addition, we assessed distinct world-language relations. If how a cue is conveyed matters for its effect, then a verb referencing an action depiction should elicit a stronger immediate effect on visual attention and language comprehension than a speaker's emotional facial expression. The latter is mediated non-referentially via the emotional connotations of an adverb. The results replicated a pronounced facilitatory effect of action depiction (relative to no action depiction). By contrast, the facilitatory effect of a preceding speaker's emotional face was less pronounced. How the facial emotion was rendered mattered in that the emotional face effect was present with natural faces (Experiment 2) but not with smileys (Experiment 1). Experiment 3 suggests that contrast, i.e., strongly opposing emotional valence information vs. non-opposing valence information, might matter for the directionality of this effect. These results are the first step toward a more principled account of how distinct visual (social) cues modulate language processing, whereby the visual cues that are referenced by language (the depicted action), copresent (the depicted action), and more natural (the natural emotional prime face) tend to exert more pronounced effects.

Empathy influences how listeners interpret intonation and meaning when words are ambiguous

This study examines how individual pragmatic skills, and more specifically, empathy, influences language processing when a temporary lexical ambiguity can be resolved via intonation. We designed a visual-world eye-tracking experiment in which participants could anticipate a referent before disambiguating lexical information became available, by inferring either a contrast meaning or a confirmatory meaning from the intonation contour alone. Our results show that individual empathy skills determine how listeners deal with the meaning alternatives of an ambiguous referent, and the way they use intonational meaning to disambiguate the referent. Listeners with better pragmatic skills (higher empathy) were sensitive to intonation cues when forming sound-meaning associations during the unfolding of an ambiguous referent, and showed higher sensitivity to all the alternative interpretations of that ambiguous referent. Less pragmatically skilled listeners showed weaker processing of intonational meaning because they needed subsequent disambiguating material to select a referent and showed less sensitivity to the set of alternative interpretations. Overall, our results call for taking into account individual pragmatic differences in the study of intonational meaning processing and sentence comprehension in general.

The role of context in verbal humor processing in autism

Difficulties in processing humor have been associated with individuals with autism. The current study investigated whether humor comprehension and appreciation could be augmented in children with autism by providing contextual support suggesting that humor was to be expected. A verbally presented riddle task was used in which participants were assessed for their subjective ratings and comprehension of the materials. They were also filmed to record any smiling or laughing. Both riddles and control stimuli were presented with supporting verbal context and also without it. The results showed that (a) the greater subjective appreciation of riddles than of control stimuli was dependent on the provision of context for the participants with autism and that (b) context statistically equated these ratings of riddles between participants with autism and matched typically developing controls. However, context had no effect on comprehension or affective response. The results of the current study demonstrate that children with autism are, even in the most conservative interpretation, able to use verbal context to recognize verbal humor. This lays the foundation of possible interventions based on training sensitivity to context.

Situating language in a minimal social context: how seeing a picture of the speaker's face affects language comprehension

Natural use of language involves at least two individuals. Some studies have focused on the interaction between senders in communicative situations and how the knowledge about the speaker can bias language comprehension. However, the mere effect of a face as a social context on language processing remains unknown. In the present study, we used event-related potentials to investigate the semantic and morphosyntactic processing of speech in the presence of a photographic portrait of the speaker. In Experiment 1, we show that the N400, a component related to semantic comprehension, increased its amplitude when processed within this minimal social context compared to a scrambled face control condition. Hence, the semantic neural processing of speech is sensitive to the concomitant perception of a picture of the speaker's face, even if irrelevant to the content of the sentences. Moreover, a late posterior negativity effect was found to the presentation of the speaker's face compared to control stimuli. In contrast, in Experiment 2, we found that morphosyntactic processing, as reflected in left anterior negativity and P600 effects, is not notably affected by the presence of the speaker's portrait. Overall, the present findings suggest that the mere presence of the speaker's image seems to trigger a minimal communicative context, increasing processing resources for language comprehension at the semantic level.

Effects of Dispositional Affect on the N400: Language Processing and Socially Situated Context

We examined whether the N400 Event-Related Potential (ERP) component would be modulated by dispositional affect during sentence processing. In this study, 33 participants read sentences manipulated by direct object type (congruent vs. incongruent) and object determiner type (definite vs. demonstrative). We were particularly interested in sentences of the form: (i) The connoisseur tasted the wine on the tour vs. (ii) The connoisseur tasted the # roof … We expected that processing incongruent direct objects (#roof) vs. congruent objects (wine) would elicit N400 effects. Previous ERP language experiments have shown that participants in (induced) positive and negative moods were differentially sensitive to semantic anomaly, resulting in different N400 effects. Presently, we ask whether individual dispositional affect scores (as measured by the Positive and Negative Affect Schedule; PANAS) would modulate N400 effects as shown previously. Namely, previous results showed larger N400 effects associated with happy moods and attenuated amplitudes associated with sad moods. Results revealed significant N400 effects, driven by the #roof vs. the wine, where larger amplitude differences were found for individuals showing smaller negative affect (NA) scores, thus partially replicating previous findings. We discuss our results in terms of theories of local (lexical) inhibition, such that low NA promotes stronger lexico-semantic links in sentences. Finally, our results support accounts of language processing that include social and biological characteristics of individuals during real-time sentence comprehension.

Grounding Language Processing: The Added Value of Specifying Linguistic/Compositional Representations and Processes

Abundant empirical evidence suggests that visual perception and motor responses are involved in language comprehension ('grounding'). However, when modeling the grounding of sentence comprehension on a word-by-word basis, linguistic representations and cognitive processes are rarely made fully explicit. This article reviews representational formalisms and associated (computational) models with a view to accommodating incremental and compositional grounding effects. Are different representation formats equally suitable and what mechanisms and representations do models assume to accommodate grounding effects? I argue that we must minimally specify compositional semantic representations, a set of incremental processes/mechanisms, and an explicit link from the assumed processes to measured behavior. Different representational formats can be contrasted in psycholinguistic modeling by holding the set of processes/mechanisms constant; contrasting different processes/mechanisms is possible by holding representations constant. Such psycholinguistic modeling could be applied across a wide range of experimental investigations and complement computational modeling.

Referential vs. Non-referential World-Language Relations: How Do They Modulate Language Comprehension in 4 to 5-Year-Olds, Younger, and Older Adults?

Age has been shown to influence language comprehension, with delays, for instance, in older adults' expectations about upcoming information. We examined to what extent expectations about upcoming event information (who-does-what-to-whom) change across the lifespan (in 4- to 5-year-old children, younger, and older adults) and as a function of different world-language relations. In a visual-world paradigm, participants in all three age groups inspected a speaker whose facial expression was either smiling or sad. Next they inspected two clipart agents (e.g., a smiling cat and a grumpy rat) depicted as acting upon a patient (e.g., a ladybug tickled by the cat and arrested by the rat). Control scenes featured the same three characters without the action depictions. While inspecting the depictions, comprehenders listened to a German sentence [e.g., Den Marienkäfer kitzelt vergnügt der Kater; literally: “The ladybug (object/patient) tickles happily the cat (subject/agent)”]. Referential verb-action relations (i.e., when the actions were present) could, in principle, cue the cat-agent and so could non-referential relations via links from the speaker's smile to “happily” and the cat's smile. We examined variation in participants' visual anticipation of the agent (the cat) before it was mentioned depending on (a) participant age and (b) whether the referentially mediated action depiction or the non-referentially associated speaker smile cued the agent. The action depictions rapidly boosted participants' visual anticipation of the agent, facilitating thematic role assignment in all age groups. By contrast, effects of the non-referentially cued speaker smile emerged in the younger adults only. We outline implications of these findings for processing accounts of the temporally coordinated interplay between listeners' age-dependent language comprehension, their interrogation of the visual context, and visual context influences.

Explicit (Not Implicit) Attitudes Mediate the Focus of Attention During Sentence Processing

Many studies showed that comprehenders monitor changes in protagonists' emotions and actions. This article reports two experiments that explored how focusing comprehenders' attention on a particular property of the protagonist dimension (e.g., emotional or action state) affects the accessibility of information about target objects mentioned in the sentence. Furthermore, the present research examined whether participants' attitudes toward the issues described in the sentence can modulate comprehension processes. To this end, we asked participants to read sentences about environmental issues that focused comprehenders' attention on different mental and physical attributes of the same entities (protagonists and objects) and then self-report their own thoughts on the topic of environment by responding to the items assessing their environmental awareness. Importantly, we manipulated the task requirements across two experiments by administering a self-report task (Experiment 1), which required the participants to rate the seriousness and the frequency of the problem mentioned in a sentence; and administering a sentence-picture verification paradigm (Experiment 2), which required the participants to merely indicate if the object depicted in the picture (related to a certain environmental problem) was mentioned in the preceding sentence. The results of these experiments suggest that the focus of a sentence on the environmental problem (rather than the protagonist's emotion and action) enhances the accessibility of information about environmental issues (e.g., plastic garbage); that the comprehender's level of environmental awareness influences one's attention during sentence processing; and that comprehender characteristics significantly modulate comprehension processes only when the measures tap into explicit (and not implicit) processes.

Common Ground Information Affects Reference Resolution: Evidence From Behavioral Data, ERPs, and Eye-Tracking

One of the most important social cognitive skills in humans is the ability to “put oneself in someone else’s shoes,” that is, to take another person’s perspective. In socially situated communication, perspective taking enables the listener to arrive at a meaningful interpretation of what is said (sentence meaning) and what is meant (speaker’s meaning) by the speaker. To successfully decode the speaker’s meaning, the listener has to take into account which information he/she and the speaker share in their common ground (CG). We here further investigated competing accounts about when and how CG information affects language comprehension by means of reaction time (RT) measures, accuracy data, event-related potentials (ERPs), and eye-tracking. Early integration accounts would predict that CG information is considered immediately and would hence not expect to find costs of CG integration. Late integration accounts would predict a rather late and effortful integration of CG information during the parsing process that might be reflected in integration or updating costs. Other accounts predict the simultaneous integration of privileged ground (PG) and CG perspectives. We used a computerized version of the referential communication game with object triplets of different sizes presented visually in CG or PG. In critical trials (i.e., conflict trials), CG information had to be integrated while privileged information had to be suppressed. Listeners mastered the integration of CG (response accuracy 99.8%). Yet, slower RTs, and enhanced late positivities in the ERPs showed that CG integration had its costs. Moreover, eye-tracking data indicated an early anticipation of referents in CG but an inability to suppress looks to the privileged competitor, resulting in later and longer looks to targets in those trials, in which CG information had to be considered. Our data therefore support accounts that foresee an early anticipation of referents to be in CG but a rather late and effortful integration if conflicting information has to be processed. We show that both perspectives, PG and CG, contribute to socially situated language processing and discuss the data with reference to theoretical accounts and recent findings on the use of CG information for reference resolution.

How In-Group Bias Influences Source Memory for Words Learned From In-Group and Out-Group Speakers

Individuals rapidly extract information about others’ social identity, including whether or not they belong to their in-group. Group membership status has been shown to affect how attentively people encode information conveyed by those others. These findings are highly relevant for the field of psycholinguistics where there exists an open debate on how words are represented in the mental lexicon and how abstract or context-specific these representations are. Here, we used a novel word learning paradigm to test our proposal that the group membership status of speakers also affects how speaker-specific representations of novel words are. Participants learned new words from speakers who either attended their own university (in-group speakers) or did not (out-group speakers) and performed a task to measure their individual in-group bias. Then, their source memory of the new words was tested in a recognition test to probe the speaker-specific content of the novel lexical representations and assess how it related to individual in-group biases. We found that speaker group membership and participants’ in-group bias affected participants’ decision biases. The stronger the in-group bias, the more cautious participants were in their decisions. This was particularly applied to in-group related decisions. These findings indicate that social biases can influence recognition threshold. Taking a broader scope, defining how information is represented is a topic of great overlap between the fields of memory and psycholinguistics. Nevertheless, researchers from these fields tend to stay within the theoretical and methodological borders of their own field, missing the chance to deepen their understanding of phenomena that are of common interest. Here, we show how methodologies developed in the memory field can be implemented in language research to shed light on an important theoretical issue that relates to the composition of lexical representations.

Anticipating a future versus integrating a recent event? Evidence from eye-tracking

When comprehending a spoken sentence that refers to a visually-presented event, comprehenders both integrate their current interpretation of language with the recent event and develop expectations about future event possibilities. Tense cues can disambiguate this linking, but temporary ambiguity in these cues may lead comprehenders to also rely on further, experience-based (e.g., frequency or an actor's gaze) cues. How comprehenders reconcile these different cues in real time is an open issue. Extant results suggest that comprehenders preferentially relate their unfolding interpretation to a recent event by inspecting its target object. We investigated to what extent this recent-event preference could be overridden by short-term experiential and situation-specific cues. In Experiments 1-2 participants saw substantially more future than recent events and listened to more sentences about future-events (75% in Experiment 1 and 88% in Experiment 2). Experiment 3 cued future target objects and event possibilities via an actor's gaze. The event frequency increase yielded a reduction in the recent event inspection preference early during sentence processing in Experiments 1-2 compared with Experiment 3 (where event frequency and utterance tense were balanced) but did not eliminate the overall recent-event preference. Actor gaze also modulated the recent-event preference, and jointly with future tense led to its reversal in Experiment 3. However, our results showed that people overall preferred to focus on recent (vs. future) events in their interpretation, suggesting that while two cues (actor gaze and short-term event frequency) can partially override the recent-event preference, the latter still plays a key role in shaping participants' interpretation.