Finding referents in time: eye-tracking evidence for the role of contrastive accents

Comprehension and processing of the universal quantifier in children, adolescents and adults

Quantifier spreading (Q-spreading), children's incorrect falsification of a universally-quantified sentence based on an 'extra-object' picture, may persist beyond childhood, and children adhere to Q-spreading without changing responses throughout testing. We examined the error patterns across wider age groups (aged 4-79) with a picture-sentence verification eye-tracking task. We also examined whether prosodic emphasis affects their comprehension and processing of universally-quantified sentences. Whereas adults' comprehension was ceiling, children/adolescents (aged 4-17) showed various comprehension patterns, splitting into: 'Adult-like responders' (consistently adult-like), 'Q-spreaders' (consistently showing Q-spreading), and 'Switchers' (shifted from Q-spreading to adult-like). While adults rarely looked at the extra-object, 'Q-spreaders' showed frequent looks throughout testing, and both 'Switchers' and 'Adult-like responders' exhibited reduced looks to the extra-object, suggesting that avoidance and correction of Q-spreading requires inhibition of the visual attention to the extra-object. The effect of prosodic emphasis on eye movement emerged later for children/adolescents than adults.

Focus Effects on Immediate and Delayed Recognition of Referents in Samoan

This paper looks at the effect of focus-marking on the immediate and delayed recognition of referents in Samoan. Focus-marking on a word can imply the presence of alternatives to that word which are relevant to the interpretation of the utterance. Consistent with this, psycholinguistic evidence is growing that alternatives to focus-marked words are selectively activated in the immediate processing of an utterance and longer term memory. However, most of this research is on Western Germanic languages which primarily use prosodic prominence to mark focus. We explore this in two experiments using immediate and delayed probe recognition tasks in the under-studied language Samoan, which primarily uses syntactic focus-marking. Participants heard short narratives ending in a critical sentence in which the object word was either focused or not, using a cleft-like construction. In the first experiment, probe recognition, alternatives to the object word which were either mentioned or unmentioned in the narrative were responded to more slowly if the object was focus-marked. In the second experiment, delayed recognition, participants were faster to correctly recognize mentioned alternatives, and slower to reject unmentioned, if the object was focus-marked. Both results are consistent with immediate and longer-term activation of focus alternatives. There was no significant effect of focus-marking on recognition of the object word itself.

Role of pitch in toddler looking to new and given referents in American English

This study examined how toddler looking to a discourse referent is mediated by the information status of the referent and the pitch contour of the referring expression. Eighteen-month-olds saw a short discourse of three sets of images with the proportion of looking time to a target analyzed during the final image. At test, the information status of the referent was either new or given and the referring expression was presented with one of three pitch contours (flat f0, monotonal (~H*), or bitonal (~L+H*)). In Experiment 1, toddlers looked reliably longer to a target referent when it was either new to the discourse or carried a non-flat pitch contour. In Experiment 2, the referring expression was removed to observe effects of information status alone on looking to a target referent. Toddlers looked significantly longer to a target when it was new versus given. More fine-grained time course analyses of eye movements revealed differences in the speed and duration of fixation to a target. Overall, the experiments show that discourse reference in toddlers is mediated by the presence of newness and pitch contours, even in the case of given information.

In Search of Salience: Focus Detection in the Speech of Different Talkers

Many different prosodic cues can help listeners predict upcoming speech. However, no research to date has assessed listeners' processing of preceding prosody from different speakers. The present experiments examine (1) whether individual speakers (of the same language variety) are likely to vary in their production of preceding prosody; (2) to the extent that there is talker variability, whether listeners are flexible enough to use any prosodic cues signaled by the individual speaker; and (3) whether types of prosodic cues (e.g., F0 versus duration) vary in informativeness. Using a phoneme-detection task, we examined whether listeners can entrain to different combinations of preceding prosodic cues to predict where focus will fall in an utterance. We used unsynthesized sentences recorded by four female native speakers of Australian English who happened to have used different preceding cues to produce sentences with prosodic focus: a combination of pre-focus overall duration cues, F0 and intensity (mean, maximum, range), and longer pre-target interval before the focused word onset (Speaker 1), only mean F0 cues, mean and maximum intensity, and longer pre-target interval (Speaker 2), only pre-target interval duration (Speaker 3), and only pre-focus overall duration and maximum intensity (Speaker 4). Listeners could entrain to almost every speaker's cues (the exception being Speaker 4's use of only pre-focus overall duration and maximum intensity), and could use whatever cues were available even when one of the cue sources was rendered uninformative. Our findings demonstrate both speaker variability and listener flexibility in the processing of prosodic focus.

On the Impact of Case and Prosody on Thematic Role Disambiguation: An Eye-Tracking Study on Hungarian

Thematic-role assignment is influenced by several classes of cues during sentence comprehension, ranging from morphological exponents of syntactic relation such as case and agreement to probabilistic cues such as prosody. The effect of these cues cross-linguistically varies, presumably reflecting their language-specific robustness in signaling thematic roles. However, language-specific frequencies are not mapped onto the cue strength in a one-to-one fashion. The present article reports two eye-tracking studies on Hungarian examining the interaction of case and prosody during the processing of case-unambiguous (Experiment 1) and case-ambiguous (Experiment 2) clauses. Eye fixations reveal that case is a strong cue for thematic role assignment, but stress only enhances the effect of case in case-unambiguous clauses. This result differs from findings reported for Italian and German in which case initial stress reduces the expectation for subject-first clauses. Furthermore, the sentence comprehension facts are not explained by corpus frequencies in Hungarian. After considering an array of hypotheses about the roots of cross-linguistic variation, we conclude that the crucial difference lies in the high reliability/availability of case cues in Hungarian in contrast to the further languages examined within this experimental paradigm.

Cognitive constraints on advance planning of sentence intonation

Pitch peaks tend to be higher at the beginning of longer than shorter sentences (e.g., ‘A farmer is pulling donkeys’ vs ‘A farmer is pulling a donkey and goat’), whereas pitch valleys at the ends of sentences are rather constant for a given speaker. These data seem to imply that speakers avoid dropping their voice pitch too low by planning the height of sentence-initial pitch peaks prior to speaking. However, the length effect on sentence-initial pitch peaks appears to vary across different types of sentences, speakers and languages. Therefore, the notion that speakers plan sentence intonation in advance due to the limitations in low voice pitch leaves part of the data unexplained. Consequently, this study suggests a complementary cognitive account of length-dependent pitch scaling. In particular, it proposes that the sentence-initial pitch raise in long sentences is related to high demands on mental resources during the early stages of sentence planning. To tap into the cognitive underpinnings of planning sentence intonation, this study adopts the methodology of recording eye movements during a picture description task, as the eye movements are the established approximation of the real-time planning processes. Measures of voice pitch (Fundamental Frequency) and incrementality (eye movements) are used to examine the relationship between (verbal) working memory (WM), incrementality of sentence planning and the height of sentence-initial pitch peaks.

Thinking probabilistically in the study of intonational speech prosody

Speech prosody, the melodic and rhythmic properties of a language, plays a critical role in our everyday communication. Researchers have identified unique patterns of prosody that segment words and phrases, highlight focal elements in a sentence, and convey holistic meanings and speech acts that interact with the information shared in context. The mapping between the sound and meaning represented in prosody is suggested to be probabilistic-the same physical instance of sounds can support multiple meanings across talkers and contexts while the same meaning can be encoded in physically distinct sound patterns (e.g., pitch movements). The current overview presents an analysis framework for probing the nature of this probabilistic relationship. Illustrated by examples from the literature and a dataset of German focus marking, we discuss the production variability within and across talkers and consider challenges that this variability imposes on the comprehension system. A better understanding of these challenges, we argue, will illuminate how the human perceptual, cognitive, and computational mechanisms may navigate the variability to arrive at a coherent understanding of speech prosody. The current paper is intended to be an introduction for those who are interested in thinking probabilistically about the sound-meaning mapping in prosody. Open questions for future research are discussed with proposals for examining prosodic production and comprehension within a comprehensive, mathematically-motivated framework of probabilistic inference under uncertainty. This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Language.

Adaptation in Predictive Prosodic Processing in Bilinguals

Native language listeners engage in predictive processing in many processing situations and adapt their predictive processing to the statistics of the input. In contrast, second language listeners engage in predictive processing in fewer processing situations. The current study uses eye-tracking data from two experiments in bilinguals’ native language (L1) and second language (L2) to explore their predictive processing based on contrastive pitch accent cues, and their adaptation in the face of prediction errors. The results of the first experiment show inhibition effects for unpredicted referents in both the L1 and the L2 that can be modeled with a Bayesian adaptation model, suggesting that bilinguals adapt their prediction in the face of prediction errors in a way that is compatible with the model. In contrast, the results of the second experiment, after a training phase that increased the predictive validity of the cue, show inhibition effects for unpredicted referents only in the L1, but not in the L2. In addition, the Bayesian adaptation model significantly predicts only the L1, but not the L2 data. The results are discussed with respect to adaptation to the statistical properties of the input.

Processing of Prosody and Semantics in Sepedi and L2 English

A phoneme-detection task shows that listeners of Sepedi use semantic information in processing but not prosody (Experiment 1). Sepedi is a language with no grammaticalised prosodic expression of focus. Sepedi listeners detected phoneme targets faster when the phoneme-bearing words were focussed (as opposed to unfocussed) but not when occurring in a context conducive to prosodic emphasis (as opposed to non-conducive). Experiment 2 tested the role of semantic focus and prosody in processing by Sepedi L1/English L2 listeners (English being a language with systematic focus-to-accent mapping). Non-native listeners detected phoneme-bearing words faster in focussed condition (as opposed to unfocussed) and in accented condition (as opposed to deaccented). The results suggest that the L2 prosodic structure is exploited by Black South African English listeners even if this feature is not present in their L1. Our experiments replicate the pattern of results found in Akker and Cutler's (Biling Lang Cogn 6:81-96, 2003) experiment for Dutch L1/ English L2 listeners, even with listeners whose L1 does not use prosody the way English does.

Pitch accents create dissociable syntactic and semantic expectations during sentence processing

The language system uses syntactic, semantic, as well as prosodic cues to efficiently guide auditory sentence comprehension. Prosodic cues, such as pitch accents, can build expectations about upcoming sentence elements. This study investigates to what extent syntactic and semantic expectations generated by pitch accents can be dissociated and if so, which cues take precedence when contradictory information is present. We used sentences in which one out of two nominal constituents was placed in contrastive focus with a third one. All noun phrases carried overt syntactic information (case-marking of the determiner) and semantic information (typicality of the thematic role of the noun). Two experiments (a sentence comprehension and a sentence completion task) show that focus, marked by pitch accents, established expectations in both syntactic and semantic domains. However, only the syntactic expectations, when violated, were strong enough to interfere with sentence comprehension. Furthermore, when contradictory cues occurred in the same sentence, the local syntactic cue (case-marking) took precedence over the semantic cue (thematic role), and overwrote previous information cued by prosody. The findings indicate that during auditory sentence comprehension the processing system integrates different sources of information for argument role assignment, yet primarily relies on syntactic information.

Crossmodal Language Comprehension-Psycholinguistic Insights and Computational Approaches

Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches.

When (not) to Look for Contrastive Alternatives: The Role of Pitch Accent Type and Additive Particles

This study investigates how pitch accent type and additive particles affect the activation of contrastive alternatives. In Experiment 1, German listeners heard declarative utterances (e.g., The swimmer wanted to put on flippers) and saw four printed words displayed on screen: one that was a contrastive alternative to the subject noun (e.g., diver), one that was non-contrastively related (e.g., pool), the object (e.g., flippers), and an unrelated distractor. Experiment 1 manipulated pitch accent type, comparing a broad focus control condition to two narrow focus conditions: with a contrastive or non-contrastive accent on the subject noun (nuclear L+H* vs. H+L*, respectively, followed by deaccentuation). In Experiment 2, the utterances in the narrow focus conditions were preceded by the unstressed additive particle auch ("also"), which may trigger alternatives itself. It associated with the accented subject. Results showed that, compared to the control condition, participants directed more fixations to the contrastive alternative when the subject was realized with a contrastive accent (nuclear L+H*) than when it was realized with non-contrastive H+L*, while additive particles had no effect. Hence, accent type is the primary trigger for signaling the presence of alternatives (i.e., contrast). Implications for theories of information structure and the processing of additive particles are discussed.

Communicative Context Affects Use of Referential Prosody

The current study assessed the extent to which the use of referential prosody varies with communicative demand. Speaker-listener dyads completed a referential communication task during which speakers attempted to indicate one of two color swatches (one bright, one dark) to listeners. Speakers' bright sentences were reliably higher pitched than dark sentences for ambiguous (e.g., bright red versus dark red) but not unambiguous (e.g., bright red versus dark purple) trials, suggesting that speakers produced meaningful acoustic cues to brightness when the accompanying linguistic content was underspecified (e.g., "Can you get the red one?"). Listening partners reliably chose the correct corresponding swatch for ambiguous trials when lexical information was insufficient to identify the target, suggesting that listeners recruited prosody to resolve lexical ambiguity. Prosody can thus be conceptualized as a type of vocal gesture that can be recruited to resolve referential ambiguity when there is communicative demand to do so.

Priming Effects of Focus in Mandarin Chinese

Psycholinguistic research has long established that focus-marked words have a processing advantage over other words in an utterance, e.g., they are recognized more quickly and remembered better. More recently, studies have shown that listeners infer contextual alternatives to a focused word in a spoken utterance, when marked with a contrastive accent, even when the alternatives are not explicitly mentioned in the discourse. This has been shown by strengthened priming of contextual alternatives to the word, but not other non-contrastive semantic associates, when it is contrastively accented, e.g., after hearing "The customer opened the window," salesman is strongly primed, but not product. This is consistent with Rooth's (1992) theory that focus-marking signals the presence of alternatives to the focus. However, almost all of the research carried out in this area has been on Germanic languages. Further, most of this work has looked only at one kind of focus-marking, by contrastive accenting (prosody). This paper reports on a cross-modal lexical priming study in Mandarin Chinese, looking at whether focus-marking heightens activation, i.e., priming, of words and their alternatives. Two kinds of focus-marking were investigated: prosodic and syntactic. Prosodic prominence is an important means of focus-marking in Chinese, however, it is realized through pitch range expansion, rather than accenting. The results showed that focused words, as well as their alternatives, were primed when the subject prime word carried contrastive prosodic prominence. Syntactic focus-marking, however, did not enhance priming of focused words or their alternatives. Non-contrastive semantic associates were not primed with either kind of focus-marking. These results extend previous findings on focus and alternative priming for the first time to Chinese. They also suggest that the processing advantages of focus, including priming alternatives, are particularly related to prosodic prominence, at least in Chinese and Germanic languages. This research sheds light on what linguistic mechanisms listeners use to identify important information, generate alternatives, and understand implicature necessary for successful communication.

Prenuclear L∗+H Activates Alternatives for the Accented Word

Previous processing studies have shown that constituents that are prosodically marked as focus lead to an activation of alternatives. We investigate the processing of constituents that are prosodically marked as contrastive topics. In German, contrastive topics are prosodically realized by prenuclear L∗+H accents. Our study tests (a) whether prenuclear accents (as opposed to nuclear accents) are able to activate contrastive alternatives, (b) whether they do this in the same way as constituents prosodically marked as focus with nuclear accents do, which is important for semantic modeling, and (c) whether the activation of alternatives is caused by pitch accent type (prenuclear L∗+H as contrastive accent vs. prenuclear L+H∗ as non-contrastive accent) or by differences in F0-excursion (related to prominence). We conducted two visual-world eye-tracking studies, in which German listeners heard declarative utterances (e.g., The swimmer wanted to put on flappers) and watched displays that depicted four printed words: one that was a contrastive alternative to the subject noun (e.g., diver), one that was non-contrastively related to it (e.g., sports), the object (e.g., flappers), which had to be clicked, and an unrelated distractor. Experiment 1 presented participants with two naturally produced intonation conditions, a broad focus control condition with a prenuclear L+H∗ accent on the subject and a contrastive topic condition with a prenuclear L∗+H accent. The results showed that participants fixated more on the contrastive alternative when the subject was produced with an L∗+H accent, with the same effect size and timing as reported for focus constituents. Experiment 2 resynthesized the stimuli so that peak height and F0-excursion were the same across intonation conditions. The effect was the same, but the time course was slightly later. Our results suggest that prenuclear L∗+H immediately leads to the activation of alternatives during online processing, and that the F0-excursion of the accent lends little. The results are discussed with regard to the processing of contrastive focus accents and theories of contrastive topic.

The effects of prosody on sentence comprehension: evidence from a neurotypical control group and seven cases of chronic stroke

Both prosody and sentence structure (e.g., canonical versus non-canonical) affect sentence comprehension. However, few previous studies have examined a possible interaction between prosody and sentence structure. In adult controls we found a significant interaction: typical sentence prosody, versus list prosody, facilitated comprehension of only some sentence structures. In seven stroke patients, impaired attentional control was related to impaired comprehension with sentence prosody but not list prosody; impaired working memory was related to impaired comprehension with list prosody, but not sentence prosody. Thus, non-canonical sentence comprehension impairments in stroke patients may be modulated by prosody, based on a patient's cognitive abilities.

Effect of F0 contour on perception of Mandarin Chinese speech against masking

Intonation has many perceptually significant functions in language that contribute to speech recognition. This study aims to investigate whether intonation cues affect the unmasking of Mandarin Chinese speech in the presence of interfering sounds. Specifically, intelligibility of multi-tone Mandarin Chinese sentences with maskers consisting of either two-talker speech or steady-state noise was measured in three (flattened, typical, and exaggerated) intonation conditions. Different from most of the previous studies, the present study only manipulate and modify the intonation information but preserve tone information. The results showed that recognition of the final keywords in multi-tone Mandarin Chinese sentences was much better under the original F0 contour condition than the decreased F0 contour or exaggerated F0 contour conditions whenever there was a noise or speech masker, and an exaggerated F0 contour reduced the intelligibility of Mandarin Chinese more under the speech masker condition than that under the noise masker condition. These results suggested that speech in a tone language (Mandarin Chinese) is harder to understand when the intonation is unnatural, even if the tone information is preserved, and an unnatural intonation contour decreases releasing Mandarin Chinese speech from masking, especially in a multi-person talking environment.

Accents, Not Just Prosodic Boundaries, Influence Syntactic Attachment

Traditionally, pitch accents are understood to relate to the information structure of a sentence and its discourse connections, while prosodic boundaries indicate groupings of words and affect how constituents attach into a syntactic structure. Here, we show that accents also affect syntactic attachment in multiple different syntactic structures. Three auditory questionnaires on ambiguous attachment sentences (such as Tom reported that Bill was bribed [last May]) find that accenting the higher or lower verb ( reported or bribed) increases the attachment of the final adverbial phrase as a modifier of the accented verb. A fourth experiment shows that accents on verbs or object nouns (in sentences like Jenny sketched a child [with crayons]) also increase the attachment of the final prepositional phrase to the accented head (sketched with crayons versus a child with crayons). Accent effects were small but consistent across sentences with different levels of bias and did not depend on prosodic boundaries. The results suggest that focused elements are important to the main assertion of the sentence and therefore draw the attachment of upcoming material (though the salience of attachment sites may also be important). The results also demonstrate that both prosodic phrasing and pitch accents can affect basic syntactic structure.

The Temporal Prediction of Stress in Speech and Its Relation to Musical Beat Perception

While rhythmic expectancies are thought to be at the base of beat perception in music, the extent to which stress patterns in speech are similarly represented and predicted during on-line language comprehension is debated. The temporal prediction of stress may be advantageous to speech processing, as stress patterns aid segmentation and mark new information in utterances. However, while linguistic stress patterns may be organized into hierarchical metrical structures similarly to musical meter, they do not typically present the same degree of periodicity. We review the theoretical background for the idea that stress patterns are predicted and address the following questions: First, what is the evidence that listeners can predict the temporal location of stress based on preceding rhythm? If they can, is it thanks to neural entrainment mechanisms similar to those utilized for musical beat perception? And lastly, what linguistic factors other than rhythm may account for the prediction of stress in natural speech? We conclude that while expectancies based on the periodic presentation of stresses are at play in some of the current literature, other processes are likely to affect the prediction of stress in more naturalistic, less isochronous speech. Specifically, aspects of prosody other than amplitude changes (e.g., intonation) as well as lexical, syntactic and information structural constraints on the realization of stress may all contribute to the probabilistic expectation of stress in speech.

Prosody in the Auditory and Visual Domains: A Developmental Perspective

The development of body movements such as hand or head gestures, or facial expressions, seems to go hand-in-hand with the development of speech abilities. We know that very young infants rely on the movements of their caregivers' mouth to segment the speech stream, that infants' canonical babbling is temporally related to rhythmic hand movements, that narrative abilities emerge at a similar time in speech and gestures, and that children make use of both modalities to access complex pragmatic intentions. Prosody has emerged as a key linguistic component in this speech-gesture relationship, yet its exact role in the development of multimodal communication is still not well understood. For example, it is not clear what the relative weights of speech prosody and body gestures are in language acquisition, or whether both modalities develop at the same time or whether one modality needs to be in place for the other to emerge. The present paper reviews existing literature on the interactions between speech prosody and body movements from a developmental perspective in order to shed some light on these issues.

Recognition and Comprehension of "Narrow Focus" by Young Adults With Prelingual Hearing Loss Using Hearing Aids or Cochlear Implants

PURPOSE

The stressed word in a sentence (narrow focus [NF]) conveys information about the intent of the speaker and is therefore important for processing spoken language and in social interactions. The ability of participants with severe-to-profound prelingual hearing loss to comprehend NF has rarely been investigated. The purpose of this study was to assess the recognition and comprehension of NF by young adults with prelingual hearing loss compared with those of participants with normal hearing (NH).

METHOD

The participants included young adults with hearing aids (HA; n = 10), cochlear implants (CI; n = 12), and NH (n = 18). The test material included the Hebrew Narrow Focus Test (Segal, Kaplan, Patael, & Kishon-Rabin, in press), with 3 subtests, which was used to assess the recognition and comprehension of NF in different contexts.

RESULTS

The following results were obtained: (a) CI and HA users successfully recognized the stressed word, with the worst performance for CI; (b) HA and CI comprehended NF less well than NH; and (c) the comprehension of NF was associated with verbal working memory and expressive vocabulary in CI users.

CONCLUSIONS

Most CI and HA users were able to recognize the stressed word in a sentence but had considerable difficulty understanding it. Different factors may contribute to this difficulty, including the memory load during the task itself and linguistic and pragmatic abilities.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.5572792.

Using prosody to infer discourse prominence in cochlear-implant users and normal-hearing listeners

Cochlear implants (CIs) provide speech perception to adults with severe-to-profound hearing loss, but the acoustic signal remains severely degraded. Limited access to pitch cues is thought to decrease sensitivity to prosody in CI users, but co-occurring changes in intensity and duration may provide redundant cues. The current study investigates how listeners use these cues to infer discourse prominence. CI users and normal-hearing (NH) listeners were presented with sentences varying in prosody (accented vs. unaccented words) while their eye-movements were measured to referents varying in discourse status (given vs. new categories). In Experiment 1, all listeners inferred prominence when prosody on nouns distinguished categories ("SANDWICH"→not sandals). In Experiment 2, CI users and NH listeners presented with natural speech inferred prominence when prosody on adjectives implied contrast across both categories and properties ("PINK horse"→not the orange horse). In contrast, NH listeners presented with simulated CI (vocoded) speech were sensitive to acoustic differences in prosody, but did not use these cues to infer discourse status. Together, this suggests that exploiting redundant cues for comprehension varies with the demands of language processing and prior experience with the degraded signal.

Effects of contrastive accents on children's discourse comprehension

What role do contrastive accents play in children's discourse comprehension? By 6 years of age, children use contrastive accents during online comprehension to predict upcoming referents (Ito et al., 2014; Sekerina & Trueswell, 2012). But, at this age, children's performance on offline tasks of accent comprehension is poor (e.g., Wells et al., 2004). To examine whether the asymmetry could reflect a developmental stage in which the processing system uses contrastive accents to make local predictions, but fails to incorporate this information into discourse representations, we tested the effect of contrastive accents on children's memory of the content of a discourse. Five-year-olds heard 12 different stories consecutively, one after another, and the critical words were manipulated so that they were produced either with a contrastive L+H* accent or with a presentational H* accent. We found that children remembered facts about the contrast set better when the target word had an appropriate contrastive accent earlier than when it had a presentational accent. The results show that by 5 years, children are able to use contrastive accents for encoding a discourse, as well as for making local predictions during online comprehension.

Asking or Telling--Real-time Processing of Prosodically Distinguished Questions and Statements

We introduce a targeted language game approach using the visual world, eye-movement paradigm to assess when and how certain intonational contours affect the interpretation of utterances. We created a computer-based card game in which elliptical utterances such as "Got a candy" occurred with a nuclear contour most consistent with a yes-no question (H* H-H%) or a statement (L* L-L%). In Experiment I we explored how such contours are integrated online. In Experiment 2 we studied the expectations listeners have for how intonational contours signal intentions: do these reflect linguistic categories or rapid adaptation to the paradigm? Prosody had an immediate effect on interpretation, as indexed by the pattern and timing of fixations. Moreover, the association between different contours and intentions was quite robust in the absence of clear syntactic cues to sentence type, and was not due to rapid adaptation. Prosody had immediate effects on interpretation even though there was a construction-based bias to interpret "got a" as a question. Taken together, we believe this paradigm will provide further insights into how intonational contours and their phonetic realization interact with other cues to sentence type in online comprehension.

Exploiting Listener Gaze to Improve Situated Communication in Dynamic Virtual Environments

Beyond the observation that both speakers and listeners rapidly inspect the visual targets of referring expressions, it has been argued that such gaze may constitute part of the communicative signal. In this study, we investigate whether a speaker may, in principle, exploit listener gaze to improve communicative success. In the context of a virtual environment where listeners follow computer-generated instructions, we provide two kinds of support for this claim. First, we show that listener gaze provides a reliable real-time index of understanding even in dynamic and complex environments, and on a per-utterance basis. Second, we show that a language generation system that uses listener gaze to provide rapid feedback improves overall task performance in comparison with two systems that do not use gaze. Aside from demonstrating the utility of listener gaze in situated communication, our findings open the door to new methods for developing and evaluating multi-modal models of situated interaction.

Misleading Bias-Driven Expectations in Referential Processing and the Facilitative Role of Contrastive Accent

Probabilistic preferences are often facilitative in language processing and may assist in discourse prediction. However, occasionally these sources of information may lead to inaccurate expectations. The current study investigated a test case of this scenario. An eye-tracking experiment examined the interpretation of ambiguous personal pronouns in the context of implicit causality biases. We tested whether reference resolution may be facilitated online by contrastive accent in cases of a bias-inconsistent referent. Implicit causality biases directed looks to the biased noun phrase; however, when the name of the bias-inconsistent antecedent was accented (e.g., JOHN envied Bill because he ...), this tendency was modulated. Contrastive accent seems to dampen the occasionally confusing prediction of implicit causality biases in referential processing. This demonstrates one way in which the spoken language comprehension system copes with occasional misguidance of otherwise helpful probabilistic information.

Prosody and language comprehension

This review provides a summary of the most recent advances on the study of how prosody is used during language comprehension. Prosody is characterized as an abstract structure composed of discrete tonal elements aligned with the segmental composition of the sentence organized in constituents of increasing size, and this structure is influenced by the phonological, syntactic, and informational structures of the sentence. Here, we discuss evidence that listeners are affected by prosody when establishing those linguistic structures. Prosody has been shown to influence the segmentation of the utterance into syllables and words, and, in some cases, whether a syllable or word is judged to be present or not. The literature on how prosody informs the structural relationship between words and phrases is also discussed, contrasting views that assume a direct (albeit probabilistic) link between syntax and prosody with those that posit a complex interface between syntax and prosodic structure. Finally, the role of prosody in conveying important aspects pertaining to the sentence's information structure (i.e., which parts of the sentence's meaning are highlighted and brought forward to the discourse, which ones are presupposed and left in the background, which attitudes are being conveyed about the concepts or propositional content) has long been recognized. Current research focuses on which prosodic elements contribute to marking the dimensions (or semantic primitives) of the information structure.

When correction turns positive: processing corrective prosody in Dutch

Current research on spoken language does not provide a consistent picture as to whether prosody, the melody and rhythm of speech, conveys a specific meaning. Perception studies show that English listeners assign meaning to prosodic patterns, and, for instance, associate some accents with contrast, whereas Dutch listeners behave more controversially. In two ERP studies we tested how Dutch listeners process words carrying two types of accents, which either provided new information (new information accents) or corrected information (corrective accents), both in single sentences (experiment 1) and after corrective and new information questions (experiment 2). In both experiments corrective accents elicited a sustained positivity as compared to new information accents, which started earlier in context than in single sentences. The positivity was not modulated by the nature of the preceding question, suggesting that the underlying neural mechanism likely reflects the construction of an interpretation to the accented word, either by identifying an alternative in context or by inferring it when no context is present. Our experimental results provide strong evidence for inferential processes related to prosodic contours in Dutch.

The interplay between referential processing and local syntactic/semantic processing: ERPs to written Chinese discourse

Two event-related brain potential experiments were conducted to investigate the functional interplay between discourse-level referential processing and local syntactic/semantic processing of phrases. We manipulated both the syntactic/semantic coherence of a noun phrase (NP) and the referential ambiguity of the same NP. Incoherence of the NP elicited a P600 effect in both experiments. Referential ambiguities elicited a sustained negativity (Nref) in a subset of the participants in both experiments. Crucially, among participants showing robust Nref effects to referential ambiguity in the coherent condition, Nref effects were absent when the NP was incoherent. These results provide evidence against theories in which referential processing is functionally independent of local syntactic/semantic processing of phrases. Instead, a local phrase anomaly can block aspects of referential processing concerning ambiguity.

White bear effects in language production: evidence from the prosodic realization of adjectives

A central problem in recent research on speech production concerns the question to what extent speakers adapt their linguistic expressions to the needs of their addressees. It is claimed that speakers sometimes leak information about objects that are only visible for them and not for their listeners. Previous research only takes the occurrence of adjectives as evidence for the leakage of privileged information. The present study hypothesizes that leaked information is also encoded in the prosody of those adjectives. A production experiment elicited adjectives that leak information and adjectives that do not leak information. An acoustic analysis and prominence rating task showed that adjectives that leak information were uttered with a higher pitch and perceived as more prominent compared to adjectives that do not leak information. Furthermore, a guessing task suggested that the adjectives' prosody relates to how listeners infer possible privileged information.

Is it or isn't it: listeners make rapid use of prosody to infer speaker meanings

A visual world experiment examined the time course for pragmatic inferences derived from visual context and contrastive intonation contours. We used the construction It looks like an X pronounced with either (a) a H(*) pitch accent on the final noun and a low boundary tone, or (b) a contrastive L+H(*) pitch accent and a rising boundary tone, a contour that can support contrastive inference (e.g., It LOOKSL+H*like a zebraL-H%… (but it is not)). When the visual display contained a single related set of contrasting pictures (e.g. a zebra vs. a zebra-like animal), effects of LOOKSL+H* emerged prior to the processing of phonemic information from the target noun. The results indicate that the prosodic processing is incremental and guided by contextually-supported expectations. Additional analyses ruled out explanations based on context-independent heuristics that might substitute for online computation of contrast.

Interpretation of contrastive pitch accent in six- to eleven-year-old English-speaking children (and adults)

Both off-line and on-line comprehension studies suggest not only toddlers and preschoolers, but also older school-age children have trouble interpreting contrast-marking pitch prominence. To test whether children achieve adult-like proficiency in processing contrast-marking prosody during school years, an eye-tracking experiment examined the effect of accent on referential resolution in six- to eleven-year-old children and adults. In all age groups, a prominent accent facilitated the detection of a target in contrastive discourse sequences (pink cat → green cat), whereas it led to a garden path in non-contrastive sequences (pink rabbit → green monkey: the initial fixations were on rabbits). While the data indicate that children as young as age six immediately interpret contrastive accent, even the oldest child group showed delayed fixations compared to adults. We argue that the children's slower recovery from the garden path reflects the gradual development in cognitive flexibility that matures independently of general oculomotor control.

Aphasias and theories of linguistic representation: representing frequency, hierarchy, constructions, and sequential structure

Error and preservation patterns in aphasic speech show that the brain makes use of the frequencies of words, constructions, and collocations, as well as category membership and hierarchical structure, during language processing. Frequency effects are evident along two quasi-independent axes: syntagmatic (the sequential context, e.g., deploying correct functors, categories, and utterance-level intonation) and paradigmatic (the choice at any given linguistic level, e.g., selecting content words and modifying structures). Frequency along the syntagmatic axis is shown to play a role in errors involving idioms, constructions, and collocations that cross major phrasal boundaries. Along the paradigmatic axis, frequency affects errors involving lexical selection, competing functors and inflected forms (e.g., using plural where singular is required). An account of language representation and processing that encompasses frequency as well as categorization and structure is compatible with what we know about how the brain works: increased experience with a linguistic structure results in increased activation-and strengthening-of the neural networks involved in processing that structure. These claims are supported by the literature on experimental work in normal speakers. Parsimony, plus the unexamined assumption that mental representation is like a written record (entries either present or absent, structure displayable in two dimensions), has been a misleading guide to modeling language representation. The substantial redundancy in representations and processing that is introduced by incorporating both frequency-based and hierarchy-based information is in fact appropriate for the brain as a fast, reliable, massively parallel error-correcting network with very large storage capacity and gradient representation strength. WIREs Cogn Sci 2013, 4:651-663. doi: 10.1002/wcs.1257 CONFLICT OF INTEREST: The authors have declared no conflicts of interest for this article. For further resources related to this article, please visit the WIREs website.

Accounting for the listener: comparing the production of contrastive intonation in typically-developing speakers and speakers with autism

The present research investigates what drives the prosodic marking of contrastive information. For example, a typically developing speaker of a Germanic language like Dutch generally refers to a pink car as a "PINK car" (accented words in capitals) when a previously mentioned car was red. The main question addressed in this paper is whether contrastive intonation is produced with respect to the speaker's or (also) the listener's perspective on the preceding discourse. Furthermore, this research investigates the production of contrastive intonation by typically developing speakers and speakers with autism. The latter group is investigated because people with autism are argued to have difficulties accounting for another person's mental state and exhibit difficulties in the production and perception of accentuation and pitch range. To this end, utterances with contrastive intonation are elicited from both groups and analyzed in terms of function and form of prosody using production and perception measures. Contrary to expectations, typically developing speakers and speakers with autism produce functionally similar contrastive intonation as both groups account for both their own and their listener's perspective. However, typically developing speakers use a larger pitch range and are perceived as speaking more dynamically than speakers with autism, suggesting differences in their use of prosodic form.

Contextual influences on children's use of vocal affect cues during referential interpretation

In three experiments, we investigated 5-year-olds' sensitivity to speaker vocal affect during referential interpretation in cases where the indeterminacy is or is not resolved by speech information. In Experiment 1, analyses of eye gaze patterns and pointing behaviours indicated that 5-year-olds used vocal affect cues at the point where an ambiguous description was encountered. In Experiments 2 and 3, we used unambiguous situations to investigate how the referential context influences the ability to use affect cues earlier in the utterance. Here, we found a differential use of speaker vocal affect whereby 5-year-olds' referential hypotheses were influenced by negative vocal affect cues in advance of the noun, but not by positive affect cues. Together, our findings reveal how 5-year-olds use a speaker's vocal affect to identify potential referents in different contextual situations and also suggest that children may be more attuned to negative vocal affect than positive vocal affect, particularly early in an utterance.

Interpreting Pitch Accents in Online Comprehension: H* vs. L+H*

Although the presence or absence of a pitch accent clearly can play an important role in signaling the discourse and information structure of an utterance, whether the form of an accent determines the type of information it conveys is more controversial. We used an eye-tracking paradigm to investigate whether H*, which has been argued to signal new information, evokes different eye fixations than L+H*, which has been argued to signal the presence of contrast. Our results demonstrate that although listeners interpret these accents differently, their interpretive domains overlap. L+H* creates a strong bias toward contrast referents whereas H* is compatible with both new and contrast referents.

Bimodal bilinguals co-activate both languages during spoken comprehension

Bilinguals have been shown to activate their two languages in parallel, and this process can often be attributed to overlap in input between the two languages. The present study examines whether two languages that do not overlap in input structure, and that have distinct phonological systems, such as American Sign Language (ASL) and English, are also activated in parallel. Hearing ASL-English bimodal bilinguals' and English monolinguals' eye-movements were recorded during a visual world paradigm, in which participants were instructed, in English, to select objects from a display. In critical trials, the target item appeared with a competing item that overlapped with the target in ASL phonology. Bimodal bilinguals looked more at competing item than at phonologically unrelated items and looked more at competing items relative to monolinguals, indicating activation of the sign-language during spoken English comprehension. The findings suggest that language co-activation is not modality specific, and provide insight into the mechanisms that may underlie cross-modal language co-activation in bimodal bilinguals, including the role that top-down and lateral connections between levels of processing may play in language comprehension.

Interactive processing of contrastive expressions by Russian children

Children's ability to interpret color adjective noun phrases (e.g., red butterfly) as contrastive was examined in an eyetracking study with 6-year-old Russian children. Pitch accent placement (on the adjective red, or on the noun butterfly) was compared within a visual context containing two red referents (a butterfly and a fox) when only one of them had a contrast member (a purple butterfly) or when both had a contrast member (a purple butterfly and a grey fox). Contrastiveness was enhanced by the Russian-specific 'split constituent' construction (e.g., Red put butterfly . . .) in which a contrastive interpretation of the color term requires pitch accent on the adjective, with the nonsplit sentences serving as control. Regardless of the experimental manipulations, children had to wait until hearing the noun (butterfly) to identify the referent, even in splits. This occurred even under conditions for which the prosody and the visual context allow adult listeners to infer the relevant contrast set and anticipate the referent prior to hearing the noun (accent on the adjective in 1-Contrast scenes). Pitch accent on the adjective did facilitate children's referential processing, but only for the nonsplit constituents. Moreover, visual contexts that encouraged the correct contrast set (1-Contrast) only facilitated referential processing after hearing the noun, even in splits. Further analyses showed that children can anticipate the reference like adults but only when the contrast set is made salient by the preceding supportive discourse, that is, when the inference about the intended contrast set is provided by the preceding utterance.

Using the visual world paradigm to study language processing: a review and critical evaluation

We describe the key features of the visual world paradigm and review the main research areas where it has been used. In our discussion we highlight that the paradigm provides information about the way language users integrate linguistic information with information derived from the visual environment. Therefore the paradigm is well suited to study one of the key issues of current cognitive psychology, namely the interplay between linguistic and visual information processing. However, conclusions about linguistic processing (e.g., about activation, competition, and timing of access of linguistic representations) in the absence of relevant visual information must be drawn with caution.

Accent detection is a slippery slope: Direction and rate of F0 change drives listeners' comprehension

The present study tests whether listeners use F0, duration, or some combination of the two to identify the presence of an accented word in a short discourse. Participants' eye movements to previously mentioned and new objects were monitored as participants listened to instructions to move objects in a display. The name of the target object on critical trials was resynthesized from naturally-produced utterances so that it had either high or low F0 and either long or short duration. Fixations to the new object were highest when there was a steep rise in F0. Fixations to the previously mentioned object were highest when there was a steep drop in F0. These results suggest that listeners use F0 slope to make decisions about the presence of an accent, and that F0 and duration by themselves do not solely determine accent interpretation.

Situated sentence processing: the coordinated interplay account and a neurobehavioral model

Empirical evidence demonstrating that sentence meaning is rapidly reconciled with the visual environment has been broadly construed as supporting the seamless interaction of visual and linguistic representations during situated comprehension. Based on recent behavioral and neuroscientific findings, however, we argue for the more deeply rooted coordination of the mechanisms underlying visual and linguistic processing, and for jointly considering the behavioral and neural correlates of scene-sentence reconciliation during situated comprehension. The Coordinated Interplay Account (CIA; Knoeferle, P., & Crocker, M. W. (2007). The influence of recent scene events on spoken comprehension: Evidence from eye movements. Journal of Memory and Language, 57(4), 519-543) asserts that incremental linguistic interpretation actively directs attention in the visual environment, thereby increasing the salience of attended scene information for comprehension. We review behavioral and neuroscientific findings in support of the CIA's three processing stages: (i) incremental sentence interpretation, (ii) language-mediated visual attention, and (iii) the on-line influence of non-linguistic visual context. We then describe a recently developed connectionist model which both embodies the central CIA proposals and has been successfully applied in modeling a range of behavioral findings from the visual world paradigm (Mayberry, M. R., Crocker, M. W., & Knoeferle, P. (2009). Learning to attend: A connectionist model of situated language comprehension. Cognitive Science). Results from a new simulation suggest the model also correlates with event-related brain potentials elicited by the immediate use of visual context for linguistic disambiguation (Knoeferle, P., Habets, B., Crocker, M. W., & Münte, T. F. (2008). Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18(4), 789-795). Finally, we argue that the mechanisms underlying interpretation, visual attention, and scene apprehension are not only in close temporal synchronization, but have co-adapted to optimize real-time visual grounding of situated spoken language, thus facilitating the association of linguistic, visual and motor representations that emerge during the course of our embodied linguistic experience in the world.