Construction and updating of event models in auditory event processing

Construction or updating? Event model processes during visual narrative comprehension

The plot of a narrative is represented in the form of event models in working memory. Because only parts of the plot are actually presented and information is continually changing, comprehenders have to infer a good portion of a narrative and keep their mental representation updated. Research has identified two related processes (e.g., Gernsbacher, 1997): During model construction (shifting, laying a foundation) at large coherence breaks an event model is completely built anew. During model updating (mapping) at smaller omissions, however, the current event model is preserved, and only changed parts are updated through inference processes. Thus far, reliably distinguishing those two processes in visual narratives like comics was difficult. We report a study (N = 80) that aimed to map the differences between constructing and updating event models in visual narratives by combining measures from narrative comprehension and event cognition research and manipulating event structure. Participants watched short visual narratives designed to (not) contain event boundaries at larger coherence breaks and elicit inferences through small omissions, while we collected viewing time measures as well as event segmentation and comprehensibility data. Viewing time, segmentation, and comprehensibility data were in line with the assumption of two distinct comprehension processes. We thus found converging evidence across multiple measures for distinct model construction and updating processes in visual narratives.

Multisensory synchrony of contextual boundaries affects temporal order memory, but not encoding or recognition

We memorize our daily life experiences, which are often multisensory in nature, by segmenting them into distinct event models, in accordance with perceived contextual or situational changes. However, very little is known about how multisensory boundaries affect segmentation, as most studies have focused on unisensory (visual or audio) segmentation. In three experiments, we investigated the effect of multisensory boundaries on segmentation in memory and perception. In Experiment 1, participants encoded lists of pictures while audio and visual contexts changed synchronously or asynchronously. After each list, we tested recognition and temporal associative memory for pictures that were encoded in the same audio-visual context or that crossed a synchronous or an asynchronous multisensory change. We found no effect of multisensory synchrony for recognition memory: synchronous and asynchronous changes similarly impaired recognition for pictures encoded at those changes, compared to pictures encoded further away from those changes. Multisensory synchrony did affect temporal associative memory, which was worse for pictures encoded at synchronous than at asynchronous changes. Follow up experiments showed that this effect was not due to the higher dimensionality of multisensory over unisensory contexts (Experiment 2), nor that it was due to the temporal unpredictability of contextual changes inherent to Experiment 1 (Experiment 3). We argue that participants formed situational expectations through multisensory synchronicity, such that synchronous multisensory changes deviated more strongly from those expectations than asynchronous changes. We discuss our findings in light of supportive and conflicting findings of uni- and multi-sensory segmentation.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

Cross-codal integration of bridging-event information in narrative understanding

Visual narratives communicate event sequences by using different code systems such as pictures and texts. Thus, comprehenders must integrate information from different codalities. This study addressed such cross-codal integration processes by investigating how the codality of bridging-event information (i.e., pictures, text) affects the understanding of visual narrative events. In Experiment 1, bridging-event information was either present (as picture or text) or absent (i.e., not shown). The viewing times for the subsequent picture depicting the end state of the action were comparable within the absent and the text conditions. Further, the viewing times for the end-state picture were significantly longer in the text condition as compared to the pictorial condition. In Experiment 2, we tested whether replacing bridging-event information with a blank panel increases viewing times in a way similar to the text condition. Bridging event information was either present (as picture) or absent (not shown vs. blank panel). The results replicated Experiment 1. Additionally, the viewing times for the end-state pictures were longest in the blank condition. In Experiment 3, we investigated the costs related to integrating information from different codalities by directly comparing the text and picture conditions with the blank condition. The results showed that the distortion caused by the blank panel is larger than the distortion caused by cross-codal integration processes. Summarizing, we conclude that cross-codal information processing during narrative comprehension is possible but associated with additional mental effort. We discuss the results with regard to theories of narrative understanding.

Event Representations and Predictive Processing: The Role of the Midline Default Network Core

The human brain is tightly coupled to the world through its sensory‐motor systems—but it also spends a lot of its metabolism talking to itself. One important function of this intrinsic activity is the establishment and updating of event models—representations of the current situation that can predictively guide perception, learning, and action control. Here, we propose that event models largely depend on the default network (DN) midline core that includes the posterior cingulate and anterior medial prefrontal cortex. An increasing body of data indeed suggests that this subnetwork can facilitate stimuli processing during both naturalistic event comprehension and cognitive tasks in which mental representations of prior situations, trials, and task rules can predictively guide attention and performance. This midline core involvement in supporting predictions through event models can make sense of an otherwise complex and conflicting pattern of results regarding the possible cognitive functions subserved by the DN.

Stawarczyk, Bezdek, and Zacks offer neuroscience evidence for a midline default network core, which appears to coordinate internal, top‐down mentation with externally‐triggered, bottom‐up attention in a push‐pull relationship. The network may enable the flexible pursuance of thoughts tuned into or detached from the current environment.

Linguistic Information in Auditory Dynamic Events Contributes to the Detection of Fine, Not Coarse Event Boundaries

Human observers (comprehenders) segment dynamic information into discrete events. That is, although there is continuous sensory information, comprehenders perceive boundaries between two meaningful units of information. In narrative comprehension, comprehenders use linguistic, non-linguistic , and physical cues for this event boundary perception. Yet, it is an open question - both from a theoretical and an empirical perspective - how linguistic and non-linguistic cues contribute to this process. The current study explores how linguistic cues contribute to the participants' ability to segment continuous auditory information into discrete, hierarchically structured events. Native speakers of German and non-native speakers, who neither spoke nor understood German, segmented a German audio drama into coarse and fine events. Whereas native participants could make use of linguistic, non-linguistic, and physical cues for segmentation, non-native participants could only use non-linguistic and physical cues. We analyzed segmentation behavior in terms of the ability to identify coarse and fine event boundaries and the resulting hierarchical structure. Non-native listeners identified almost identical coarse event boundaries as native listeners, but missed some of the fine event boundaries identified by the native listeners. Interestingly, hierarchical event perception (as measured by hierarchical alignment and enclosure) was comparable for native and non-native participants. In summary, linguistic cues contributed particularly to the identification of certain fine event boundaries. The results are discussed with regard to the current theories of event cognition.

Filling the gap despite full attention: the role of fast backward inferences for event completion

The comprehension of dynamic naturalistic events poses at least two challenges to the cognitive system: filtering relevant information with attention and dealing with information that was missing or missed. With four experiments, we studied the completion of missing information despite full attention. Participants watched short soccer video clips and we informed participants that we removed a critical moment of ball contact in half of the clips. We asked participants to detect whether these moments of ball contact were present or absent. In Experiment 1, participants gave their detection responses either directly during an event or delayed after an event. Although participants directed their full attention toward the critical contact moment, they were more likely to indicate seeing the missing ball contact if it was followed by a causally matching scene than if it was followed by an unrelated scene, both for the immediate and delayed responses. Thus, event completion occurs quickly. In Experiment 2, only a causally matching scene but neither a white mask nor an irrelevant scene caused the completion of missing information. This indicates that the completion of missing information is caused by backward inferences rather than predictive perception. In Experiment 3, we showed that event completion occurs directly during a trial and does not depend on expectations built up after seeing the same causality condition multiple times. In Experiment 4, we linked our findings to event cognition by asking participants to perform a natural segmentation task. We conclude that observers complete missing information during coherent events based on a fast backward inference mechanism even when directing their attention toward the missing information.