Individual differences in the neural dynamics of visual narrative comprehension: The effects of proficiency and age of acquisition

Context modulates brain state dynamics and behavioral responses during narrative comprehension

Narrative comprehension is inherently context-sensitive, yet the brain and cognitive mechanisms by which brief contextual priming shapes story interpretation remain unclear. Using hidden Markov modeling (HMM) of fMRI data, we identified dynamic brain states as participants listened to an ambiguous spoken story under two distinct narrative contexts (affair vs. paranoia). We uncovered both context-invariant states-engaging auditory, language, and default mode networks-and context-specific states characterized by differential recruitment of control, salience, and visual networks. Narrative context selectively modulated the influence of character speech and linguistic features on brain state expression, with the central character's speech enhancing activation in shared states but suppressing activation in context-specific ones. Independent behavioral analyses revealed parallel context-dependent effects, with character-driven features exerting strong, selectively modulated influences on participants' judgments of narrative evidence. These findings demonstrate that brief narrative priming actively reshapes brain state dynamics and feature sensitivity during story comprehension, revealing how context guides moment-by-moment interpretive processing in naturalistic settings.

Are We Moving Too Fast?: Representation of Speed in Static Images

Despite pictures being static representations, they use various cues to suggest dynamic motion. To investigate the effectiveness of different motion cues in conveying speed in static images, we conducted 3 experiments. In Experiment 1, we compared subjective speed ratings given for motion lines trailing behind movers, suppletion lines replacing parts of the movers and backfixing lines set in the background against the baseline of having no extra cue. Experiment 2 was a replication of the first experiment with an addition of several motion lines considering the effect of quantity on conveyed speed. Experiment 3 then examined the actual time assessments of each cue and bare objects indicated for movers to complete their paths. Our results showed that motion cues vary in their effectiveness in depicting speed, with some influence from proficiency in reading manga. Motion lines, which index the path being traversed, remained less effective than suppletion and backfixing lines, which we argue encode the speed component of motion rather than directionality. However, adding more motion lines intensified the perceived speed of the movers. These static cues also influenced the actual time durations individuals indicated for fictitious motion events, in line with the subjective speed ratings. Altogether, our results suggest that different aspects of motion can be captured by different cues, and that the effectiveness of cues might be modulated by exposure to such patterns, in line with the premises of a visual lexicon view.

Brain-model neural similarity reveals abstractive summarization performance

Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS) between the abstractive summarization (ABS) models and the human brain and its correlation to the performance of ABS tasks. Specifically, representational similarity analysis (RSA) was used to measure the similarity between the representational patterns (RPs) of the BART, PEGASUS, and T5 models' hidden layers and the human brain's language RPs under different spatiotemporal conditions. Layer-wise ablation manipulation, including attention ablation and noise addition was employed to examine the hidden layers' effect on model performance. The results demonstrate that as the depth of hidden layers increases, the models' text encoding becomes increasingly similar to the human brain's language RPs. Manipulating deeper layers leads to more substantial decline in summarization performance compared to shallower layers, highlighting the crucial role of deeper layers in integrating essential information. Notably, the study confirms the hypothesis that the hidden layers exhibiting higher similarity to human brain activity play a more critical role in model performance, with their correlations reaching statistical significance even after controlling for perplexity. These findings deepen our understanding of the cognitive mechanisms underlying language representations in DLMs and their neural correlates, potentially providing insights for optimizing and improving language models by aligning them with the human brain's language-processing mechanisms.

Attention-language interface in Multilingual Assessment instrument for Narratives

The current study employed the Multilingual Assessment Instrument for Narratives (MAIN) to test comprehension of narrative macrostructure in Russian in a visual world eye-tracking paradigm. The four MAIN visual narratives are structurally similar and question referents' goals and internal states (IS). Previous research revealed that children's MAIN comprehension differed among the four narratives in German, Swedish, Russian, and Turkish, but it is not clear why. We tested whether the difference in comprehension was (a) present, (b) caused by complicated inferences in understanding IS compared with goals, and (c) ameliorated by orienting visual attention to the referents whose IS was critical for accurate comprehension. Our findings confirmed (a) and (b) but found no effect of attentional cues on accuracy for (c). The multidimensional theory of narrative organization of children's knowledge of macrostructure needs to consider the type of inferences necessary for IS that are influenced by subjective interpretation and reasoning.

Anaphoric distance dependencies in visual narrative structure and processing

Linguistic syntax has often been claimed as uniquely complex due to features like anaphoric relations and distance dependencies. However, visual narratives of sequential images, like those in comics, have been argued to use sequencing mechanisms analogous to those in language. These narrative structures include "refiner" panels that "zoom in" on the contents of another panel. Similar to anaphora in language, refiners indexically connect inexplicit referential information in one unit (refiner, pronoun) to a more informative "antecedent" elsewhere in the discourse. Also like in language, refiners can follow their antecedents (anaphoric) or precede them (cataphoric), along with having either proximal or distant connections. We here explore the constraints on visual narrative refiners created by modulating these features of order and distance. Experiment 1 examined participants' preferences for where refiners are placed in a sequence using a force-choice test, which revealed that refiners are preferred to follow their antecedents and have proximal distances from them. Experiment 2 then showed that distance dependencies lead to slower self-paced viewing times. Finally, measurements of event-related brain potentials (ERPs) in Experiment 3 revealed that these patterns evoke similar brain responses as referential dependencies in language (i.e., N400, LAN, Nref). Across all three studies, the constraints and (neuro)cognitive responses to refiners parallel those shown to anaphora in language, suggesting domain-general constraints on the sequencing of referential dependencies.

Semantic processing of verbal narratives compared to semantic processing of visual narratives: an ERP study of school-aged children

There is a misconception that pictures are easy to comprehend, which is problematic in pedagogical practices that include pictures. For example, if a child has difficulties with verbal narration to picture sequences, it may be interpreted as specific to spoken language even though the child may have additional difficulties with comprehension of visual narratives in the form of picture sequences. The purpose of the present study was therefore to increase our understanding of semantic processing in the pictorial domain in relation to semantic processing in the verbal domain, focusing on 9-13 years-old children with typical language development. To this end, we measured electrical brain responses (event related potentials, ERPs) in 17 children to (i) pictures (panels) that were predicted versus unpredicted in sequences of panels that conveyed visual narratives and (ii) words that were predicted versus unpredicted in sentences that conveyed verbal narratives. Results demonstrated similarities as there were no significant difference in the magnitude of the N400 effect across domains. The only difference between domains was the predicted difference in distribution, that is, a more posterior N400 effect in the verbal domain than in the pictorial domain. The study contributes to an increased understanding of the complexity of processing of visual narratives and its shared features with processing of verbal narratives, which should be considered in pedagogical practices.