Encoding modality and spatial memory retrieval

Matching different-structured advertising pictorial metaphors with verbalization forms: incongruity-based evoked response potentials evidence

Many studies emphasize the need of verbally representing pictorial metaphors, but few have empirically investigated whether and how the particular verbalization form match different types of pictorial metaphors. Using evoked response potentials (ERP), a 3 (pictorial structure: fusion, juxtaposition, literal image) × 2 [verbalization form: A(is) B, A(is like) B] within-group experiment was conducted among 36 participants. ERPs were time-locked to the onset of the verb [/(is/is like)] of the metaphor sentence that follows a pictorial metaphor to detect the verbo-pictorial incongruity in metaphor comprehension. The incongruity-based ERP analysis showed that pictorial metaphors, when verbalized in two forms, all induced frontal N1 effect, regardless of pictorial structures, only with a larger N1 amplitude for literal images in "A(is) B." A central stronger P2 was observed in "A(is like) B" for three structures. Despite a general elicitation of posterior P3 in all conditions, a larger P3 was found for juxtapositions verbalized in "A(is like) B" and for literal images verbalized in "A(is) B." There was no significant difference between two verbalization forms for fusion-structured pictorial metaphors. These findings suggest: (1) verbo-pictorial metaphors could induce incongruity-based attention; (2) higher verbo-pictorial semantic congruity and relatedness, indexed by stronger P2 and P3, confirmed "A(is like) B" to be the more effective verbalization form in representing pictorial metaphors, specifically for juxtaposition-structured pictorial metaphors; (3) for non-metaphor advertising pictures, verbal metaphor showed an interference effect. The study not only reveals the neuro-cognitive mechanism of processing verbo-pictorial metaphors, but also offers neural reference for the design of effective multi-modal metaphor by finding an optimal match between PMs and verbalization forms.

The linguistic context effects on the processing of body-object interaction words: An ERP study on second language learners

Embodied theories of cognition argue that the processing of both concrete and abstract concepts requires the activation of sensorimotor systems. The present study examined the time course for embedding a sensorimotor context in order to elicit sensitivity to the sensorimotor consequences of understanding body-object interaction (BOI) words. In the study, Event-Related Potentials (ERPs) were recorded while subjects performed a sentence acceptability task. Target BOI words were preceded by rich or poor sensorimotor sentential contexts. The behavioural results replicated previous findings in that high BOI words received a response faster than low BOI words. In addition to this, however, there was a context effect in the sensorimotor region as well as a BOI effect in the parietal region (involved in object representation). The results indicate that the sentential sensorimotor context contributes to the subsequent BOI processing and that action-and perception-related language leads to the activation of the same brain areas, which is consistent with the embodiment theory.

Chinese-English bilinguals processing temporal-spatial metaphor

The conceptual projection of time onto the domain of space constitutes one of the most challenging issues in the cognitive embodied theories. In Chinese, spatial order (e.g.,/da shu qian/, in front of a tree) shares the same terms with temporal sequence (", /san yue qian/, before March). In comparison, English natives use different sets of prepositions to describe spatial and temporal relationship, i.e., "before" to express temporal sequencing and "in front of" to express spatial order. The linguistic variations regarding the specific lexical encodings indicate that some flexibility might be available in how space-time parallelisms are formulated across different languages. In the present study, ERP (Event-related potentials) data were collected when Chinese-English bilinguals processed temporal ordering and spatial sequencing in both their first language (L1) Chinese (Experiment 1) and the second language (L2) English (Experiment 2). It was found that, despite the different lexical encodings, early sensorimotor simulation plays a role in temporal sequencing processing in both L1 Chinese and L2 English. The findings well support the embodied theory that conceptual knowledge is grounded in sensory-motor systems (Gallese and Lakoff, Cogn Neuropsychol 22:455-479, 2005). Additionally, in both languages, neural representations during comprehending temporal sequencing and spatial ordering are different. The time-spatial relationship is asymmetric, in that space schema could be imported into temporal sequence processing but not vice versa. These findings support the weak view of the Metaphoric Mapping Theory.

Spatial metaphor processing during temporal sequencing comprehension

People all over the world use spatial metaphors to represent temporal sequencing, but so far the online stages during the comprehension of temporal sequencing still lack evidence. To explore this issue, the current EEG study constructed vertical anomaly and horizontal anomaly by exchanging spatial terms in vertical and horizontal temporal sequencing concepts in Mandarin. Both the vertical and the horizontal anomaly indicated an N400 effect (340 ms after stimulus onset) and a positive effect in a late time window (620 ms after stimulus onset). Additionally, the horizontal anomaly showed a reduced P2 effect (200-260 ms after stimulus onset) that might related to spatial processing. Direct comparison between the two anomaly effects revealed that the vertical anomaly evoked a stronger effect in an early N400 time window (from 360 to 440 ms), whereas the horizontal anomaly evoked a stronger effect in a late N400 time window (from 500 to 580 ms). These results indicate multiple stages during the comprehension of temporal sequencing. The relation between the current results and spatial metaphor differences is also discussed.

Cross-sensory transfer of reference frames in spatial memory

Two experiments investigated whether visual cues influence spatial reference frame selection for locations learned through touch. Participants experienced visual cues emphasizing specific environmental axes and later learned objects through touch. Visual cues were manipulated and haptic learning conditions were held constant. Imagined perspective taking when recalling touched objects was best from perspectives aligned with visually-defined axes, providing evidence for cross-sensory reference frame transfer. These findings advance spatial memory theory by demonstrating that multimodal spatial information can be integrated within a common spatial representation.