Are visual processes causally involved in "perceptual simulation" effects in the sentence-picture verification task?

No support for a causal role of primary motor cortex in construing meaning from language: An rTMS study

Embodied cognition theories predict a functional involvement of sensorimotor processes in language understanding. In a preregistered experiment, we tested this idea by investigating whether interfering with primary motor cortex (M1) activation can change how people construe meaning from action language. Participants were presented with sentences describing actions (e.g., "turning off the light") and asked to choose between two interpretations of their meaning, one more concrete (e.g., "flipping a switch") and another more abstract (e.g., "going to sleep"). Prior to this task, participants' M1 was disrupted using repetitive transcranial magnetic stimulation (rTMS). The results yielded strong evidence against the idea that M1-rTMS affects meaning construction (BF01 > 30). Additional analyses and control experiments suggest that the absence of effect cannot be accounted for by failure to inhibit M1, lack of construct validity of the task, or lack of power to detect a small effect. In sum, these results do not support a causal role for primary motor cortex in building meaning from action language.

The influence of sentence focus on mental simulation: A possible cause of ACE instability

Recent studies have revealed the instability of the action-sentence compatibility effect (ACE). The current study was designed to demonstrate the hypothesis that the instability of the ACE may be attributed to the instability of focused information in a sentence. A pilot study indicated that the focused information of sentences was relatively stable in the sentence-picture verification task but exhibited significant interindividual variability in the action-sentence compatibility paradigm in previous studies. Experiments 1 and 2 examined the effect of sentence focus on the shape match effect and the ACE by manipulating the focused information of sentences using the focus marker word "" (is). Experiment 1 found that the shape match effect occurred in the original sentence, while it disappeared when the word "" (is) was used to make an object noun no longer the focus of a sentence. Experiment 2 failed to observe the ACE regardless of whether the sentence focus was on the action information. Experiment 3 modified the focus manipulation to observe its impact on the ACE using different fonts and underlines to highlight the focused information. The results indicated that the ACE only occurred when the action information was the sentence focus. These findings suggest that sentence focus influences mental simulation, and the instability of the ACE is likely to be associated with the instability of sentence focus in previous studies. This outcome highlights the crucial role of identifying specific information as the critical element expressed in the current linguistic context for successful simulation.

Brain Signatures of Embodied Semantics and Language: A Consensus Paper

According to embodied theories (including embodied, embedded, extended, enacted, situated, and grounded approaches to cognition), language representation is intrinsically linked to our interactions with the world around us, which is reflected in specific brain signatures during language processing and learning. Moving on from the original rivalry of embodied vs. amodal theories, this consensus paper addresses a series of carefully selected questions that aim at determining when and how rather than whether motor and perceptual processes are involved in language processes. We cover a wide range of research areas, from the neurophysiological signatures of embodied semantics, e.g., event-related potentials and fields as well as neural oscillations, to semantic processing and semantic priming effects on concrete and abstract words, to first and second language learning and, finally, the use of virtual reality for examining embodied semantics. Our common aim is to better understand the role of motor and perceptual processes in language representation as indexed by language comprehension and learning. We come to the consensus that, based on seminal research conducted in the field, future directions now call for enhancing the external validity of findings by acknowledging the multimodality, multidimensionality, flexibility and idiosyncrasy of embodied and situated language and semantic processes.

Embodied Processing at Six Linguistic Granularity Levels: A Consensus Paper

Language processing is influenced by sensorimotor experiences. Here, we review behavioral evidence for embodied and grounded influences in language processing across six linguistic levels of granularity. We examine (a) sub-word features, discussing grounded influences on iconicity (systematic associations between word form and meaning); (b) words, discussing boundary conditions and generalizations for the simulation of color, sensory modality, and spatial position; (c) sentences, discussing boundary conditions and applications of action direction simulation; (d) texts, discussing how the teaching of simulation can improve comprehension in beginning readers; (e) conversations, discussing how multi-modal cues improve turn taking and alignment; and (f) text corpora, discussing how distributional semantic models can reveal how grounded and embodied knowledge is encoded in texts. These approaches are converging on a convincing account of the psychology of language, but at the same time, there are important criticisms of the embodied approach and of specific experimental paradigms. The surest way forward requires the adoption of a wide array of scientific methods. By providing complimentary evidence, a combination of multiple methods on various levels of granularity can help us gain a more complete understanding of the role of embodiment and grounding in language processing.

Is time an embodied property of concepts?

A haircut usually lasts under an hour. But how long does it take to recognize that something is an instance of a haircut? And is this "time-to-perceive" a part of the representation of concepts like haircut? Across three experiments testing lexical decision, word recognition, and semantic decision, we show that the amount of time people say it takes to perceive a concept in the world (e.g., haircut, dandelion, or merit) predicts how long it takes for them to respond to a word referring to that thing, over and above the effects of other lexical-semantic variables (e.g., word frequency, concreteness) and other variables related to conceptual complexity (e.g., how confusable a concept is with other, similar concepts, or the diversity of the contexts in which a concept appears). These results suggest that our experience of how long it takes to recognize an instance of a concept can become a part of its representation, and that we simulate this information when reading words. Consequently, we suggest that time may be an embodied property of concepts.

Sentence-based mental simulations: Evidence from behavioral experiments using garden-path sentences

Language comprehenders activate mental representations of sensorimotor experiences related to the content of utterances they process. However, it is still unclear whether these sensorimotor simulations are driven by associations with words or by a more complex process of meaning composition into larger linguistic expressions, such as sentences. In two experiments, we investigated whether comprehenders indeed create sentence-based simulations. Materials were constructed such that simulation effects could only emerge from sentence meaning and not from word-based associations alone. We additionally asked when during sentence processing these simulations are constructed, using a garden-path paradigm. Participants read either a garden-path sentence (e.g., "As Mary ate the egg was in the fridge") or a corresponding unambiguous control with the same meaning and words (e.g., "The egg was in the fridge as Mary ate"). Participants then judged whether a depicted entity was mentioned in the sentence or not. In both experiments, picture response times were faster when the picture was compatible (vs. incompatible) with the sentence-based interpretation of the target entity (e.g., both for garden-path and control sentence: an unpeeled egg), suggesting that participants created simulations based on the sentence content and only operating over the sentence as a whole.

Simulating background settings during spoken and written sentence comprehension

Previous findings from the sentence-picture verification task demonstrated that comprehenders simulate visual information about intrinsic attributes of described objects. Of interest is whether comprehenders may also simulate the setting in which an event takes place, such as, for example, the light information. To address this question, four experiments were conducted in which participants (total N = 412) either listened to (Experiment 1) or read (Experiment 3) sentences like “The sun is shining onto a bench” followed by a picture with the matching object (bench) and either the matching lighting condition of the scene (sunlit bench against the sunlit background) or the mismatching one (moonlit bench against the moonlit background). In both experiments, response times (RTs) were shorter when the lighting condition of the pictured scene matched the one implied in the sentence. However, no difference in RTs was observed when the processing of spoken sentences was interfered with visual noise (Experiment 2). Specifically, the results showed that visual interference disrupted incongruent visual content activated by listening to the sentences, as evidenced by faster responses on mismatching trials. Similarly, no difference in RTs was observed when the lighting condition of the pictured scene matched sentence context, but the target object presented for verification mismatched sentence context (Experiment 4). Thus, the locus of simulation effect is on the lighting representation of the target object rather than the lighting representation of the background. These findings support embodied and situated accounts of cognition, suggesting that comprehenders do not simulate objects independently of background settings.

Grammatical Aspect and Mental Activation of Implied Instruments: A Mouse-Tracking Study in Persian

Can grammatical cues affect the way people activate event knowledge? We used mouse tracking to study the effect of grammatical aspect on the mental activation of instruments in Persian. Verb aspect is defined as how an action or event is extended over time, whether it is perfective (complete) or imperfective (incomplete). We predicted that imperfective aspect would trigger the mental activation of event knowledge, thus making participants mistakenly believe that the instrument with which the action is normally performed was present in the sentence. We tested this hypothesis using manual action verbs. Fifty female participants read a simple active sentence in which an actor had done or was doing an action (e.g., Sara has sliced/is slicing the zucchinis) with an implied instrument (knife). Then, they were presented with a picture of the implied instrument and judged whether the instrument was mentioned in the sentence they just read by clicking on the PRESENT or ABSENT response option. We predicted that participants would be less efficient at clicking ABSENT in the imperfective condition. In line with this prediction, we found that the imperfective condition caused significant deviation to the incorrect response PRESENT. However, no significant time differences emerged. The results are consistent with embodied views of language comprehension.

The state of the onion: Grammatical aspect modulates object representation during event comprehension

The present ERP study assessed whether grammatical aspect is used as a cue in online event comprehension, in particular when reading about events in which an object is visually changed. While perfective aspect cues holistic event representations, including an event's endpoint, progressive aspect highlights intermediate phases of an event. In a 2 × 3 design, participants read SVO sentences describing a change-of-state event (e.g., to chop an onion), with grammatical Aspect manipulated (perfective "chopped" vs progressive "was chopping"). Thereafter, they saw a Picture of an object either having undergone substantial state-change (SC; a chopped onion), no state-change (NSC; an onion in its original state) or an unrelated object (U; a cactus, acting as control condition). Their task was to decide whether the object in the Picture was mentioned in the sentence. We focused on N400 modulation, with ERPs time-locked to picture onset. U pictures elicited an N400 response as expected, suggesting detection of categorical mismatches in object type. For SC and NSC pictures, a whole-head follow-up analysis revealed a P300, implying people were engaged in detailed evaluation of pictures of matching objects. SC pictures received most positive responses overall. Crucially, there was an interaction of Aspect and Picture: SC pictures resulted in a higher amplitude P300 after sentences in the perfective compared to the progressive. Thus, while the perfective cued for a holistic event representation, including the resultant state of the affected object (i.e., the chopped onion) constraining object representations online, the progressive defocused event completion and object-state change. Grammatical aspect thus guided online event comprehension by cueing the visual representation(s) of an object's state.

Towards Strong Inference in Research on Embodiment - Possibilities and Limitations of Causal Paradigms

A central question in the cognitive sciences is which role embodiment plays for high-level cognitive functions, such as conceptual processing. Here, we propose that one reason why progress regarding this question has been slow is a lacking focus on what Platt (1964) called “strong inference”. Strong inference is possible when results from an experimental paradigm are not merely consistent with a hypothesis, but they provide decisive evidence for one particular hypothesis compared to competing hypotheses. We discuss how causal paradigms, which test the functional relevance of sensory-motor processes for high-level cognitive functions, can move the field forward. In particular, we explore how congenital sensory-motor disorders, acquired sensory-motor deficits, and interference paradigms with healthy participants can be utilized as an opportunity to better understand the role of sensory experience in conceptual processing. Whereas all three approaches can bring about valuable insights, we highlight that the study of congenitally and acquired sensorimotor disorders is particularly effective in the case of conceptual domains with strong unimodal basis (e.g., colors), whereas interference paradigms with healthy participants have a broader application, avoid many of the practical and interpretational limitations of patient studies, and allow a systematic and step-wise progressive inference approach to causal mechanisms.

Explicit (Not Implicit) Attitudes Mediate the Focus of Attention During Sentence Processing

Many studies showed that comprehenders monitor changes in protagonists' emotions and actions. This article reports two experiments that explored how focusing comprehenders' attention on a particular property of the protagonist dimension (e.g., emotional or action state) affects the accessibility of information about target objects mentioned in the sentence. Furthermore, the present research examined whether participants' attitudes toward the issues described in the sentence can modulate comprehension processes. To this end, we asked participants to read sentences about environmental issues that focused comprehenders' attention on different mental and physical attributes of the same entities (protagonists and objects) and then self-report their own thoughts on the topic of environment by responding to the items assessing their environmental awareness. Importantly, we manipulated the task requirements across two experiments by administering a self-report task (Experiment 1), which required the participants to rate the seriousness and the frequency of the problem mentioned in a sentence; and administering a sentence-picture verification paradigm (Experiment 2), which required the participants to merely indicate if the object depicted in the picture (related to a certain environmental problem) was mentioned in the preceding sentence. The results of these experiments suggest that the focus of a sentence on the environmental problem (rather than the protagonist's emotion and action) enhances the accessibility of information about environmental issues (e.g., plastic garbage); that the comprehender's level of environmental awareness influences one's attention during sentence processing; and that comprehender characteristics significantly modulate comprehension processes only when the measures tap into explicit (and not implicit) processes.

Towards Understanding the Task Dependency of Embodied Language Processing: The Influence of Colour During Language-Vision Interactions

A main challenge for theories of embodied cognition is to understand the task dependency of embodied language processing. One possibility is that perceptual representations (e.g., typical colour of objects mentioned in spoken sentences) are not activated routinely but the influence of perceptual representation emerges only when context strongly supports their involvement in language. To explore this question, we tested the effects of colour representations during language processing in three visual-world eye-tracking experiments. On critical trials, participants listened to sentence-embedded words associated with a prototypical colour (e.g., ‘…spinach…’) while they inspected a visual display with four printed words (Experiment 1), coloured or greyscale line drawings (Experiment 2) and a ‘blank screen’ after a preview of coloured or greyscale line drawings (Experiment 3). Visual context always presented a word/object (e.g., frog) associated with the same prototypical colour (e.g. green) as the spoken target word and three distractors. When hearing spinach participants did not prefer the written word frog compared to other distractor words (Experiment 1). In Experiment 2, colour competitors attracted more overt attention compared to average distractors, but only for the coloured condition and not for greyscale trials. Finally, when the display was removed at the onset of the sentence, and in contrast to the previous blank-screen experiments with semantic competitors, there was no evidence of colour competition in the eye-tracking record (Experiment 3). These results fit best with the notion that the main role of perceptual representations in language processing is to contextualize language in the immediate environment.

Challenges and Opportunities for Grounding Cognition

According to the grounded perspective, cognition emerges from the interaction of classic cognitive processes with the modalities, the body, and the environment. Rather than being an autonomous impenetrable module, cognition incorporates these other domains intrinsically into its operation. The Situated Action Cycle offers one way of understanding how the modalities, the body, and the environment become integrated to ground cognition. Seven challenges and opportunities are raised for this perspective: (1) How does cognition emerge from the Situated Action Cycle and in turn support it? (2) How can we move beyond simply equating embodiment with action, additionally establishing how embodiment arises in the autonomic, neuroendocrine, immune, cardiovascular, respiratory, digestive, and integumentary systems? (3) How can we better understand the mechanisms underlying multimodal simulation, its functions across the Situated Action Cycle, and its integration with other representational systems? (4) How can we develop and assess theoretical accounts of symbolic processing from the grounded perspective (perhaps using the construct of simulators)? (5) How can we move beyond the simplistic distinction between concrete and abstract concepts, instead addressing how concepts about the external and internal worlds pattern to support the Situated Action Cycle? (6) How do individual differences emerge from different populations of situational memories as the Situated Action Cycle manifests itself differently across individuals? (7) How can constructs from grounded cognition provide insight into the replication and generalization crises, perhaps from a quantum perspective on mechanisms (as exemplified by simulators).

Is Complex Visual Information Implicated During Language Comprehension? The Case of Cast Shadows

Previous research showed that sensorimotor information affects the perception of properties associated with implied perceptual context during language comprehension. Three experiments addressed a novel question of whether perceptual context may contribute to a simulation of information about such out-of-sight objects as cast shadows. In Experiment 1, participants read a sentence that implied a particular shadow cast on a target (blinds vs. an open window) and then verified the picture of the object onto which a shadow was cast. Responses were faster when the shadow of blinds cast on the object matched that implied by the sentence. However, the data did not show the same matching effect for pictures with cast shadows from an open window. In Experiments 2 and 3, we found that verification times for pictures with no cast shadows were faster when preceded by an "open window" sentence, thus suggesting that reading the sentence does not elicit a visual simulation of any specific shadow. Experiment 3 showed that the objects superimposed with a cast shadow of the blinds and blinds themselves were verified faster after reading a "blinds" sentence. However, the results of an order analysis showed the temporal stability of the "blinds shadows" effect, but the disappearance of the "blinds" effect in the second half of the data. We conclude that the results are compatible, to a lesser or greater extent, with multiple accounts, and discuss our findings in the context of a mental imagery view, a mental simulation view, and an amodal representation view.

Making It Harder to "See" Meaning: The More You See Something, the More Its Conceptual Representation Is Susceptible to Visual Interference

Does the perceptual system for looking at the world overlap with the conceptual system for thinking about it? We conducted two experiments (N = 403) to investigate this question. Experiment 1 showed that when people make simple semantic judgments on words, interference from a concurrent visual task scales in proportion to how much visual experience they have with the things the words refer to. Experiment 2 showed that when people make the same judgments on the very same words, interference from a concurrent manual task scales in proportion to how much manual (but critically, not visual) experience people have with those same things. These results suggest that the meanings of frequently visually experienced things are represented (in part) in the visual system used for actually seeing them, that this visually represented information is a functional part of conceptual knowledge, and that the extent of these visual representations is influenced by visual experience.

Six Challenges for Embodiment Research

Twenty years after Barsalou’s seminal perceptual-symbols article, embodied cognition, the notion that cognition involves simulations of sensory, motor, or affective states, has moved from an outlandish proposal to a mainstream position adopted by many researchers in the psychological and cognitive sciences (and neurosciences). Though it has generated productive work in the cognitive sciences as a whole, it has had a particularly strong impact on research into language comprehension. The view of a mental lexicon based on symbolic word representations, which are arbitrarily linked to sensory aspects of their referents, was generally accepted since the cognitive revolution in the 1950s. This has radically changed. Given the current status of embodiment as a main theory of cognition, it is somewhat surprising that a close look at the literature reveals that the debate about the nature of the processes involved in language comprehension is far from settled, and key questions remain unanswered. We present several suggestions for a productive way forward.

Work design for airport security officers: Effects of rest break schedules and adaptable automation

This study investigated whether there is empirical support for the current EU regulation mandating breaks of at least 10 min after each period of 20 min continuously reviewing X-ray images in airport security screening. As a second goal, it examined whether providing more autonomy to airport security officers (in the form of spontaneous rest breaks and adaptable automation) would improve their performance and subjective state. Seventy-two student participants had to indicate the presence (or absence) of a threat item (either a gun or a knife) in a series of grey-scaled X-ray images of cabin baggage. Three work-rest schedules were examined: spontaneous breaks (i.e. participants could take breaks at any time), two 5-min breaks and two 10-min breaks during a 1-h testing session. Furthermore, half of the participants were assisted in their task by an adaptable support system offering three levels of automation: (1) no support, (2) cues indicating the presence of a potential threat item, and (3) cues indicating the exact location of a potential threat item. Results showed no performance differences between break regimes, which suggests that there may be viable alternatives to the current EU regulations. It also emerged that providing participants with adaptable automation did not lead to better detection performance but resulted in a less positive response bias than participants without automatic support. Implications for current aviation security regulations are discussed.