Mental animation in the visuospatial sketchpad: evidence from dual-task studies

Verbal interference paradigms: A systematic review investigating the role of language in cognition

This paper presents a systematic review of the empirical literature that uses dual-task interference methods for investigating the on-line involvement of language in various cognitive tasks. In these studies, participants perform some primary task X putatively recruiting linguistic resources while also engaging in a secondary, concurrent task. If performance on the primary task decreases under interference, there is evidence for language involvement in the primary task. We assessed studies (N = 101) reporting at least one experiment with verbal interference and at least one control task (either primary or secondary). We excluded papers with an explicitly clinical, neurological, or developmental focus. The primary tasks identified include categorization, memory, mental arithmetic, motor control, reasoning (verbal and visuospatial), task switching, theory of mind, visual change, and visuospatial integration and wayfinding. Overall, the present review found that covert language is likely to play a facilitative role in memory and categorization when items to be remembered or categorized have readily available labels, when inner speech can act as a form of behavioral self-cuing (inhibitory control, task set reminders, verbal strategy), and when inner speech is plausibly useful as "workspace," for example, for mental arithmetic. There is less evidence for the role of covert language in cross-modal integration, reasoning relying on a high degree of visual detail or items low on nameability, and theory of mind. We discuss potential pitfalls and suggestions for streamlining and improving the methodology.

Multiple mental representations in picture processing

This article investigates whether goal-directed learning of pictures leads to multiple mental representations which are differently useful for different purposes. The paper further investigates the effects of prompts on picture processing. 136 undergraduate students were presented maps of a fictitious city. One half of the participants were instructed to learn their map as preparation to draw it from memory as precisely as possible (PrepDraw), which should stimulate the creation of an elaborated surface representation. The other half were instructed to learn the map as preparation for finding the shortest traffic connection from various locations to other locations (PrepConnect), which should stimulate the construction of a task-oriented deep-structure representation (mental model). Within both experimental groups, one-third of the participants received the map without prompts. Another third received the map with survey prompts (stimulating processing of what is where), and the final third received the map with connect prompts (stimulating processing of how train stations are connected). In the following test phase, participants received a recognition task, a recall task, and an inference task. For recognition and recall, two surface structure scores (extent, accuracy) and two deep structure scores (extent, accuracy) were calculated. The inference task served also to indicate deep structure accuracy. The PrepDraw group outperformed the PrepConnect group in terms of surface structure related variables, whereas the PrepConnect group outperformed the PrepDraw group in terms of deep structure-related variables. Map processing was not enhanced by prompts aligned with the instruction, but non-aligned prompts tended to interfere with learning.

Texts and pictures serve different functions in conjoint mental model construction and adaptation

In this study we examined the different functions of text and pictures during text-picture integration in multimedia learning. In Study 1, 144 secondary school students (age = 11 to 14 years; 72 females, 72 males) received six text-picture units under two conditions. In the delayed-question condition, students first read the units without a specific question (no-question phase), to stimulate initial coherence-oriented mental model construction. Afterward the question was presented (question-answering phase), to stimulate task-adaptive mental model specification. In the preposed-question condition, students received a specific question from the beginning, stimulating both kinds of processing. Analyses of the participants' eye movement patterns confirmed the assumption that students allocated a higher percentage of available resources to text processing during the initial mental model construction than during adaptive model specification. Conversely, students allocated a higher percentage of available resources to picture processing during adaptive mental model specification than during the initial mental model construction. In Study 2 (N = 12, age = 12 to 16; seven females, five males), we ruled out that these findings were due to the effect of rereading, by implementing a no-question phase either once or twice. To sum up, texts seem to provide more explicit conceptual guidance in mental model construction than pictures do, whereas pictures support mental model adaptation more than text does, by providing flexible access to specific information for task-oriented updates.

Origami folding: Taxing resources necessary for the acquisition of sequential skills

Sequential skill learning with practice is fundamental to human activity (e.g., tying shoes). Given the lack of prior knowledge in most participants, Origami folding is a promising task to study the acquisition of a sequential skill. While previous Origami folding studies mainly dealt with the question, which forms of instruction can lead to better learning outcomes, we employ a dual-task approach to test which resources are necessary for folding and for improvement with practice. Participants (N = 53) folded five Origami figures for four times each, which were randomly paired with five types of secondary tasks to cause either cognitive (verbal vs. visuospatial) or motoric (isochronous vs. non-isochronous tapping) memory load or none (control condition). Origami performance showed a typical learning curve from Trial 1 (first run of folding the figure) to Trial 4 (fourth run of folding the same figure). We tested for a dissociation between variants of memory load influencing Origami folding performance vs. the variants influencing learning (i.e. change in performance across practice). In line with theories suggesting that learning operates on the level that (at a given point in practice) demands the most control, we did not observe cases where a dual-task variant influenced performance while it did not affect learning. Memory load from the cognitive visuospatial secondary task as well as the isochronous tapping secondary task interfered with improvement in Origami folding with practice. This might be due to the use of visuospatial sketchpad and absolute timing mechanism during the acquisition of Origami folding.

Mnemonic effects of action simulation from pictures and phrases

Several theoretical approaches suggest that language comprehension and action observation rely on similar mental simulations. Granted that these two simulations partially overlap, we assumed that simulations stemming from action observations are more direct than those stemming from action phrases. The implied prediction was that simulation from action observation should prevail on simulation from action phrases when their effects are contrasted. The results of three experiments confirmed that, when at encoding the phrases were paired with pictures of actions whose kinematics was incongruent with the implied kinematics of the actions described in the phrases, memory for action phrases was impaired (Experiment 1). However, the reverse was not true: when the pictures were paired with phrases representing actions whose kinematics were incongruent with the kinematics of the actions portrayed in the pictures, memory for pictures portraying actions was not impaired (Experiment 2). Also, in line with evidence that simulations from action phrases and those from action observation partially overlap, when their effects were not contrasted their products were misrecognized. In our experiments, when action phrases only presented at recognition described actions depicted in pictures seen at encoding, they were misrecognized as had already been read at encoding (Experiment 1); further, when pictures only presented at recognition portrayed actions described in phrases presented at encoding, they were misrecognized as seen at encoding (Experiment 2). A third experiment excluded the possibility that the pattern of findings was simply a consequence of better memory for pictures of actions as opposed to memory for action phrases (Experiment 3). The implications of our results in relation to the literature on simulation in language comprehension and action observation are discussed.

Auditory and motor imagery modulate learning in music performance

Skilled performers such as athletes or musicians can improve their performance by imagining the actions or sensory outcomes associated with their skill. Performers vary widely in their auditory and motor imagery abilities, and these individual differences influence sensorimotor learning. It is unknown whether imagery abilities influence both memory encoding and retrieval. We examined how auditory and motor imagery abilities influence musicians' encoding (during Learning, as they practiced novel melodies), and retrieval (during Recall of those melodies). Pianists learned melodies by listening without performing (auditory learning) or performing without sound (motor learning); following Learning, pianists performed the melodies from memory with auditory feedback (Recall). During either Learning (Experiment 1) or Recall (Experiment 2), pianists experienced either auditory interference, motor interference, or no interference. Pitch accuracy (percentage of correct pitches produced) and temporal regularity (variability of quarter-note interonset intervals) were measured at Recall. Independent tests measured auditory and motor imagery skills. Pianists' pitch accuracy was higher following auditory learning than following motor learning and lower in motor interference conditions (Experiments 1 and 2). Both auditory and motor imagery skills improved pitch accuracy overall. Auditory imagery skills modulated pitch accuracy encoding (Experiment 1): Higher auditory imagery skill corresponded to higher pitch accuracy following auditory learning with auditory or motor interference, and following motor learning with motor or no interference. These findings suggest that auditory imagery abilities decrease vulnerability to interference and compensate for missing auditory feedback at encoding. Auditory imagery skills also influenced temporal regularity at retrieval (Experiment 2): Higher auditory imagery skill predicted greater temporal regularity during Recall in the presence of auditory interference. Motor imagery aided pitch accuracy overall when interference conditions were manipulated at encoding (Experiment 1) but not at retrieval (Experiment 2). Thus, skilled performers' imagery abilities had distinct influences on encoding and retrieval of musical sequences.

An association study on the polymorphisms of dopaminergic genes with working memory in a healthy Chinese Han population

Working memory (WM) is a highly heritable cognitive trait that is involved in many higher-level cognitive functions. In the past few years, much evidence has indicated that the reduction of dopamine activity in human brain can impair the WM system of the neuropsychiatric disorders. In this study, we hypothesized that some genes in the dopamine system were involved in the individual difference of the cognitive ability in healthy population. To confirm this hypothesis, a population-based study was performed to examine the effects of COMT, DAT (1), DRD (1), DRD (2), DRD (3), and DRD (4) on WM spans. Our results indicated there were significant associations of TaqIA and TaqIB in DRD (2) with digital WM span, respectively (χ(2) = 9.460, p = 0.009; χ(2) = 6.845, p = 0.033). On the other hand, we found a significant interaction between Ser9Gly in DRD (3) and TaqIA of DRD (2) on digital WM span (F = 3.207, p = 0.013). COMT, DAT (1) , DRD (1), and DRD (4), however, had no significant effects on digital and spatial WM spans (χ(2)<3.84, p > 0.05). These preliminary results further indicated that certain functional variants in dopamine system, such as TaqIA and TaqIB of DRD (2), were possibly involved in difference of WM in a healthy population.

[The importance of full graphic display in a graphic organizer to facilitate discourse comprehension]

In order to examine the importance of fully representing graphic information items in graphic aids to facilitate comprehension of explanatory texts, we established and randomly assigned fifty university students into the following four groups: (a) participants who study the text without the aid, (b) participants who study the text with the aid, whose literal (key words) and graphic (arrows, boxes, etc.) information items are fully displayed, (c) participants who study the text with the aid, whose graphic information items are fully displayed but whose literal information items are partially displayed, and (d) participants who study the text with the aid, whose literal and graphic information items are partially displayed. The results of two kinds of comprehension tests "textbase and situation model" revealed that groups (b) and (c) outperformed groups (a) and (d). These findings suggest that graphic aids can facilitate students' text comprehension when graphic information items are fully displayed and literal information items are displayed either fully or partially; however, the aid cannot facilitate comprehension when both literal and graphic elements are displayed partially.

Imagining predictions: mental imagery as mental emulation

We argue that the primary function of mental imagery is to allow us to generate specific predictions based upon past experience. All imagery allows us to answer 'what if' questions by making explicit and accessible the likely consequences of being in a specific situation or performing a specific action. Imagery is also characterized by its reliance on perceptual representations and activation of perceptual brain systems. We use this conception of imagery to argue that all imagery is simulation-more specifically, it is a specific type of simulation in which the mental processes that 'run' the simulation emulate those that would actually operate in the simulated scenario. This type of simulation, which we label emulation, has benefits over other types of simulations that merely mimic the content of the simulated scenario.

Grounded cognition

Grounded cognition rejects traditional views that cognition is computation on amodal symbols in a modular system, independent of the brain's modal systems for perception, action, and introspection. Instead, grounded cognition proposes that modal simulations, bodily states, and situated action underlie cognition. Accumulating behavioral and neural evidence supporting this view is reviewed from research on perception, memory, knowledge, language, thought, social cognition, and development. Theories of grounded cognition are also reviewed, as are origins of the area and common misperceptions of it. Theoretical, empirical, and methodological issues are raised whose future treatment is likely to affect the growth and impact of grounded cognition.

Mechanical reasoning by mental simulation

Recent studies have provided evidence for mental simulation as a strategy in mechanical reasoning. This type of reasoning can be dissociated from reasoning based on descriptive knowledge in that it depends on different abilities and memory stores, is expressed more easily in gesture than in language, exhibits analog properties, and can result in correct inferences in situations where people do not have correct descriptive knowledge. Although it is frequently accompanied by imagery, mental simulation is not a process of inspecting a holistic visual image in the 'mind's eye'. Mental simulations are constructed piecemeal, include representations of non-visible properties and can be used in conjunction with non-imagery processes, such as task decomposition and rule-based reasoning.

Representing a described sequence of events: a dynamic view of narrative comprehension

This study explored the representation that readers construct when advancing through the description of an unfolding occurrence. In 3 experiments, participants read narratives describing a sequence of events and at a certain moment were tested for the accessibility of an entity from a past event. Entities were less accessible when the temporal distance between that past event and the current now point in the described world was relatively long than when it was shorter. This effect occurred when temporal distance was varied in terms of the duration of an intervening event but not when it was varied in terms of a temporal shift. The results suggest that the representation constructed for the description of an unfolding occurrence mimics its temporal structure. This is consistent with a dynamic view of narrative comprehension.

Brain regions involved in simple and complex grammatical transformations

Grammatical transformation is a verbal reasoning task requiring judging the veracity of statements describing the spatial order of letter sets. We studied 18 adults with FMRI while they performed grammatical transformations of varying complexity levels (2-letter, 3-letter, and 4-letter sentences). Brain regions activated by 2-letter sentences included the visuospatial processing regions of the bilateral parietal lobes and the frontal operculum. A linear increase in sentence complexity engaged dorsolateral and ventrolateral prefrontal cortex as well as significantly increased activation within 2LTR areas. These data provide evidence that grammatical transformation reasoning relies primarily on the posterior visuospatial working memory system and need not necessarily engage the prefrontal cortex. Increasing the complexity of grammatical transformation, though, activates prefrontal cortex.

Perceptual components of situation models

These experiments examined the hypothesis that situation model construction involves perceptual processing--specifically, processing that involves visuospatial information. In this research, a dual-task paradigm was used to demonstrate that tasks that engage visuospatial processes interfere more with the generation of a situation model than tasks that are less likely to involve these processes or tasks that are verbal in nature. Using Albrecht and O'Brien's (1993) contradiction effect as evidence of situation model construction, Experiment 1 demonstrated that participants reading short texts while simultaneously holding high-imagery sentences in memory failed to show a significant contradiction effect in comparison with readers holding low-imagery sentences in memory. In Experiment 2, participants reading texts while retaining a difficult visuospatial memory load showed disrupted comprehension in comparison with readers retaining a verbal memory load.

Einflüsse der Visualisierungsform auf die Konstruktion mentaler Modelle beim Text- und Bildverstehen

Zusammenfassung. Im folgenden Beitrag wird ein integratives Modell des Text- und des Bildverstehens dargestellt. Text- und Bildverstehen werden darin als anforderungsorientierte Konstruktion von deskriptionalen und depiktionalen mentalen Repräsentationen durch Selektions-, Organisations-, Symbolverarbeitungs- und Strukturabbildungsprozesse sowie Prozesse der mentalen Modellkonstruktion und Modellinspektion angesehen. Ausgehend von diesem theoretischen Modell wurde ein Experiment durchgeführt, in dem Lernende anhand eines Hypertexts mit unterschiedlichen, jedoch informationsäquivalenten Bildern selbstgesteuert Wissen über einen komplexen Sachverhalt (Zeit- und Datumsunterschiede auf der Erde) erwerben sollten. Dabei wurde untersucht, unter welchen Bedingungen die Probanden bevorzugt auf welche Informationsquelle zurückgreifen. Außerdem wurde geprüft, ob die Form der Visualisierung Einfluß auf die Struktureigenschaften des beim Bildverstehen konstruierten Modells hat. Die Ergebnisse weisen darauf hin, daß einfachere Bilder Lernende eher zu einer oberflächlichen Verarbeitung veranlassen, bei der Text- und Bildverstehen einander teilweise ersetzen, während anspruchsvollere Bilder eher zu einer intensiveren Verarbeitung führen, bei der Text- und Bildverstehen einander wechselseitig stimulieren. Außerdem sprechen die Ergebnisse dafür, daß die Bildoberflächenstruktur zumindest teilweise auf die Struktur des mentalen Modells abgebildet wird und daß die Darbietung einer nicht aufgabenadäquaten Visualisierung mit der erforderlichen mentalen Modellkonstruktion interferieren kann. Bei der Gestaltung von Texten mit Bildern verdient demnach die Form der Visualisierung besondere Aufmerksamkeit.

Physical imagery: kinematic versus dynamic models

Physical imagery occurs when people imagine one object causing a change to a second object. To make inferences through physical imagery, people must represent information that coordinates the interactions among the imagined objects. The current research contrasts two proposals for how this coordinating information is realized in physical imagery. In the traditional kinematic formulation, imagery transformations are coordinated by geometric information in analog spatial representations. In the dynamic formulation, transformations may also be regulated by analog representations of force and resistance. Four experiments support the dynamic formulation. They show, for example, that without making changes to the spatial properties of a problem, dynamic perceptual information (e.g., torque) and beliefs about physical properties (e. g., viscosity) affect the inferences that people draw through imagery. The studies suggest that physical imagery is not so much an analog of visual perception as it is an analog of physical action. A simple model that represents force as a rate helps explain why inferences can emerge through imagined actions even though people may not know the answer explicitly. It also explains how and when perception, beliefs, and learning can influence physical imagery.