The problem of serial order in behavior: Lashley’s legacy

How language production shapes language form and comprehension

Language production processes can provide insight into how language comprehension works and language typology-why languages tend to have certain characteristics more often than others. Drawing on work in memory retrieval, motor planning, and serial order in action planning, the Production-Distribution-Comprehension (PDC) account links work in the fields of language production, typology, and comprehension: (1) faced with substantial computational burdens of planning and producing utterances, language producers implicitly follow three biases in utterance planning that promote word order choices that reduce these burdens, thereby improving production fluency. (2) These choices, repeated over many utterances and individuals, shape the distributions of utterance forms in language. The claim that language form stems in large degree from producers' attempts to mitigate utterance planning difficulty is contrasted with alternative accounts in which form is driven by language use more broadly, language acquisition processes, or producers' attempts to create language forms that are easily understood by comprehenders. (3) Language perceivers implicitly learn the statistical regularities in their linguistic input, and they use this prior experience to guide comprehension of subsequent language. In particular, they learn to predict the sequential structure of linguistic signals, based on the statistics of previously-encountered input. Thus, key aspects of comprehension behavior are tied to lexico-syntactic statistics in the language, which in turn derive from utterance planning biases promoting production of comparatively easy utterance forms over more difficult ones. This approach contrasts with classic theories in which comprehension behaviors are attributed to innate design features of the language comprehension system and associated working memory. The PDC instead links basic features of comprehension to a different source: production processes that shape language form.

Prospective and retrospective effects in a virtual pointing task

Over two decades ago prospective and retrospective effects of posture selection in a sequential task were described for the first time. Since then, both effects have been reproduced in a number of reaching studies. We asked (1) whether retrospective effects would also be found in a sequential pointing task and (2) whether pro/retrospective effects of posture selection would transfer to the end-effector position in the absence of haptic feedback. To this end, we created a sequential, perceptual-motor task in a virtual environment. Participants had to point to a row of targets in the frontal plane in sequential order. In a control experiment, physical targets were placed at the same locations. Results showed that kinematic parameters were similar in the virtual and real environment. Retrospective effects of posture/position were found in neither environment, indicating that pointing movements require lower cognitive planning costs than reaching movements. Prospective effects of posture were found both in the virtual and real environment. Prospective effects of position, on the other hand, were present in the virtual but not in the real environment, indicating that the absence of haptic feedback may result in unconscious shifts of the end-effector position.

Using goal- and grip-related information for understanding the correctness of other's actions: an ERP study

Detecting errors in other’s actions is of pivotal importance for joint action, competitive behavior and observational learning. Although many studies have focused on the neural mechanisms involved in detecting low-level errors, relatively little is known about error-detection in everyday situations. The present study aimed to identify the functional and neural mechanisms whereby we understand the correctness of other’s actions involving well-known objects (e.g. pouring coffee in a cup). Participants observed action sequences in which the correctness of the object grasped and the grip applied to a pair of objects were independently manipulated. Observation of object violations (e.g. grasping the empty cup instead of the coffee pot) resulted in a stronger P3-effect than observation of grip errors (e.g. grasping the coffee pot at the upper part instead of the handle), likely reflecting a reorienting response, directing attention to the relevant location. Following the P3-effect, a parietal slow wave positivity was observed that persisted for grip-errors, likely reflecting the detection of an incorrect hand-object interaction. These findings provide new insight in the functional significance of the neurophysiological markers associated with the observation of incorrect actions and suggest that the P3-effect and the subsequent parietal slow wave positivity may reflect the detection of errors at different levels in the action hierarchy. Thereby this study elucidates the cognitive processes that support the detection of action violations in the selection of objects and grips.

Identification of intermittent control in man and machine

Regulation by negative feedback is fundamental to engineering and biological processes. Biological regulation is usually explained using continuous feedback models from both classical and modern control theory. An alternative control paradigm, intermittent control, has also been suggested as a model for biological control systems, particularly those involving the central nervous system. However, at present, there is no identification method explicitly formulated to distinguish intermittent from continuous control; here, we present such a method. The identification experiment uses a special paired-step set-point sequence. The corresponding data analysis use a conventional ARMA model to relate a theoretically derived equivalent set-point to control signal; the novelty lies in sequentially and iteratively adjusting the timing of the steps of this equivalent set-point to optimize the linear time-invariant fit. The method was verified using realistic simulation data and was found to robustly distinguish not only between continuous and intermittent control but also between event-driven intermittent and clock-driven intermittent control. When applied to human pursuit tracking, event-driven intermittent control was identified, with an intermittent interval of 260-310 ms (n = 6, p < 0.05). This new identification method is applicable for machine and biological applications.

The end-state comfort effect facilitates joint action

Motor experts can accurately predict the future actions of others by observing their movements. This report describes three experiments that investigate such predictions in everyday object manipulations and test whether these predictions facilitate responses to the actions of others. Observing video excerpts showing an actor reaching for a vertically mounted dial, participants in Experiment 1 needed to predict how the actor would rotate it. Their predictions were specific to the direction and extent of the dial rotation and improved proportionate to the length of the video clip shown. Testing whether such predictions facilitate responses, in the subsequent experiments responders had to undo an actor's actions, back-rotating a dial (Exp 2) and a bar (Exp 3). The responders' actions were initiated faster when the actors' movements obeyed the so-called end-state comfort principle than when they did not. Our experiments show that humans exploit the end-state comfort effect to tweak their predictions of the future actions of others. The results moreover suggest that the precision of these predictions is mediated by perceptual learning rather than by motor simulation.

Neuronal chains for actions in the parietal lobe: a computational model

The inferior part of the parietal lobe (IPL) is known to play a very important role in sensorimotor integration. Neurons in this region code goal-related motor acts performed with the mouth, with the hand and with the arm. It has been demonstrated that most IPL motor neurons coding a specific motor act (e.g., grasping) show markedly different activation patterns according to the final goal of the action sequence in which the act is embedded (grasping for eating or grasping for placing). Some of these neurons (parietal mirror neurons) show a similar selectivity also during the observation of the same action sequences when executed by others. Thus, it appears that the neuronal response occurring during the execution and the observation of a specific grasping act codes not only the executed motor act, but also the agent's final goal (intention).In this work we present a biologically inspired neural network architecture that models mechanisms of motor sequences execution and recognition. In this network, pools composed of motor and mirror neurons that encode motor acts of a sequence are arranged in form of action goal-specific neuronal chains. The execution and the recognition of actions is achieved through the propagation of activity bursts along specific chains modulated by visual and somatosensory inputs.The implemented spiking neuron network is able to reproduce the results found in neurophysiological recordings of parietal neurons during task performance and provides a biologically plausible implementation of the action selection and recognition process.Finally, the present paper proposes a mechanism for the formation of new neural chains by linking together in a sequential manner neurons that represent subsequent motor acts, thus producing goal-directed sequences.

Imitation of hand and tool actions is effector-independent

Following the theoretical notion that tools often extend one’s body, in the present study, we investigated whether imitation of hand or tool actions is modulated by effector-specific information. Subjects performed grasping actions toward an object with either a handheld tool or their right hand. Actions were initiated in response to pictures representing a grip at an object that could be congruent or incongruent with the required action (grip-type congruency). Importantly, actions could be cued by means of a tool cue, a hand cue, and a symbolic cue (effector-type congruency). For both hand and tool actions, an action congruency effect was observed, reflected in faster reaction times if the observed grip type was congruent with the required movement. However, neither hand actions nor tool actions were differentially affected by the effector represented in the picture (i.e., when performing a tool action, the action congruency effect was similar for tool cues and hand cues). This finding suggests that imitation of hand and tool actions is effector-independent and thereby supports generalist rather than specialist theories of imitation.

Relation of ordinal position signals to the expectation of reward and passage of time in four areas of the macaque frontal cortex

Neurons in several areas of the monkey frontal cortex exhibit rank selectivity, firing differentially as a function of the stage attained during the performance of a serial order task. The activity of these neurons is commonly thought to represent ordinal position within the trial. However, they might also be sensitive to factors correlated with ordinal position including time elapsed during the trial (which is greater for each successive stage) and the degree of anticipation of reward (which probably increases at each successive stage). To compare the influences of these factors, we monitored neuronal activity in the supplementary motor area (SMA), presupplementary motor area (pre-SMA), supplementary eye field (SEF), and dorsolateral prefrontal cortex during the performance of a serial order task (requiring a series of saccades in three specified directions), a variable reward task (in which a cue displayed early in the trial indicated whether the reward received at the end of the trial would be large or small), and a long delay task (in which the monkey had simply to maintain fixation during a period of time approximating the duration of an average trial in the serial order task). We found that rank signals were partially correlated with sensitivity to elapsed time and anticipated reward. The connection to elapsed time was strongest in the pre-SMA. The connection to anticipated reward was most pronounced in the SMA and SEF. However, critically, these factors could not fully explain rank selectivity in any of the areas tested.

Psychologically distinct classes of motor behavior inferred from individual differences: evidence from a sequential stacking task

A number of studies have demonstrated regularities in how individuals select and perform single object manipulations, but little work has been concerned with the manipulation of multiple objects. To this end, the authors asked participants to stack a set of linearly spaced containers onto various goal locations. Our aim was to determine whether participants adopted specific strategies to complete this task. We focused on whether the distance between the objects, the goal location of the objects, or both, determined the classes of movement sequences that individuals used to perform the task. The results showed that some individuals tended to use one hand for lifting and releasing the containers whereas other individuals tended to use both hands for lifting and releasing the containers. Those participants who tended to use one hand varied which hand was used according to the goal location of the containers but not the distance between containers. The emergence of these individual differences provides a new basis for inferring psychologically distinct classes of motor behavior.

A network centered on ventral premotor cortex exerts both facilitatory and inhibitory control over primary motor cortex during action reprogramming

Ventral premotor cortex (PMv) is widely accepted to exert an important influence over primary motor cortex (M1) when hand movements are made. Although study of these interactions has typically focused on their excitatory nature, given its strong connections with both ventral and opercular frontal regions, one feature of the influence of PMv over M1 may be inhibitory. Paired-pulse transcranial magnetic stimulation (ppTMS) was used to examine functional interactions between human PMv and M1 during the selection and reprogramming of a naturalistic goal-directed action. One of two cylinders was illuminated on each trial. It was then grasped and picked up. On some trials, however, subjects had to reprogram the action as the illuminated cylinder was switched off and the other illuminated simultaneously with reach initiation. At a neurophysiological level, the PMv paired-pulse effect (PPE) on M1 corticospinal activity was facilitatory after the initial target presentation and during movement initiation. When reprogramming was required, however, the PPE became strongly inhibitory. This context-dependent change from facilitation to inhibition occurred within 75 ms of the change of target. Behaviorally, PMv-M1 ppTMS disrupted reprogramming. Diffusion-weighted magnetic resonance image scans were taken of each subject. Intersubject differences in the facilitation-inhibition contrast of PMv-M1 interactions were correlated with fractional anisotropy of white-matter in ventral prefrontal, premotor, and intraparietal brain areas. These results suggest that a network of brain areas centered on PMv inhibits M1 corticospinal activity associated with undesired movements when action plans change.

Mirror systems

Mirror neurons are a class of visuomotor neurons, discovered in the monkey premotor cortex and in an anatomically connected area of the inferior parietal lobule, that activate both during action execution and action observation. They constitute a circuit dedicated to match actions made by others with the internal motor representations of the observer. It has been proposed that this matching system enables individuals to understand others' behavior and motor intentions. Here we will describe the main features of mirror neurons in monkeys. Then we will present evidence of the presence of a mirror system in humans and of its involvement in several social-cognitive functions, such as imitation, intention, and emotion understanding. This system may have several implications at a cognitive level and could be linked to specific social deficits in humans such as autism. Recent investigations addressed the issue of the plasticity of the mirror neuron system in both monkeys and humans, suggesting also their possible use in rehabilitation. WIREs Cogn Sci 2011 2 22-38 DOI: 10.1002/wcs.89 For further resources related to this article, please visit the WIREs website.

The Development of End-State Comfort Planning in Preschool Children

We investigated the development of the end-state comfort effect in young children. Fifty-one children from three age-groups (3, 4, and 5 years old) participated in the study. They performed the dowel placing task, which required them to reach for a horizontal dowel and to insert one of its ends into a target disk. Depending on which end was instructed, end-state comfort could be reached by picking up the dowel either with an overhand or with an underhand grip. All children reached for the dowel with an overhand grasp when this resulted in a comfortable end-state (i.e., thumb-up posture). A different pattern emerged when an underhand grip had to be selected. Here, 18% of the 3-year-olds, 45% of the 4-year-olds, and 67% of the 5-year-olds used an underhand grip and finished the action comfortably. For the first time, these results show a distinct pattern of gradual improvement in children’s sensitivity to reach end-state comfort across three age-groups.

Cellular dynamical mechanisms for encoding the time and place of events along spatiotemporal trajectories in episodic memory

Understanding the mechanisms of episodic memory requires linking behavioral data and lesion effects to data on the dynamics of cellular membrane potentials and population interactions within brain regions. Linking behavior to specific membrane channels and neurochemicals has implications for therapeutic applications. Lesions of the hippocampus, entorhinal cortex and subcortical nuclei impair episodic memory function in humans and animals, and unit recording data from these regions in behaving animals indicate episodic memory processes. Intracellular recording in these regions demonstrates specific cellular properties including resonance, membrane potential oscillations and bistable persistent spiking that could underlie the encoding and retrieval of episodic trajectories. A model presented here shows how intrinsic dynamical properties of neurons could mediate the encoding of episodic memories as complex spatiotemporal trajectories. The dynamics of neurons allow encoding and retrieval of unique episodic trajectories in multiple continuous dimensions including temporal intervals, personal location, the spatial coordinates and sensory features of perceived objects and generated actions, and associations between these elements. The model also addresses how cellular dynamics could underlie unit firing data suggesting mechanisms for coding continuous dimensions of space, time, sensation and action.

Generating variable birdsong syllable sequences with branching chain networks in avian premotor nucleus HVC

Songs of songbird species such as Bengalese finch consist of sequences of syllables. While syllables are temporally stereotypical, syllable sequences can vary and follow complex, probabilistic transition rules. Recent experiments and computational models suggest that a syllable is encoded in a chain network of projection neurons in premotor nucleus HVC (proper name). Precisely timed spikes propagate along the chain, driving vocalization of the syllable through downstream nuclei. However, the neural basis of the probabilistic transitions between the syllables is not understood. Here we propose that variable syllable sequences are generated through spike propagations in a network in HVC in which the syllable-encoding chain networks are connected into a branching chain pattern. The neurons mutually inhibit each other through the inhibitory HVC interneurons, and are driven by external inputs from nuclei upstream of HVC. At a branching point that connects the final group of a chain to the first groups of several chains, the spike activity selects one branch to continue the propagation. The selection is probabilistic, and is due to the winner-take-all mechanism mediated by the inhibition and noise. The transitions between the chains are Markovian. If the same syllable can be driven by multiple chains, the generated syllable sequences are statistically described by partially observable Markov models. We suggest that the syntax of birdsong syllable sequences is embedded in the connection patterns of HVC projection neurons.

The effect of sublexical and lexical frequency on speech production: An fMRI investigation

There is no consensus regarding the fundamental phonetic units that underlie speech production. There is, however, general agreement that the frequency of occurrence of these units is a significant factor. Investigators often use the effects of manipulating frequency to support the importance of particular units. Studies of pseudoword production have been used to show the importance of sublexical units, such as initial syllables, phonemes, and biphones. However, it is not clear that these units play the same role when the production of pseudowords is compared to the production of real words. In this study, participants overtly repeated real and pseudowords that were similar for length, complexity, and initial syllable frequency while undergoing functional magnetic resonance imaging. Compared to real words, production of pseudowords produced greater activation in much of the speech production network, including bilateral inferior frontal cortex, precentral gyri and supplementary motor areas and left superior temporal cortex and anterior insula. Only middle right frontal gyrus showed greater activation for real words than for pseudowords. Compared to a no-speech control condition, production of pseudowords or real words resulted in activation of all of the areas shown to comprise the speech production network. Our data, in conjunction with previous studies, suggest that the unit that is identified as the basic unit of speech production is influenced by the nature of the speech that is being studied, i.e., real words as compared to other real words, pseudowords as compared to other pseudowords, or real words as compared to pseudowords.

Moving and memorizing: motor planning modulates the recency effect in serial and free recall

Motor planning has generally been studied in situations where participants carry out physical actions without a particular purpose. Yet in everyday life physical actions are usually carried out for higher-order goals. We asked whether two previously discovered motor planning phenomena--the end-state comfort effect and motor hysteresis--would hold up if the actions were carried out in the service of higher-order goals. The higher-order goal we chose to study was memorization. By focusing on memorization, we asked not only how and whether motor planning is affected by the need to memorize, but also how memory performance might depend on the cognitive demands of motor planning. We asked university-student participants to retrieve cups from a column of drawers and memorize as many letters as possible from the inside of the cups. The drawers were opened either in a random order (Experiment 1) or in a regular order (Experiments 2 and 3). The end-state comfort effect and motor hysteresis were replicated in these conditions, indicating that the effects hold up when physical actions are carried out for the sake of a higher-order goal. Surprisingly, one of the most reliable effects in memory research was eliminated, namely, the tendency of recent items to be recalled better than earlier items--the recency effect. This outcome was not an artifact of memory being uniformly poor, because the tendency of initial items to be recalled better than later items--the primacy effect--was obtained. Elimination of the recency effect was not due to the requirement that participants recall items in their correct order, for the recency effect was also eliminated when the items could be recalled in any order (Experiment 3). These and other aspects of the results support recent claims for tighter links between perceptual-motor control and intellectual (symbolic) processing than have been assumed in the past.

Increased dependence of action selection on recent motor history in Parkinson's disease

It is well known that the basal ganglia are involved in switching between movement sequences. Here we test the hypothesis that this contribution is an instance of a more general role of the basal ganglia in selecting actions that deviate from the context defined by the recent motor history, even when there is no sequential structure to learn or implement. We investigated the effect of striatal dopamine depletion [in Parkinson's disease (PD)] on the ability to switch between independent action plans. PD patients with markedly lateralized signs performed a hand laterality judgment task that involved action selection of their most and least affected hand. Trials where patients selected the same (repeat) or the alternative (switch) hand as in a previous trial were compared, and this was done separately for the most and least affected hand. Behaviorally, PD patients showed switch-costs that were specific to the most affected hand and that increased with disease severity. Functional magnetic resonance imaging (fMRI) showed that this behavioral effect was related to the state of the frontostriatal system: as disease severity increased, contributions of the basal ganglia to the selection process and their effective connectivity with the medial frontal cortex (MFC) decreased, whereas involvement of the MFC increased. We conclude that the basal ganglia are important for rapidly switching toward novel motor plans even when there is no sequential structure to learn or implement. The enhanced MFC activity may result either from reduced focusing abilities of the basal ganglia or from compensatory processes.

Visuospatial working memory capacity predicts the organization of acquired explicit motor sequences

Studies have suggested that cognitive processes such as working memory and temporal control contribute to motor sequence learning. These processes engage overlapping brain regions with sequence learning, but concrete evidence has been lacking. In this study, we determined whether limits in visuospatial working memory capacity and temporal control abilities affect the temporal organization of explicitly acquired motor sequences. Participants performed an explicit sequence learning task, a visuospatial working memory task, and a continuous tapping timing task. We found that visuospatial working memory capacity, but not the CV from the timing task, correlated with the rate of motor sequence learning and the chunking pattern observed in the learned sequence. These results show that individual differences in short-term visuospatial working memory capacity, but not temporal control, predict the temporal structure of explicitly acquired motor sequences.

Movement-specific repetition suppression in ventral and dorsal premotor cortex during action observation

There are several models of premotor cortex contributions to sensorimotor behavior. For instance, the ventral premotor cortex (PMv) appears to be involved in processing visuospatial object properties for grasping, whereas the dorsal premotor cortex (PMd) is involved in using arbitrary rules to guide advance motor planning. These models have focused on individual movements. Here, we examine the premotor responses evoked during the processing of individual movements functionally embedded in an action. We tested whether processing hand-object interactions and action end states would differentially engage PMv and PMd. We used a repetition suppression (RS)-functional magnetic resonance imaging paradigm in which we independently manipulated the observed grip, the end position of the object (independent of its spatial location), and the hand trajectory. By comparing novel and repeated trials for each of these action components, we could isolate RS effects specific to each of them. Repeating the grasp component attenuated activity in right PMv, whereas repeating the end state of the action reduced blood oxygen level-dependent activity in the left PMd. These results suggest that PMv is involved in controlling the kinematic means of an appropriate hand-object interaction, whereas PMd is focused on specifying the desired end state of an action.

The cognitive nature of action - functional links between cognitive psychology, movement science, and robotics

This paper examines the cognitive architecture of human action, showing how it is organized over several levels and how it is built up. Basic action concepts (BACs) are identified as major building blocks on a representation level. These BACs are cognitive tools for mastering the functional demands of movement tasks. Results from different lines of research showed that not only the structure formation of mental representations in long-term memory but also chunk formation in working memory are built up on BACs and relate systematically to movement structures. It is concluded that such movement representations might provide the basis for action implementation and action control in skilled voluntary movements in the form of cognitive reference structures. To simulate action implementation we discuss challenges and issues that arise when we try to replicate complex movement abilities in robots. Among the key issues to be addressed is the question how structured representations can arise during skill acquisition and how the underlying processes can be understood sufficiently succinctly to replicate them on robot platforms. Working towards this goal, we translate our findings in studies of motor control in humans into models that can guide the implementation of cognitive robot architectures. Focusing on the issue of manual action control, we illustrate some results in the context of grasping with a five-fingered anthropomorphic robot hand.

Hand path priming in manual obstacle avoidance: rapid decay of dorsal stream information

The dorsal, action-related, visual stream has been thought to have little or no memory. This hypothesis has seemed credible because functions related to the dorsal stream have been generally unsusceptible to priming from previous experience. Tests of this claim have yielded inconsistent results, however. We argue that these inconsistencies may be due to methodological differences in the time between primes and test stimuli. In this study we sought to clarify the effect of time between primes and test stimuli by having participants complete a visually guided manual obstacle avoidance task with varying times between trials. Consistent with a previous study using this task, we found that hand path curvature depended on the presence or absence of an obstacle in the previous trial. This hand path priming effect decayed quickly as the time between trials increased, and was almost, though not entirely, eliminated when 1000 ms separated successive trials. The results are consistent with the hypothesis that the dorsal stream can be primed but that this priming attenuates rapidly. We suggest that this outcome may indicate that the period over which the dorsal stream retains information may be related to the sequential statistics of action.