Space-time mapping relationships in sensorimotor communication during asymmetric joint action

Background

Sensorimotor communication is frequently observed in complex joint actions and social interactions. However, it remains challenging to explore the cognitive foundations behind sensorimotor communication.

Methods

The present study extends previous research by introducing a single-person baseline condition and formulates two distinct categories of asymmetric joint action tasks: distance tasks and orientation tasks. This research investigates the action performance of 65 participants under various experimental conditions utilizing a 2 (cooperative intention: Coop, No-coop) × 2 (task characteristic: distance, orientation) × 4 (target: T1, T2, T3, T4) repeated-measures experimental design to investigate the cognitive mechanisms underlying sensorimotor communication between individuals.

Results

The results showed that (1) target key dwell time, motion time, total motion time, and maximum motion height in the Coop condition are more than in the No-coop condition. (2) In the distance task without cooperative intention, the dwell time of T4 is smaller than T1, T2, T3, and its variability of T1, T2, T3, and T4 were no different. In the distance task with cooperative intention, the dwell time and its variability of T1, T2, T3, and T4 displayed an increasing trend. (3) In the orientation task without cooperative intention, the dwell time of T1 is smaller than T2, T3, T4, and variability of the target keys T1, T2, T3, and T4 had no difference. In the orientation task with cooperative intention, the dwell time and variability of the target keys T1, T2, T3, and T4 had increasing trends.

Conclusions

Those findings underscore the importance of cooperative intention for sensorimotor communication. In the distance task with cooperative intention, message senders establish a mapping relationship characterized by "near-small, far-large" between the task distance and the individual's action characteristics through sensorimotor experience. In the orientation task with cooperative intention, message senders combined sensorimotor experience and verbal metaphors to establish a mapping relationship between task orientation and action characteristics, following the sequence of "left-up, right-up, left-down, right-down" to transmit the message to others.

Flexible Cultural Learning Through Action Coordination

The cultural transmission of technical know-how has proven vital to the success of our species. The broad diversity of learning contexts and social configurations, as well as the various kinds of coordinated interactions they involve, speaks to our capacity to flexibly adapt to and succeed in transmitting vital knowledge in various learning contexts. Although often recognized by ethnographers, the flexibility of cultural learning has so far received little attention in terms of cognitive mechanisms. We argue that a key feature of the flexibility of cultural learning is that both the models and learners recruit cognitive mechanisms of action coordination to modulate their behavior contingently on the behavior of their partner, generating a process of mutual adaptation supporting the successful transmission of technical skills in diverse and fluctuating learning environments. We propose that the study of cultural learning would benefit from the experimental methods, results, and insights of joint-action research and, complementarily, that the field of joint-action research could expand its scope by integrating a learning and cultural dimension. Bringing these two fields of research together promises to enrich our understanding of cultural learning, its contextual flexibility, and joint action coordination.

A systematic review of handover actions in human dyads

Introduction

Handover actions are joint actions in which an object is passed from one actor to another. In order to carry out a smooth handover action, precise coordination of both actors' movements is of critical importance. This requires the synchronization of both the kinematics of the reaching movement and the grip forces of the two actors during the interaction. Psychologists, for example, may be interested in studying handover actions in order to identify the cognitive mechanisms underlying the interaction of two partners. In addition, robotic engineers may utilize insights from sensorimotor information processing in human handover as models for the design controllers in robots in hybrid (human-robot) interaction scenarios. To date, there is little knowledge transfer between researchers in different disciplines and no common framework or language for the study of handover actions.

Methods

For this reason, we systematically reviewed the literature on human-human handover actions in which at least one of the two types of behavioral data, kinematics or grip force, was measured.

Results

Nine relevant studies were identified. The different methodologies and results of the individual studies are here described and contextualized.

Discussion

Based on these results, a common framework is suggested that, provides a distinct and straightforward language and systematics for use in future studies. We suggest to term the actors as giver and receiver, as well as to subdivide the whole action into four phases: (1) Reach and grasp, (2) object transport, (3) object transfer, and (4) end of handover to comprehensively and clearly describe the handover action. The framework aims to foster the necessary exchange between different scientific disciplines to promote research on handover actions. Overall, the results support the assumption that givers adapt their executions according to the receiver's intentions, that the start of the release of the object is processed feedforward and that the release process is feedback-controlled in the transfer phase. We identified the action planning of the receiver as a research gap.

Understanding others' distal goals from proximal communicative actions

Many social interactions require individuals to coordinate their actions and to inform each other about their goals. Often these goals concern an immediate (i.e., proximal) action, as when people give each other a brief handshake, but they sometimes also refer to a future (i.e. distal) action, as when football players perform a passing sequence. The present study investigates whether observers can derive information about such distal goals by relying on kinematic modulations of an actor's instrumental actions. In Experiment 1 participants were presented with animations of a box being moved at different velocities towards an apparent endpoint. The distal goal, however, was for the object to be moved past this endpoint, to one of two occluded target locations. Participants then selected the location which they considered the likely distal goal of the action. As predicted, participants were able to detect differences in movement velocity and, based on these differences, systematically mapped the movements to the two distal goal locations. Adding a distal goal led to more variation in the way participants mapped the observed movements onto different target locations. The results of Experiments 2 and 3 indicated that this cannot be explained by difficulties in perceptual discrimination. Rather, the increased variability likely reflects differences in interpreting the underlying connection between proximal communicative actions and distal goals. The present findings extend previous research on sensorimotor communication by demonstrating that communicative action modulations are not restricted to predicting proximal goals but can also be used to infer more distal goals.

Communication and action predictability: two complementary strategies for successful cooperation

Making one's actions predictable and communicating what one intends to do are two strategies to achieve interpersonal coordination. It is less clear whether these two strategies are mutually exclusive or whether they can be used in parallel. Here, we asked how the availability of communication channels affects the use of strategy to make one's actions predictable. In three experiments, we investigated how people reach joint decisions if they are not allowed to communicate at all (Experiment 1), allowed minimal reciprocal communication (Experiment 2), or allowed to use the full range of conventional communication (Experiment 3). We found that when participants were not allowed to communicate, coordination was achieved by increasing action predictability. When conventional communication was allowed, there were no attempts to increase action predictability. In the minimal reciprocal communication condition, successful pairs both increased action predictability and established a communication system. Overall, this study demonstrates that people are able to flexibly adapt to coordination challenges during joint decision making and that communication reduces behavioural constraints on joint action coordination.

Social Action Effects: Representing Predicted Partner Responses in Social Interactions

The sociomotor framework outlines a possible role of social action effects on human action control, suggesting that anticipated partner reactions are a major cue to represent, select, and initiate own body movements. Here, we review studies that elucidate the actual content of social action representations and that explore factors that can distinguish action control processes involving social and inanimate action effects. Specifically, we address two hypotheses on how the social context can influence effect-based action control: first, by providing unique social features such as body-related, anatomical codes, and second, by orienting attention towards any relevant feature dimensions of the action effects. The reviewed empirical work presents a surprisingly mixed picture: while there is indirect evidence for both accounts, previous studies that directly addressed the anatomical account showed no signs of the involvement of genuinely social features in sociomotor action control. Furthermore, several studies show evidence against the differentiation of social and non-social action effect processing, portraying sociomotor action representations as remarkably non-social. A focus on enhancing the social experience in future studies should, therefore, complement the current database to establish whether such settings give rise to the hypothesized influence of social context.

Instantaneous systems of communicative conventions through virtual bargaining

People can instantaneously create novel conventions that link individual communicative signals to meanings, both in experiments and everyday communication. Yet a basic principle of natural communication is that the meaning of a signal typically contrasts with the meanings of alternative signals that were available but not chosen. That is, communicative conventions typically form a system, rather than consisting of isolated signal-meaning pairs. Accordingly, creating a novel convention linking a specific signal and meaning seems to require creating a system of conventions linking possible signals to possible meanings, of which the signal-meaning pair to be communicated is merely a sub-case. If so, people will not link signals and meanings in isolation; signal-meaning pairings will depend on alternative signals and meanings. We outline and address theoretical challenges concerning how instantaneous conventions can be formed, building on prior work on "virtual bargaining," in which people simulate the results of a process of negotiation concerning which convention, or system of conventions, to choose. Moreover, we demonstrate empirically that instantaneous systems of conventions can flexibly be created in a 'minimal' experimental paradigm. Experimental evidence from 158 people playing a novel signaling game shows that modifying both the set of signals, and the set of meanings, can indeed systematically modify the signal-meaning mappings that people may instantaneously construct. While consistent with the virtual bargaining account, accounting for these results may be challenging for some accounts of pragmatic inference.

Hierarchical Integration of Communicative and Spatial Perspective‐Taking Demands in Sensorimotor Control of Referential Pointing

Recognized as a simple communicative behavior, referential pointing is cognitively complex because it invites a communicator to consider an addressee's knowledge. Although we know referential pointing is affected by addressees’ physical location, it remains unclear whether and how communicators’ inferences about addressees’ mental representation of the interaction space influence sensorimotor control of referential pointing. The communicative perspective‐taking task requires a communicator to point at one out of multiple referents either to instruct an addressee which one should be selected (communicative, COM) or to predict which one the addressee will select (non‐communicative, NCOM), based on either which referents can be seen (Level‐1 perspective‐taking, PT1) or how the referents were perceived (Level‐2 perspective‐taking, PT2) by the addressee. Communicators took longer to initiate the movements in PT2 than PT1 trials, and they held their pointing fingers for longer at the referent in COM than NCOM trials. The novel findings of this study pertain to trajectory control of the pointing movements. Increasing both communicative and perspective‐taking demands led to longer pointing trajectories, with an under‐additive interaction between those two experimental factors. This finding suggests that participants generate communicative behaviors that are as informative as required rather than overly exaggerated displays, by integrating communicative and perspective‐taking information hierarchically during sensorimotor control. This observation has consequences for models of human communication. It implies that the format of communicative and perspective‐taking knowledge needs to be commensurate with the movement dynamics controlled by the sensorimotor system.

Your action does matter to me: Examining the role of the co-actor's action-effects in resolving the self-other discrimination problem

Sharing a task with another person can introduce the need to discriminate representations that refer to our own action from that of the other person's. The current understanding is that information about the stimulus event drives the self-other discrimination process, as it promotes (via the reactivation of feature codes) the representation that encodes the corresponding action. However, this mechanistic explanation relies on experimental situations in which stimulus event information (e.g., spatial location) is always and directly available. Thus, it remains unclear whether and how we could successfully discriminate between self- and other related action representations in the absence of such information. The present study addressed this unanswered question using a novel joint Simon task-based paradigm. We report the results of three experiments in which we manipulated the availability of stimulus event information into the contralateral space. Our findings demonstrate that participants are able to compensate for the absence of stimulus event information by relying on temporal features of their co-actor's action-effects (Experiment 1). Even more surprising was that participants continued to monitor the temporal features of their co-actor's actions even when given a verbal signal by their co-actor (Experiments 2a), or full access to the common workspace (Experiment 2b). Our results are strong evidence that the representation of actions is not purely stimulus driven. They suggest that the temporal dimension of the other person's actions is able to drive the self-other discrimination process, in the same way as other perceptual dimensions and feature codes that are shared with the stimulus event.

Coordinating With a Robot Partner Affects Neural Processing Related to Action Monitoring

Robots start to play a role in our social landscape, and they are progressively becoming responsive, both physically and socially. It begs the question of how humans react to and interact with robots in a coordinated manner and what the neural underpinnings of such behavior are. This exploratory study aims to understand the differences in human-human and human-robot interactions at a behavioral level and from a neurophysiological perspective. For this purpose, we adapted a collaborative dynamical paradigm from the literature. We asked 12 participants to hold two corners of a tablet while collaboratively guiding a ball around a circular track either with another participant or a robot. In irregular intervals, the ball was perturbed outward creating an artificial error in the behavior, which required corrective measures to return to the circular track again. Concurrently, we recorded electroencephalography (EEG). In the behavioral data, we found an increased velocity and positional error of the ball from the track in the human-human condition vs. human-robot condition. For the EEG data, we computed event-related potentials. We found a significant difference between human and robot partners driven by significant clusters at fronto-central electrodes. The amplitudes were stronger with a robot partner, suggesting a different neural processing. All in all, our exploratory study suggests that coordinating with robots affects action monitoring related processing. In the investigated paradigm, human participants treat errors during human-robot interaction differently from those made during interactions with other humans. These results can improve communication between humans and robot with the use of neural activity in real-time.

Progress in Joint-Action Research

Humans have a striking ability to coordinate their actions with each other to achieve joint goals. The tight interpersonal coordination that characterizes joint actions is achieved through processes that help with preparing for joint action as well as processes that are active while joint actions are being performed. To prepare for joint action, partners form representations of each other’s actions and tasks and the relation between them. This enables them to predict each other’s upcoming actions, which, in turn, facilitates coordination. While performing joint actions, partners’ coordination is maintained by (a) monitoring whether individual and joint outcomes correspond to what was planned, (b) predicting partners’ action parameters on the basis of familiarity with their individual actions, (c) communicating task-relevant information unknown to partners in an action-based fashion, and (d) relying on coupling of predictions through dense perceptual-information flow between coactors. The next challenge for the field of joint action is to generate an integrated perspective that links coordination mechanisms to normative, evolutionary, and communicative frameworks.

Crossmodal correspondences as common ground for joint action

When performing joint actions, people rely on common ground - shared information that provides the required basis for mutual understanding. Common ground can be based on people's interaction history or on knowledge and expectations people share, e.g., because they belong to the same culture or social class. Here, we suggest that people rely on yet another form of common ground, one that originates in their similarities in multisensory processing. Specifically, we focus on 'crossmodal correspondences' - nonarbitrary associations that people make between stimulus features in different sensory modalities, e.g., between stimuli in the auditory and the visual modality such as high-pitched sounds and small objects. Going beyond previous research that focused on investigating crossmodal correspondences in individuals, we propose that people can use these correspondences for communicating and coordinating with others. Initial support for our proposal comes from a communication game played in a public space (an art gallery) by pairs of visitors. We observed that pairs created nonverbal communication systems by spontaneously relying on 'crossmodal common ground'. Based on these results, we conclude that crossmodal correspondences not only occur within individuals but that they can also be actively used in joint action to facilitate the coordination between individuals.

The communicative advantage: how kinematic signaling supports semantic comprehension

Humans are unique in their ability to communicate information through representational gestures which visually simulate an action (eg. moving hands as if opening a jar). Previous research indicates that the intention to communicate modulates the kinematics (e.g., velocity, size) of such gestures. If and how this modulation influences addressees' comprehension of gestures have not been investigated. Here we ask whether communicative kinematic modulation enhances semantic comprehension (i.e., identification) of gestures. We additionally investigate whether any comprehension advantage is due to enhanced early identification or late identification. Participants (n = 20) watched videos of representational gestures produced in a more- (n = 60) or less-communicative (n = 60) context and performed a forced-choice recognition task. We tested the isolated role of kinematics by removing visibility of actor's faces in Experiment I, and by reducing the stimuli to stick-light figures in Experiment II. Three video lengths were used to disentangle early identification from late identification. Accuracy and response time quantified main effects. Kinematic modulation was tested for correlations with task performance. We found higher gesture identification performance in more- compared to less-communicative gestures. However, early identification was only enhanced within a full visual context, while late identification occurred even when viewing isolated kinematics. Additionally, temporally segmented acts with more post-stroke holds were associated with higher accuracy. Our results demonstrate that communicative signaling, interacting with other visual cues, generally supports gesture identification, while kinematic modulation specifically enhances late identification in the absence of other cues. Results provide insights into mutual understanding processes as well as creating artificial communicative agents.

Informing, Coordinating, and Performing: A Perspective on Functions of Sensorimotor Communication

Sensorimotor communication is a form of communication instantiated through body movements that are guided by both instrumental, goal-directed intentions and communicative, social intentions. Depending on the social interaction context, sensorimotor communication can serve different functions. This article aims to disentangle three of these functions: (a) an informing function of body movements, to highlight action intentions for an observer; (b) a coordinating function of body movements, to facilitate real-time action prediction in joint action; and (c) a performing function of body movements, to elicit emotional or aesthetic experiences in an audience. We provide examples of research addressing these different functions as well as some influencing factors, relating to individual differences, task characteristics, and situational demands. The article concludes by discussing the benefits of a closer dialog between separate lines of research on sensorimotor communication across different social contexts.

Motion tracking of parents' infant- versus adult-directed actions reveals general and action-specific modulations

Parents tend to modulate their movements when demonstrating actions to their infants. Thus far, these modulations have primarily been quantified by human raters and for entire interactions, thereby possibly overlooking the intricacy of such demonstrations. Using optical motion tracking, the precise modulations of parents' infant-directed actions were quantified and compared to adult-directed actions and between action types. Parents demonstrated four novel objects to their 14-month-old infants and adult confederates. Each object required a specific action to produce a unique effect (e.g. rattling). Parents were asked to demonstrate an object at least once before passing it to their demonstration partner, and they were subsequently free to exchange the object as often as desired. Infants' success at producing the objects' action-effects was coded during the demonstration session and their memory of the action-effects was tested after a several-minute delay. Indicating general modulations across actions, parents repeated demonstrations more often, performed the actions in closer proximity and demonstrated action-effects for longer when interacting with their infant compared to the adults. Meanwhile, modulations of movement size and velocity were specific to certain action-effect pairs. Furthermore, a 'just right' modulation of proximity was detected, since infants' learning, memory, and parents' prior evaluations of their infants' motor abilities, were related to demonstrations that were performed neither too far from nor too close to the infants. Together, these findings indicate that infant-directed action modulations are not solely overall exaggerations but are dependent upon the characteristics of the to-be learned actions, their effects, and the infant learners.

When Height Carries Weight: Communicating Hidden Object Properties for Joint Action

In the absence of pre-established communicative conventions, people create novel communication systems to successfully coordinate their actions toward a joint goal. In this study, we address two types of such novel communication systems: sensorimotor communication, where the kinematics of instrumental actions are systematically modulated, versus symbolic communication. We ask which of the two systems co-actors preferentially create when aiming to communicate about hidden object properties such as weight. The results of three experiments consistently show that actors who knew the weight of an object transmitted this weight information to their uninformed co-actors by systematically modulating their instrumental actions, grasping objects of particular weights at particular heights. This preference for sensorimotor communication was reduced in a fourth experiment where co-actors could communicate with weight-related symbols. Our findings demonstrate that the use of sensorimotor communication extends beyond the communication of spatial locations to non-spatial, hidden object properties.