How to bootstrap a human communication system

How could we make a social robot? A virtual bargaining approach

What is required to allow an artificial agent to engage in rich, human-like interactions with people? I argue that this will require capturing the process by which humans continually create and renegotiate 'bargains' with each other. These hidden negotiations will concern topics including who should do what in a particular interaction, which actions are allowed and which are forbidden, and the momentary conventions governing communication, including language. Such bargains are far too numerous, and social interactions too rapid, for negotiation to be conducted explicitly. Moreover, the very process of communication presupposes innumerable momentary agreements concerning the meaning of communicative signals, thus raising the threat of circularity. Thus, the improvised 'social contracts' that govern our interactions must be implicit. I draw on the recent theory of virtual bargaining, according to which social partners mentally simulate a process of negotiation, to outline how these implicit agreements can be made, and note that this viewpoint raises substantial theoretical and computational challenges. Nonetheless, I suggest that these challenges must be met if we are ever to create AI systems that can work collaboratively alongside people, rather than serving primarily as valuable special-purpose computational tools. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

The Effects of Iconicity and Conventionalization on Word Order Preferences

Of the six possible orderings of the three main constituents of language (subject, verb, and object), two-SOV and SVO-are predominant cross-linguistically. Previous research using the silent gesture paradigm in which hearing participants produce or respond to gestures without speech has shown that different factors such as reversibility, salience, and animacy can affect the preferences for different orders. Here, we test whether participants' preferences for orders that are conditioned on the semantics of the event change depending on (i) the iconicity of individual gestural elements and (ii) the prior knowledge of a conventional lexicon. Our findings demonstrate the same preference for semantically conditioned word order found in previous studies, specifically that SOV and SVO are preferred differentially for different types of events. We do not find that iconicity of individual gestures affects participants' ordering preferences; however, we do find that learning a lexicon leads to a stronger preference for SVO-like orders overall. Finally, we compare our findings from English speakers, using an SVO-dominant language, with data from speakers of an SOV-dominant language, Turkish. We find that, while learning a lexicon leads to an increase in SVO preference for both sets of participants, this effect is mediated by language background and event type, suggesting that an interplay of factors together determines preferences for different ordering patterns. Taken together, our results support a view of word order as a gradient phenomenon responding to multiple biases.

Simultaneity as an Emergent Property of Efficient Communication in Language: A Comparison of Silent Gesture and Sign Language

Sign languages use multiple articulators and iconicity in the visual modality which allow linguistic units to be organized not only linearly but also simultaneously. Recent research has shown that users of an established sign language such as LIS (Italian Sign Language) use simultaneous and iconic constructions as a modality‐specific resource to achieve communicative efficiency when they are required to encode informationally rich events. However, it remains to be explored whether the use of such simultaneous and iconic constructions recruited for communicative efficiency can be employed even without a linguistic system (i.e., in silent gesture) or whether they are specific to linguistic patterning (i.e., in LIS). In the present study, we conducted the same experiment as in Slonimska et al. (2020) with 23 Italian speakers using silent gesture and compared the results of the two studies. The findings showed that while simultaneity was afforded by the visual modality to some extent, its use in silent gesture was nevertheless less frequent and qualitatively different than when used within a linguistic system. Thus, the use of simultaneous and iconic constructions for communicative efficiency constitutes an emergent property of sign languages. The present study highlights the importance of studying modality‐specific resources and their use for linguistic expression in order to promote a more thorough understanding of the language faculty and its modality‐specific adaptive capabilities.

The Seeds of the Noun–Verb Distinction in the Manual Modality: Improvisation and Interaction in the Emergence of Grammatical Categories

The noun–verb distinction has long been considered a fundamental property of human language, and has been found in some form even in the earliest stages of language emergence, including homesign and the early generations of emerging sign languages. We present two experimental studies that use silent gesture to investigate how noun–verb distinctions develop in the manual modality through two key processes: (i) improvising using novel signals by individuals, and (ii) using those signals in the interaction between communicators. We operationalise communicative interaction in two ways: a setting in which members of the dyad were in separate booths and were given a comprehension test after each stimulus vs. a more naturalistic face-to-face conversation without comprehension checks. There were few differences between the two conditions, highlighting the robustness of the paradigm. Our findings from both experiments reflect patterns found in naturally emerging sign languages. Some formal distinctions arise in the earliest stages of improvisation and do not require interaction to develop. However, the full range of formal distinctions between nouns and verbs found in naturally emerging language did not appear with either improvisation or interaction, suggesting that transmitting the language to a new generation of learners might be necessary for these properties to emerge.

Gesture is the primary modality for language creation

How language began is one of the oldest questions in science, but theories remain speculative due to a lack of direct evidence. Here, we report two experiments that generate empirical evidence to inform gesture-first and vocal-first theories of language origin; in each, we tested modern humans' ability to communicate a range of meanings (995 distinct words) using either gesture or non-linguistic vocalization. Experiment 1 is a cross-cultural study, with signal Producers sampled from Australia (n = 30, M_age = 32.63, s.d. = 12.42) and Vanuatu (n = 30, M_age = 32.40, s.d. = 11.76). Experiment 2 is a cross-experiential study in which Producers were either sighted (n = 10, M_age = 39.60, s.d. = 11.18) or severely vision-impaired (n = 10, M_age = 39.40, s.d. = 10.37). A group of undergraduate student Interpreters guessed the meaning of the signals created by the Producers (n = 140). Communication success was substantially higher in the gesture modality than the vocal modality (twice as high overall; 61.17% versus 29.04% success). This was true within cultures, across cultures and even for the signals produced by severely vision-impaired participants. The success of gesture is attributed in part to its greater universality (i.e. similarity in form across different Producers). Our results support the hypothesis that gesture is the primary modality for language creation.

Expression unleashed: The evolutionary and cognitive foundations of human communication

Human expression is open-ended, versatile, and diverse, ranging from ordinary language use to painting, from exaggerated displays of affection to micro-movements that aid coordination. Here we present and defend the claim that this expressive diversity is united by an interrelated suite of cognitive capacities, the evolved functions of which are the expression and recognition of informative intentions. We describe how evolutionary dynamics normally leash communication to narrow domains of statistical mutual benefit, and how expression is unleashed in humans. The relevant cognitive capacities are cognitive adaptations to living in a partner choice social ecology; and they are, correspondingly, part of the ordinarily developing human cognitive phenotype, emerging early and reliably in ontogeny. In other words, we identify distinctive features of our species' social ecology to explain how and why humans, and only humans, evolved the cognitive capacities that, in turn, lead to massive diversity and open-endedness in means and modes of expression. Language use is but one of these modes of expression, albeit one of manifestly high importance. We make cross-species comparisons, describe how the relevant cognitive capacities can evolve in a gradual manner, and survey how unleashed expression facilitates not only language use, but also novel behaviour in many other domains too, focusing on the examples of joint action, teaching, punishment, and art, all of which are ubiquitous in human societies but relatively rare in other species. Much of this diversity derives from graded aspects of human expression, which can be used to satisfy informative intentions in creative and new ways. We aim to help reorient cognitive pragmatics, as a phenomenon that is not a supplement to linguistic communication and on the periphery of language science, but rather the foundation of the many of the most distinctive features of human behaviour, society, and culture.

Simplification Is Not Dominant in the Evolution of Chinese Characters

Linguistic systems are hypothesised to be shaped by pressures towards communicative efficiency that drive processes of simplification. A longstanding illustration of this idea is the claim that Chinese characters have progressively simplified over time. Here we test this claim by analyzing a dataset with more than half a million images of Chinese characters spanning more than 3,000 years of recorded history. We find no consistent evidence of simplification through time, and contrary to popular belief we find that modern Chinese characters are higher in visual complexity than their earliest known counterparts. One plausible explanation for our findings is that simplicity trades off with distinctiveness, and that characters have become less simple because of pressures towards distinctiveness. Our findings are therefore compatible with functional accounts of language but highlight the diverse and sometimes counterintuitive ways in which linguistic systems are shaped by pressures for communicative efficiency.

Situating Language in the Real-World: The Role of Multimodal Iconicity and Indexicality

In the last decade, a growing body of work has convincingly demonstrated that languages embed a certain degree of non-arbitrariness (mostly in the form of iconicity, namely the presence of imagistic links between linguistic form and meaning). Most of this previous work has been limited to assessing the degree (and role) of non-arbitrariness in the speech (for spoken languages) or manual components of signs (for sign languages). When approached in this way, non-arbitrariness is acknowledged but still considered to have little presence and purpose, showing a diachronic movement towards more arbitrary forms. However, this perspective is limited as it does not take into account the situated nature of language use in face-to-face interactions, where language comprises categorical components of speech and signs, but also multimodal cues such as prosody, gestures, eye gaze etc. We review work concerning the role of context-dependent iconic and indexical cues in language acquisition and processing to demonstrate the pervasiveness of non-arbitrary multimodal cues in language use and we discuss their function. We then move to argue that the online omnipresence of multimodal non-arbitrary cues supports children and adults in dynamically developing situational models.

A Cross-Sectional Test of Sign Creation by Children in the Gesture and Vocal Modalities

Naturalistic studies show that children can create language-like communication systems in the absence of conventional language. However, experimental evidence is mixed. We address this discrepancy using an experimental paradigm that simulates naturalistic sign creation. Specifically, we tested if a sample of 6- to 12-year-old children (52 girls and 56 boys drawn from an urban, predominantly white population in Western Australia) can comprehend and create novel gestural and vocal signs. Experiment 1 tested children's ability to comprehend novel signs. Experiment 2 tested children's ability to create novel signs. Results show that children can comprehend and create gestural and vocal signs, that communication is more successful in the gesture modality, and that older children outperform younger children.

Novel vocalizations are understood across cultures

Linguistic communication requires speakers to mutually agree on the meanings of words, but how does such a system first get off the ground? One solution is to rely on iconic gestures: visual signs whose form directly resembles or otherwise cues their meaning without any previously established correspondence. However, it is debated whether vocalizations could have played a similar role. We report the first extensive cross-cultural study investigating whether people from diverse linguistic backgrounds can understand novel vocalizations for a range of meanings. In two comprehension experiments, we tested whether vocalizations produced by English speakers could be understood by listeners from 28 languages from 12 language families. Listeners from each language were more accurate than chance at guessing the intended referent of the vocalizations for each of the meanings tested. Our findings challenge the often-cited idea that vocalizations have limited potential for iconic representation, demonstrating that in the absence of words people can use vocalizations to communicate a variety of meanings.

Pantomimic fossils in modern human communication

Bodily mimesis, the capacity to use the body representationally, was one of the key innovations that allowed early humans to go beyond the 'baseline' of generalized ape communication and cognition. We argue that the original human-specific communication afforded by bodily mimesis was based on signs that involve three entities: an expression that represents an object (i.e. communicated content) for an interpreter. We further propose that the core component of this communication, pantomime, was able to transmit referential information that was not limited to select semantic domains or the 'here-and-now', by means of motivated-most importantly iconic-signs. Pressures for expressivity and economy then led to conventionalization of signs and a growth of linguistic characteristics: semiotic systematicity and combinatorial expression. Despite these developments, both naturalistic and experimental data suggest that the system of pantomime did not disappear and is actively used by modern humans. Its contemporary manifestations, or pantomimic fossils, emerge when language cannot be used, for instance when people do not share a common language, or in situations where the use of (spoken) language is difficult, impossible or forbidden. Under such circumstances, people bootstrap communication by means of pantomime and, when these circumstances persist, newly emergent pantomimic communication becomes increasingly language-like. This article is part of the theme issue 'Reconstructing prehistoric languages'.

Commitment Ladder in the Relationship between Service Providers and Customers as Added Value in Sustainable Services Development

The socioeconomic sphere and the relationships in which commitment occurs are important elements in the development of sustainable services. The study reported in this article identifies the elements that influence the development of the relationship between service providers and their customers and proposes a model that describes the state of the relationship between service providers and customers in terms of symmetrical commitment of both parties. Qualitative research including interviews with experts and case studies was completed, resulting in a ‘ladder of commitment’ model that identifies distinct commitment levels and specific commitment factors functioning at each of those levels. In practice, the proposed model makes it possible to assess the state of customer and provider commitment, identifying commitment deficits on the part of the customer or service provider. This article can provide practical added value for managers who are looking for ways to analyze customer commitment in order to develop sustainable services.

Persuasive conversation as a new form of communication in Homo sapiens

The aim of this paper is twofold: to propose that conversation is the distinctive feature of Homo sapiens' communication; and to show that the emergence of modern language is tied to the transition from pantomime to verbal and grammatically complex forms of narrative. It is suggested that (animal and human) communication is a form of persuasion and that storytelling was the best tool developed by humans to convince others. In the early stage of communication, archaic hominins used forms of pantomimic storytelling to persuade others. Although pantomime is a powerful tool for persuasive communication, it is proposed that it is not an effective tool for persuasive conversation: conversation is characterized by a form of reciprocal persuasion among peers; instead, pantomime has a mainly asymmetrical character. The selective pressure towards persuasive reciprocity of the conversational level is the evolutionary reason that allowed the transition from pantomime to grammatically complex codes in H. sapiens, which favoured the evolution of speech. This article is part of the theme issue 'Reconstructing prehistoric languages'.

Innovation and enculturation in child communication: a cross-sectional study

How can people achieve successful communication when using novel signs? Previous studies show that iconic signs (i.e. signs that directly resemble their referent) enhance communication success. In this paper, we test if enculturated signs (i.e. signs informed by interlocutors' shared culture) also enhance communication success. Children, who have spent less time in their linguistic community, have less cultural knowledge to inform their sign innovation. A natural prediction is that younger children's signs will be less enculturated, more diverse and less successful compared with older children and adults. We examined sign innovation in children aged between 6 and 12 years (N = 54) and adults (N = 18). Sign enculturation, diversity and iconicity were rated. As predicted, younger children innovated less enculturated and more diverse signs, and communicated less successfully than older children and adults. Sign enculturation and iconicity uniquely contributed to communication success. This is the first study to demonstrate that enculturated signs enhance communication.

Interpreting Silent Gesture: Cognitive Biases and Rational Inference in Emerging Language Systems

Natural languages make prolific use of conventional constituent-ordering patterns to indicate "who did what to whom," yet the mechanisms through which these regularities arise are not well understood. A series of recent experiments demonstrates that, when prompted to express meanings through silent gesture, people bypass native language conventions, revealing apparent biases underpinning word order usage, based on the semantic properties of the information to be conveyed. We extend the scope of these studies by focusing, experimentally and computationally, on the interpretation of silent gesture. We show cross-linguistic experimental evidence that people use variability in constituent order as a cue to obtain different interpretations. To illuminate the computational principles that govern interpretation of non-conventional communication, we derive a Bayesian model of interpretation via biased inductive inference and estimate these biases from the experimental data. Our analyses suggest people's interpretations balance the ambiguity that is characteristic of emerging language systems, with ordering preferences that are skewed and asymmetric, but defeasible.

Inferring Behavior From Partial Social Information Plays Little or No Role in the Cultural Transmission of Adaptive Traits

Many human cultural traits become increasingly beneficial as they are repeatedly transmitted, thanks to an accumulation of modifications made by successive generations. But how do later generations typically avoid modifications which revert traits to less beneficial forms already sampled and rejected by earlier generations? And how can later generations do so without direct exposure to their predecessors' behavior? One possibility is that learners are sensitive to cues of non-random production in others' behavior, and that particular variants (e.g., those containing structural regularities unlikely to occur spontaneously) have been produced deliberately and with some effort. If this non-random behavior is attributed to an informed strategy, then the learner may infer that apparent avoidance of certain possibilities indicates that these have already been sampled and rejected. This could potentially prevent performance plateaus resulting from learners modifying inherited behaviors randomly. We test this hypothesis in four experiments in which participants, either individually or in interacting dyads, attempt to locate rewards in a search grid, guided by partial information about another individual's experience of the task. We find that in some contexts, valid inferences about another's behavior can be made from partial information, and these inferences can be used in a way which facilitates trait adaptation. However, the benefit of these inferences appears to be limited, and in many contexts-including some which have the potential to make inferring the experience of another individual easier-there appears to be no benefit at all. We suggest that inferring previous behavior from partial social information plays a minimal role in the adaptation of cultural traits.

A World Unto Itself: Human Communication as Active Inference

Recent theoretical work in developmental psychology suggests that humans are predisposed to align their mental states with those of other individuals. One way this manifests is in cooperative communication; that is, intentional communication aimed at aligning individuals' mental states with respect to events in their shared environment. This idea has received strong empirical support. The purpose of this paper is to extend this account by proposing an integrative model of the biobehavioral dynamics of cooperative communication. Our formulation is based on active inference. Active inference suggests that action-perception cycles operate to minimize uncertainty and optimize an individual's internal model of the world. We propose that humans are characterized by an evolved adaptive prior belief that their mental states are aligned with, or similar to, those of conspecifics (i.e., that 'we are the same sort of creature, inhabiting the same sort of niche'). The use of cooperative communication emerges as the principal means to gather evidence for this belief, allowing for the development of a shared narrative that is used to disambiguate interactants' (hidden and inferred) mental states. Thus, by using cooperative communication, individuals effectively attune to a hermeneutic niche composed, in part, of others' mental states; and, reciprocally, attune the niche to their own ends via epistemic niche construction. This means that niche construction enables features of the niche to encode precise, reliable cues about the deontic or shared value of certain action policies (e.g., the utility of using communicative constructions to disambiguate mental states, given expectations about shared prior beliefs). In turn, the alignment of mental states (prior beliefs) enables the emergence of a novel, contextualizing scale of cultural dynamics that encompasses the actions and mental states of the ensemble of interactants and their shared environment. The dynamics of this contextualizing layer of cultural organization feedback, across scales, to constrain the variability of the prior expectations of the individuals who constitute it. Our theory additionally builds upon the active inference literature by introducing a new set of neurobiologically plausible computational hypotheses for cooperative communication. We conclude with directions for future research.

Pictionary-Style Word Guessing on Hand-Drawn Object Sketches: Dataset, Analysis and Deep Network Models

The ability of intelligent agents to play games in human-like fashion is popularly considered a benchmark of progress in Artificial Intelligence. In our work, we introduce the first computational model aimed at Pictionary, the popular word-guessing social game. We first introduce Sketch-QA, a guessing task. Styled after Pictionary, Sketch-QA uses incrementally accumulated sketch stroke sequences as visual data. Sketch-QA involves asking a fixed question ("What object is being drawn?") and gathering open-ended guess-words from human guessers. We analyze the resulting dataset and present many interesting findings therein. To mimic Pictionary-style guessing, we propose a deep neural model which generates guess-words in response to temporally evolving human-drawn object sketches. Our model even makes human-like mistakes while guessing, thus amplifying the human mimicry factor. We evaluate our model on the large-scale guess-word dataset generated via Sketch-QA task and compare with various baselines. We also conduct a Visual Turing Test to obtain human impressions of the guess-words generated by humans and our model. Experimental results demonstrate the promise of our approach for Pictionary and similarly themed games.

Multimodality and the origin of a novel communication system in face-to-face interaction

Face-to-face communication is multimodal at its core: it consists of a combination of vocal and visual signalling. However, current evidence suggests that, in the absence of an established communication system, visual signalling, especially in the form of visible gesture, is a more powerful form of communication than vocalization and therefore likely to have played a primary role in the emergence of human language. This argument is based on experimental evidence of how vocal and visual modalities (i.e. gesture) are employed to communicate about familiar concepts when participants cannot use their existing languages. To investigate this further, we introduce an experiment where pairs of participants performed a referential communication task in which they described unfamiliar stimuli in order to reduce reliance on conventional signals. Visual and auditory stimuli were described in three conditions: using visible gestures only, using non-linguistic vocalizations only and given the option to use both (multimodal communication). The results suggest that even in the absence of conventional signals, gesture is a more powerful mode of communication compared with vocalization, but that there are also advantages to multimodality compared to using gesture alone. Participants with an option to produce multimodal signals had comparable accuracy to those using only gesture, but gained an efficiency advantage. The analysis of the interactions between participants showed that interactants developed novel communication systems for unfamiliar stimuli by deploying different modalities flexibly to suit their needs and by taking advantage of multimodality when required.

Evolving artificial sign languages in the lab: From improvised gesture to systematic sign

Recent work on emerging sign languages provides evidence for how key properties of linguistic systems are created. Here we use laboratory experiments to investigate the contribution of two specific mechanisms-interaction and transmission-to the emergence of a manual communication system in silent gesturers. We show that the combined effects of these mechanisms, rather than either alone, maintain communicative efficiency, and lead to a gradual increase of regularity and systematic structure. The gestures initially produced by participants are unsystematic and resemble pantomime, but come to develop key language-like properties similar to those documented in newly emerging sign systems.

Multimodal communication and language origins: integrating gestures and vocalizations

The presence of divergent and independent research traditions in the gestural and vocal domains of primate communication has resulted in major discrepancies in the definition and operationalization of cognitive concepts. However, in recent years, accumulating evidence from behavioural and neurobiological research has shown that both human and non-human primate communication is inherently multimodal. It is therefore timely to integrate the study of gestural and vocal communication. Herein, we review evidence demonstrating that there is no clear difference between primate gestures and vocalizations in the extent to which they show evidence for the presence of key language properties: intentionality, reference, iconicity and turn-taking. We also find high overlap in the neurobiological mechanisms producing primate gestures and vocalizations, as well as in ontogenetic flexibility. These findings confirm that human language had multimodal origins. Nonetheless, we note that in great apes, gestures seem to fulfil a carrying (i.e. predominantly informative) role in close-range communication, whereas the opposite holds for face-to-face interactions of humans. This suggests an evolutionary shift in the carrying role from the gestural to the vocal stream, and we explore this transition in the carrying modality. Finally, we suggest that future studies should focus on the links between complex communication, sociality and cooperative tendency to strengthen the study of language origins.

Repeated imitation makes human vocalizations more word-like

People have long pondered the evolution of language and the origin of words. Here, we investigate how conventional spoken words might emerge from imitations of environmental sounds. Does the repeated imitation of an environmental sound gradually give rise to more word-like forms? In what ways do these forms resemble the original sounds that motivated them (i.e. exhibit iconicity)? Participants played a version of the children's game 'Telephone'. The first generation of participants imitated recognizable environmental sounds (e.g. glass breaking, water splashing). Subsequent generations imitated the previous generation of imitations for a maximum of eight generations. The results showed that the imitations became more stable and word-like, and later imitations were easier to learn as category labels. At the same time, even after eight generations, both spoken imitations and their written transcriptions could be matched above chance to the category of environmental sound that motivated them. These results show how repeated imitation can create progressively more word-like forms while continuing to retain a resemblance to the original sound that motivated them, and speak to the possible role of human vocal imitation in explaining the origins of at least some spoken words.

The emergence of systematicity: How environmental and communicative factors shape a novel communication system

Where does linguistic structure come from? We suggest that systematicity in language evolves adaptively in response to environmental and contextual affordances associated with the practice of communication itself. In two experiments, we used a silent gesture referential game paradigm to investigate environmental and social factors promoting the propagation of systematicity in a novel communication system. We found that structure in the emerging communication systems evolve contingent on structural properties of the environment. More specifically, interlocutors spontaneously relied on structural features of the referent stimuli they communicated about to motivate systematic aspects of the evolving communication system even when idiosyncratic iconic strategies were equally afforded. Furthermore, we found systematicity to be promoted by the nature of the referent environment. When the referent environment was open and unstable, analytic systematic strategies were more likely to emerge compared to stimulus environments with a closed set of referents. Lastly, we found that displacement of communication promoted systematicity. That is, when interlocutors had to communicate about items not immediately present in the moment of communication, they were more likely to evolve systematic solutions, supposedly due to working memory advantages. Together, our findings provide experimental evidence for the idea that linguistic structure evolves adaptively from contextually situated language use.

Language is more abstract than you think, or, why aren't languages more iconic?

How abstract is language? We show that abstractness pervades every corner of language, going far beyond the usual examples of freedom and justice In the light of the ubiquity of abstract words, the need to understand where abstract meanings come from becomes ever more acute. We argue that the best source of knowledge about abstract meanings may be language itself. We then consider a seemingly unrelated question: Why isn't language more iconic? Iconicity-a resemblance between the form of words and their meanings-can be immensely useful in language learning and communication. Languages could be much more iconic than they currently are. So why aren't they? We suggest that one reason is that iconicity is inimical to abstraction because iconic forms are too connected to specific contexts and sensory depictions. Form-meaning arbitrariness may allow language to better convey abstract meanings.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.

Universal Principles of Human Communication: Preliminary Evidence From a Cross-cultural Communication Game

The present study points to several potentially universal principles of human communication. Pairs of participants, sampled from culturally and linguistically distinct societies (Western and Japanese, N = 108: 16 Western-Western, 15 Japanese-Japanese and 23 Western-Japanese dyads), played a dyadic communication game in which they tried to communicate a range of experimenter-specified items to a partner by drawing, but without speaking or using letters or numbers. This paradigm forced participants to create a novel communication system. A range of similar communication behaviors were observed among the within-culture groups (Western-Western and Japanese-Japanese) and the across-culture group (Western-Japanese): They (a) used iconic signs to bootstrap successful communication, (b) addressed breakdowns in communication using other-initiated repairs, (c) simplified their communication behavior over repeated social interactions, and (d) aligned their communication behavior over repeated social interactions. While the across-culture Western-Japanese dyads found the task more challenging, and cultural differences in communication behavior were observed, the same basic findings applied across all groups. Our findings, which rely on two distinct cultural and linguistic groups, offer preliminary evidence for several universal principles of human communication.

How to Create Shared Symbols

Human cognition and behavior are dominated by symbol use. This paper examines the social learning strategies that give rise to symbolic communication. Experiment 1 contrasts an individual-level account, based on observational learning and cognitive bias, with an inter-individual account, based on social coordinative learning. Participants played a referential communication game in which they tried to communicate a range of recurring meanings to a partner by drawing, but without using their conventional language. Individual-level learning, via observation and cognitive bias, was sufficient to produce signs that became increasingly effective, efficient, and shared over games. However, breaking a referential precedent eliminated these benefits. The most effective, most efficient, and most shared signs arose when participants could directly interact with their partner, indicating that social coordinative learning is important to the creation of shared symbols. Experiment 2 investigated the contribution of two distinct aspects of social interaction: behavior alignment and concurrent partner feedback. Each played a complementary role in the creation of shared symbols: Behavior alignment primarily drove communication effectiveness, and partner feedback primarily drove the efficiency of the evolved signs. In conclusion, inter-individual social coordinative learning is important to the evolution of effective, efficient, and shared symbols.

People Can Create Iconic Vocalizations to Communicate Various Meanings to Naïve Listeners

The innovation of iconic gestures is essential to establishing the vocabularies of signed languages, but might iconicity also play a role in the origin of spoken words? Can people create novel vocalizations that are comprehensible to naïve listeners without prior convention? We launched a contest in which participants submitted non-linguistic vocalizations for 30 meanings spanning actions, humans, animals, inanimate objects, properties, quantifiers and demonstratives. The winner was determined by the ability of naïve listeners to infer the meanings of the vocalizations. We report a series of experiments and analyses that evaluated the vocalizations for: (1) comprehensibility to naïve listeners; (2) the degree to which they were iconic; (3) agreement between producers and listeners in iconicity; and (4) whether iconicity helps listeners learn the vocalizations as category labels. The results show contestants were able to create successful iconic vocalizations for most of the meanings, which were largely comprehensible to naïve listeners, and easier to learn as category labels. These findings demonstrate how iconic vocalizations can enable interlocutors to establish understanding in the absence of conventions. They suggest that, prior to the advent of full-blown spoken languages, people could have used iconic vocalizations to ground a spoken vocabulary with considerable semantic breadth.

Rising tones and rustling noises: Metaphors in gestural depictions of sounds

Communicating an auditory experience with words is a difficult task and, in consequence, people often rely on imitative non-verbal vocalizations and gestures. This work explored the combination of such vocalizations and gestures to communicate auditory sensations and representations elicited by non-vocal everyday sounds. Whereas our previous studies have analyzed vocal imitations, the present research focused on gestural depictions of sounds. To this end, two studies investigated the combination of gestures and non-verbal vocalizations. A first, observational study examined a set of vocal and gestural imitations of recordings of sounds representative of a typical everyday environment (ecological sounds) with manual annotations. A second, experimental study used non-ecological sounds whose parameters had been specifically designed to elicit the behaviors highlighted in the observational study, and used quantitative measures and inferential statistics. The results showed that these depicting gestures are based on systematic analogies between a referent sound, as interpreted by a receiver, and the visual aspects of the gestures: auditory-visual metaphors. The results also suggested a different role for vocalizations and gestures. Whereas the vocalizations reproduce all features of the referent sounds as faithfully as vocally possible, the gestures focus on one salient feature with metaphors based on auditory-visual correspondences. Both studies highlighted two metaphors consistently shared across participants: the spatial metaphor of pitch (mapping different pitches to different positions on the vertical dimension), and the rustling metaphor of random fluctuations (rapidly shaking of hands and fingers). We interpret these metaphors as the result of two kinds of representations elicited by sounds: auditory sensations (pitch and loudness) mapped to spatial position, and causal representations of the sound sources (e.g. rain drops, rustling leaves) pantomimed and embodied by the participants' gestures.

Minimal Requirements for the Emergence of Learned Signaling

The emergence of signaling systems has been observed in numerous experimental and real‐world contexts, but there is no consensus on which (if any) shared mechanisms underlie such phenomena. A number of explanatory mechanisms have been proposed within several disciplines, all of which have been instantiated as credible working models. However, they are usually framed as being mutually incompatible. Using an exemplar‐based framework, we replicate these models in a minimal configuration which allows us to directly compare them. This reveals that the development of optimal signaling is driven by similar mechanisms in each model, which leads us to propose three requirements for the emergence of conventional signaling. These are the creation and transmission of referential information, a systemic bias against ambiguity, and finally some form of information loss. Considering this, we then discuss some implications for theoretical and experimental approaches to the emergence of learned communication.

Iconicity and the Emergence of Combinatorial Structure in Language

In language, recombination of a discrete set of meaningless building blocks forms an unlimited set of possible utterances. How such combinatorial structure emerged in the evolution of human language is increasingly being studied. It has been shown that it can emerge when languages culturally evolve and adapt to human cognitive biases. How the emergence of combinatorial structure interacts with the existence of holistic iconic form-meaning mappings in a language is still unknown. The experiment presented in this paper studies the role of iconicity and human cognitive learning biases in the emergence of combinatorial structure in artificial whistled languages. Participants learned and reproduced whistled words for novel objects with the use of a slide whistle. Their reproductions were used as input for the next participant, to create transmission chains and simulate cultural transmission. Two conditions were studied: one in which the persistence of iconic form-meaning mappings was possible and one in which this was experimentally made impossible. In both conditions, cultural transmission caused the whistled languages to become more learnable and more structured, but this process was slightly delayed in the first condition. Our findings help to gain insight into when and how words may lose their iconic origins when they become part of an organized linguistic system.

Environmental constraints shaping constituent order in emerging communication systems: Structural iconicity, interactive alignment and conventionalization

Where does linguistic structure come from? Recent gesture elicitation studies have indicated that constituent order (corresponding to for instance subject-verb-object, or SVO in English) may be heavily influenced by human cognitive biases constraining gesture production and transmission. Here we explore the alternative hypothesis that syntactic patterns are motivated by multiple environmental and social-interactional constraints that are external to the cognitive domain. In three experiments, we systematically investigate different motivations for structure in the gestural communication of simple transitive events. The first experiment indicates that, if participants communicate about different types of events, manipulation events (e.g. someone throwing a cake) and construction events (e.g. someone baking a cake), they spontaneously and systematically produce different constituent orders, SOV and SVO respectively, thus following the principle of structural iconicity. The second experiment shows that participants' choice of constituent order is also reliably influenced by social-interactional forces of interactive alignment, that is, the tendency to re-use an interlocutor's previous choice of constituent order, thus potentially overriding affordances for iconicity. Lastly, the third experiment finds that the relative frequency distribution of referent event types motivates the stabilization and conventionalization of a single constituent order for the communication of different types of events. Together, our results demonstrate that constituent order in emerging gestural communication systems is shaped and stabilized in response to multiple external environmental and social factors: structural iconicity, interactive alignment and distributional frequency.

Iconicity can ground the creation of vocal symbols

Studies of gestural communication systems find that they originate from spontaneously created iconic gestures. Yet, we know little about how people create vocal communication systems, and many have suggested that vocalizations do not afford iconicity beyond trivial instances of onomatopoeia. It is unknown whether people can generate vocal communication systems through a process of iconic creation similar to gestural systems. Here, we examine the creation and development of a rudimentary vocal symbol system in a laboratory setting. Pairs of participants generated novel vocalizations for 18 different meanings in an iterative 'vocal' charades communication game. The communicators quickly converged on stable vocalizations, and naive listeners could correctly infer their meanings in subsequent playback experiments. People's ability to guess the meanings of these novel vocalizations was predicted by how close the vocalization was to an iconic 'meaning template' we derived from the production data. These results strongly suggest that the meaningfulness of these vocalizations derived from iconicity. Our findings illuminate a mechanism by which iconicity can ground the creation of vocal symbols, analogous to the function of iconicity in gestural communication systems.

Production and comprehension show divergent constituent order preferences: Evidence from elicited pantomime

All natural languages develop devices to communicate who did what to whom. Elicited pantomime provides one model for studying this process, by providing a window into how humans (hearing non-signers) behave in a natural communicative modality (silent gesture) without established conventions from a grammar. Most studies in this paradigm focus on production, although they sometimes make assumptions about how comprehenders would likely behave. Here, we directly assess how naïve speakers of English (Experiments 1 & 2), Korean (Experiment 1), and Turkish (Experiment 2) comprehend pantomimed descriptions of transitive events, which are either semantically reversible (Experiments 1 & 2) or not (Experiment 2). Contrary to previous assumptions, we find no evidence that Person-Person-Action sequences are ambiguous to comprehenders, who simply adopt an agent-first parsing heuristic for all constituent orders. We do find that Person-Action-Person sequences yield the most consistent interpretations, even in native speakers of SOV languages. The full range of behavior in both production and comprehension provides counter-evidence to the notion that producers' utterances are motivated by the needs of comprehenders. Instead, we argue that production and comprehension are subject to different sets of cognitive pressures, and that the dynamic interaction between these competing pressures can help explain synchronic and diachronic constituent order phenomena in natural human languages, both signed and spoken.

How communication changes when we cannot mime the world: Experimental evidence for the effect of iconicity on combinatoriality

Communication systems are exposed to two different pressures: a pressure for transmission efficiency, such that messages are simple to produce and perceive, and a pressure for referential efficiency, such that messages are easy to understand with their intended meaning. A solution to the first pressure is combinatoriality--the recombination of a few basic meaningless forms to express an infinite number of meanings. A solution to the second is iconicity--the use of forms that resemble what they refer to. These two solutions appear to be incompatible with each other, as iconic forms are ill-suited for use as meaningless combinatorial units. Furthermore, in the early stages of a communication system, when basic referential forms are in the process of being established, the pressure for referential efficiency is likely to be particularly strong, which may lead it to trump the pressure for transmission efficiency. This means that, where iconicity is available as a strategy, it is likely to impede the emergence of combinatoriality. Although this hypothesis seems consistent with some observations of natural language, it was unclear until recently how it could be soundly tested. This has changed thanks to the development of a line of research, known as Experimental Semiotics, in which participants construct novel communication systems in the laboratory using an unfamiliar medium. We conducted an Experimental Semiotic study in which we manipulated the opportunity for iconicity by varying the kind of referents to be communicated, while keeping the communication medium constant. We then measured the combinatoriality and transmission efficiency of the communication systems. We found that, where iconicity was available, it provided scaffolding for the construction of communication systems and was overwhelmingly adopted. Where it was not available, however, the resulting communication systems were more combinatorial and their forms more efficient to produce. This study enriches our understanding of the fundamental design principles of human communication and contributes tools to enrich it further.

The bridge of iconicity: from a world of experience to the experience of language

Iconicity, a resemblance between properties of linguistic form (both in spoken and signed languages) and meaning, has traditionally been considered to be a marginal, irrelevant phenomenon for our understanding of language processing, development and evolution. Rather, the arbitrary and symbolic nature of language has long been taken as a design feature of the human linguistic system. In this paper, we propose an alternative framework in which iconicity in face-to-face communication (spoken and signed) is a powerful vehicle for bridging between language and human sensori-motor experience, and, as such, iconicity provides a key to understanding language evolution, development and processing. In language evolution, iconicity might have played a key role in establishing displacement (the ability of language to refer beyond what is immediately present), which is core to what language does; in ontogenesis, iconicity might play a critical role in supporting referentiality (learning to map linguistic labels to objects, events, etc., in the world), which is core to vocabulary development. Finally, in language processing, iconicity could provide a mechanism to account for how language comes to be embodied (grounded in our sensory and motor systems), which is core to meaningful communication.

Cultural selection drives the evolution of human communication systems

Human communication systems evolve culturally, but the evolutionary mechanisms that drive this evolution are not well understood. Against a baseline that communication variants spread in a population following neutral evolutionary dynamics (also known as drift models), we tested the role of two cultural selection models: coordination- and content-biased. We constructed a parametrized mixed probabilistic model of the spread of communicative variants in four 8-person laboratory micro-societies engaged in a simple communication game. We found that selectionist models, working in combination, explain the majority of the empirical data. The best-fitting parameter setting includes an egocentric bias and a content bias, suggesting that participants retained their own previously used communicative variants unless they encountered a superior (content-biased) variant, in which case it was adopted. This novel pattern of results suggests that (i) a theory of the cultural evolution of human communication systems must integrate selectionist models and (ii) human communication systems are functionally adaptive complex systems.

From iconic handshapes to grammatical contrasts: longitudinal evidence from a child homesigner

Many sign languages display crosslinguistic consistencies in the use of two iconic aspects of handshape, handshape type and finger group complexity. Handshape type is used systematically in form-meaning pairings (morphology): Handling handshapes (Handling-HSs), representing how objects are handled, tend to be used to express events with an agent ("hand-as-hand" iconicity), and Object handshapes (Object-HSs), representing an object's size/shape, are used more often to express events without an agent ("hand-as-object" iconicity). Second, in the distribution of meaningless properties of form (morphophonology), Object-HSs display higher finger group complexity than Handling-HSs. Some adult homesigners, who have not acquired a signed or spoken language and instead use a self-generated gesture system, exhibit these two properties as well. This study illuminates the development over time of both phenomena for one child homesigner, "Julio," age 7;4 (years; months) to 12;8. We elicited descriptions of events with and without agents to determine whether morphophonology and morphosyntax can develop without linguistic input during childhood, and whether these structures develop together or independently. Within the time period studied: (1) Julio used handshape type differently in his responses to vignettes with and without an agent; however, he did not exhibit the same pattern that was found previously in signers, adult homesigners, or gesturers: while he was highly likely to use a Handling-HS for events with an agent (82%), he was less likely to use an Object-HS for non-agentive events (49%); i.e., his productions were heavily biased toward Handling-HSs; (2) Julio exhibited higher finger group complexity in Object- than in Handling-HSs, as in the sign language and adult homesigner groups previously studied; and (3) these two dimensions of language developed independently, with phonological structure showing a sign language-like pattern at an earlier age than morphosyntactic structure. We conclude that iconicity alone is not sufficient to explain the development of linguistic structure in homesign systems. Linguistic input is not required for some aspects of phonological structure to emerge in childhood, and while linguistic input is not required for morphology either, it takes time to emerge in homesign.

Creating a communication system from scratch: gesture beats vocalization hands down

How does modality affect people's ability to create a communication system from scratch? The present study experimentally tests this question by having pairs of participants communicate a range of pre-specified items (emotions, actions, objects) over a series of trials to a partner using either non-linguistic vocalization, gesture or a combination of the two. Gesture-alone outperformed vocalization-alone, both in terms of successful communication and in terms of the creation of an inventory of sign-meaning mappings shared within a dyad (i.e., sign alignment). Combining vocalization with gesture did not improve performance beyond gesture-alone. In fact, for action items, gesture-alone was a more successful means of communication than the combined modalities. When people do not share a system for communication they can quickly create one, and gesture is the best means of doing so.

Phonological iconicity

The arbitrariness of the linguistic sign is a fundamental assumption in modern linguistic theory. In recent years, however, a growing amount of research has investigated the nature of non-arbitrary relations between linguistic sounds and semantics. This review aims at illustrating the amount of findings obtained so far and to organize and evaluate different lines of research dedicated to the issue of phonological iconicity. In particular, we summarize findings on the processing of onomatopoetic expressions, ideophones, and phonaesthemes, relations between syntactic classes and phonology, as well as sound-shape and sound-affect correspondences at the level of phonemic contrasts. Many of these findings have been obtained across a range of different languages suggesting an internal relation between sublexical units and attributes as a potentially universal pattern.

The cultural evolution of human communication systems in different sized populations: usability trumps learnability

This study examines the intergenerational transfer of human communication systems. It tests if human communication systems evolve to be easy to learn or easy to use (or both), and how population size affects learnability and usability. Using an experimental-semiotic task, we find that human communication systems evolve to be easier to use (production efficiency and reproduction fidelity), but harder to learn (identification accuracy) for a second generation of naïve participants. Thus, usability trumps learnability. In addition, the communication systems that evolve in larger populations exhibit distinct advantages over those that evolve in smaller populations: the learnability loss (from the Initial signs) is more muted and the usability benefits are more pronounced. The usability benefits for human communication systems that evolve in a small and large population is explained through guided variation reducing sign complexity. The enhanced performance of the communication systems that evolve in larger populations is explained by the operation of a content bias acting on the larger pool of competing signs. The content bias selects for information-efficient iconic signs that aid learnability and enhance usability.