Sound-Action Symbolism

The size and shape of sound: The role of articulation and acoustics in iconicity and crossmodal correspondencesa)

Onomatopoeias like hiss and peep are iconic because their forms resemble their meanings. Iconicity can also involve forms and meanings in different modalities, such as when people match the nonce words bouba and kiki to round and angular objects, and mil and mal to small and large ones, also known as "sound symbolism." This paper focuses on what specific analogies motivate such correspondences in spoken language: do people associate shapes and size with how phonemes sound (auditory), or how they are produced (articulatory)? Based on a synthesis of empirical evidence probing the cognitive mechanisms underlying different types of sound symbolism, this paper argues that analogies based on acoustics alone are often sufficient, rendering extant articulatory explanations for many iconic phenomena superfluous. This paper further suggests that different types of crossmodal iconicity in spoken language can fruitfully be understood as an extension of onomatopoeia: when speakers iconically depict such perceptual characteristics as size and shape, they mimic the acoustics that are correlated with these characteristics in the natural world.

Forms, Mechanisms, and Roles of Iconicity in Spoken Language: A Review

Historically, debates over relationships between spoken lexical form and meaning have been dominated by views of arbitrariness. However more recent research revealed a different perspective, in which non-arbitrary mappings play an important role in the makeup of a lexicon. It is now clear that phoneme-sound symbolism - along with other types of form-to-meaning mappings - contributes to non-arbitrariness (iconicity) of spoken words, which is present in many forms and degrees in different languages. Attempts have been made to provide a mechanistic explanation of the phenomenon, and these theories largely centre around cross-modal correspondences. We build on these views to explore iconicity within the evolutionary context and the neurobiological framework for human language processing. We argue that the multimodal bihemsipheric communicative system, to which iconicity is integral, has important phylogenetic and ontogenetic advantages, facilitating language learning, comprehension, and processing. Despite its numerous advantages however, iconicity must compete with arbitrariness, forcing language systems to balance the competing needs of perceptual grounding of the linguistic form and ensuring an effective signal. We conclude that, on balance, iconicity should be viewed as integral to language, and not merely a marginal phenomenon.

Front Is High and Back Is Low: Sound-Space Iconicity in Finnish

Previous investigations have shown various interactions between spatial concepts and speech sounds. For instance, the front-high vowel [i] is associated with the concept of forward, and the back-high vowel [o] is associated with the concept of backward. Three experiments investigated whether the concepts of forward/front and backward/back are associated with high- and low-pitched vocalizations, respectively, in Finnish. In Experiments 1 and 2, the participants associated the high-pitched vocalization with the forward-directed movement and the low-pitched vocalizations with the backward-directed movement. In Experiment 3, the same effect was observed in relation to the concepts of front of and back of. We propose that these observations present a novel sound-space symbolism phenomenon in which spatial concepts of forward/front and backward/back are iconically associated with high- and low-pitched speech sounds. This observation is discussed in relation to the grounding of semantic knowledge of these spatial concepts in the movements of articulators such as relative front/back-directed movements of the tongue.

The alveolar trill is perceived as jagged/rough by speakers of different languagesa)

Typological research shows that across languages, trilled [r] sounds are more common in adjectives describing rough as opposed to smooth surfaces. In this study, this lexical research is built on with an experiment with speakers of 28 different languages from 12 different families. Participants were presented with images of a jagged and a straight line and imagined running their finger along each. They were then played an alveolar trill [r] and an alveolar approximant [l] and matched each sound to one of the lines. Participants showed a strong tendency to match [r] with the jagged line and [l] with the straight line, even more consistently than in a comparable cross-cultural investigation of the bouba/kiki effect. The pattern is strongest for matching [r] to the jagged line, but also very strong for matching [l] to the straight line. While this effect was found with speakers of languages with different phonetic realizations of the rhotic sound, it was weaker when trilled [r] was the primary variant. This suggests that when a sound is used phonologically to make systemic meaning contrasts, its iconic potential may become more limited. These findings extend our understanding of iconic crossmodal correspondences, highlighting deep-rooted connections between auditory perception and touch/vision.

Can sequencing of articulation ease explain the in-out effect? A preregistered test

Words whose consonantal articulation places move from the front of the mouth to the back (e.g. BADAKA; inward) receive more positive evaluations than words whose consonantal articulation places move from the back of the mouth to the front (e.g. KADABA; outward). This in-out effect has a variety of affective, cognitive, and even behavioural consequences, but its underlying mechanisms remain elusive. Most recently, a linguistic explanation has been proposed applying the linguistic easy-first account and the so-called labial-coronal effect from developmental speech research and phonology to the in-out effect: Labials (front) are easier to process than coronals (middle); and people prefer easy followed by harder motor components. Disentangling consonantal articulation direction and articulation place, the present three preregistered experiments (total N = 1012) found in-out effects for coronal-dorsal (back), and labial-dorsal articulation places. Critically, no in-out effect emerged for labial-coronal articulation places. Thus, the in-out effect is unlikely an instantiation of easy first.

Sound-action symbolism in relation to precision manipulation and whole-hand grasp usage

It has been suggested that actions can provide a fruitful conceptual context for sound symbolism phenomena, and that tight interaction between manual and articulatory processes might cause that hand actions, in particular, are sound-symbolically associated with specific speech sounds. Experiment 1 investigated whether novel words, built from speech sounds that have been previously linked to precision or power grasp responses, are implicitly associated with perceived actions that present precision manipulation or whole-hand grasp tool-use or the corresponding utilisation pantomimes. In the two-alternative forced-choice task, the participants were more likely to match novel words to tool-use actions and corresponding pantomimes that were sound-symbolically congruent with the words. Experiment 2 showed that the same or even larger sound-action symbolism effect can be observed when the pantomimes present unfamiliar utilisation actions. Based on this we propose that the sound-action symbolism might originate from the same sensorimotor mechanisms that process the meaning of iconic gestural signs. The study presents a novel sound-action phenomenon and supports the view that hand-mouth interaction might manifest itself by associating specific speech sounds with grasp-related utilisations.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Neural Basis Of Sound-Symbolic Pseudoword-Shape Correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants ( n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

HIGHLIGHTS

fMRI investigation of sound-symbolic correspondences between auditory pseudowords and visual shapesFaster reaction times for congruent than incongruent audiovisual stimuliGreater activation in auditory and visual cortices for congruent stimuliHigher classification accuracy for congruent stimuli in language and visual areasSound symbolism involves language processing and multisensory integration.

Cross-modal associations between paintings and sounds: Effects of embodiment

The present study investigated cross-modal associations between a series of paintings and sounds. We studied the effects of sound congruency (congruent vs. non-congruent sounds) and embodiment (embodied vs. synthetic sounds) on the evaluation of abstract and figurative paintings. Participants evaluated figurative and abstract paintings paired with congruent and non-congruent embodied and synthetic sounds. They also evaluated the perceived meaningfulness of the paintings, aesthetic value and immersive experience of the paintings. Embodied sounds (sounds associated with bodily sensations, bodily movements and touch) were more strongly associated with figurative paintings, while synthetic sounds (non-embodied sounds) were more strongly associated with abstract paintings. Sound congruency increased the perceived meaningfulness, immersive experience and aesthetic value of paintings. Sound embodiment increased immersive experience of paintings.