Touching words is not enough: How visual experience influences haptic–auditory associations in the “Bouba–Kiki” effect

Cracking arbitrariness: A data-driven study of auditory iconicity in spoken English

Auditory iconic words display a phonological profile that imitates their referents' sounds. Traditionally, those words are thought to constitute a minor portion of the auditory lexicon. In this article, we challenge this assumption by assessing the pervasiveness of onomatopoeia in the English auditory vocabulary through a novel data-driven procedure. We embed spoken words and natural sounds into a shared auditory space through (a) a short-time Fourier transform, (b) a convolutional neural network trained to classify sounds, and (c) a network trained on speech recognition. Then, we employ the obtained vector representations to measure their objective auditory resemblance. These similarity indexes show that imitation is not limited to some circumscribed semantic categories, but instead can be considered as a widespread mechanism underlying the structure of the English auditory vocabulary. We finally empirically validate our similarity indexes as measures of iconicity against human judgments.

Susceptibility to spatial illusions does not depend on visual experience: Evidence from sighted and blind children

Visuospatial illusions may be a by-product of learned regularities in the environment or they may reflect the recruitment of sensory mechanisms that, in some contexts, provide an erroneous spatial estimate. Young children experience visual illusions, and blind adults are susceptible using touch alone, suggesting that the perceptual inferences influencing illusions are amodal and rapidly acquired. However, other evidence, such as visual illusions in the newly sighted, points to the involvement of innate mechanisms. To help tease apart cognitive from sensory influences, we investigated susceptibility to the Ebbinghaus, Müller-Lyer and Vertical-Horizontal illusions in children aged 6-14 years following visual-only, haptic-only and bimodal exploration. Consistent with previous findings, children of all ages were susceptible to all three visual illusions. In addition, illusions of extent but not of size were experienced using haptics alone. We then tested 17 congenitally blind children to investigate whether illusions were mediated by vision. Similar to their sighted counterparts, blind children were also susceptible to illusions following haptic exploration suggesting that early visual experience is not necessary for spatial illusions to be perceived. Reduced susceptibility in older children to some illusions further implies that explicit or formal knowledge of spatial relations is unlikely to mediate these experiences. Instead, the results are consistent with previous evidence for cross-modal interactions in 'visual' brain regions and point to the possibility that illusions may be driven by innate developmental processes that are not entirely dependent on, although are refined by, visual experience.

Sound effects have only minor contribution to perceptions of anthropomorphism and animacy of simple animated shapes

While studies of anthropomorphism have spanned many decades, there is little evidence of the role that sound effects may play. We present two studies into sound's influence on perceptions of anthropomorphism and animacy using simple geometric animated shapes. For the first study, conducted on 149 participants, we simplified the animation to just two "bumping" squares. Study Two recreated the Heider-Simmel study of 1944, and was conducted on 250 participants under five conditions: without sound, and with one of two different sound types (interface sounds and "anthropomorphic" robot sounds) with two stereo modes (fixed in stereo position, or binaurally panned with the movement). We had participants answer both the Individual Differences in Anthropomorphism Questionnaire and the Godspeed Questionnaire, with three additional questions added. Results showed that the sound had a minor impact on anthropomorphism and potency in Study One, but did not impact animacy. Study Two showed no significant effect on anthropomorphism or animacy, but did show an impact on perceived intelligence and perceptions of activity.

The alveolar trill is perceived as jagged/rough by speakers of different languagesa)

Typological research shows that across languages, trilled [r] sounds are more common in adjectives describing rough as opposed to smooth surfaces. In this study, this lexical research is built on with an experiment with speakers of 28 different languages from 12 different families. Participants were presented with images of a jagged and a straight line and imagined running their finger along each. They were then played an alveolar trill [r] and an alveolar approximant [l] and matched each sound to one of the lines. Participants showed a strong tendency to match [r] with the jagged line and [l] with the straight line, even more consistently than in a comparable cross-cultural investigation of the bouba/kiki effect. The pattern is strongest for matching [r] to the jagged line, but also very strong for matching [l] to the straight line. While this effect was found with speakers of languages with different phonetic realizations of the rhotic sound, it was weaker when trilled [r] was the primary variant. This suggests that when a sound is used phonologically to make systemic meaning contrasts, its iconic potential may become more limited. These findings extend our understanding of iconic crossmodal correspondences, highlighting deep-rooted connections between auditory perception and touch/vision.

Prior visual experience increases children's use of effective haptic exploration strategies in audio-tactile sound-shape correspondences

Sound-shape correspondence refers to the preferential mapping of information across the senses, such as associating a nonsense word like bouba with rounded abstract shapes and kiki with spiky abstract shapes. Here we focused on audio-tactile (AT) sound-shape correspondences between nonsense words and abstract shapes that are felt but not seen. Despite previous research indicating a role for visual experience in establishing AT associations, it remains unclear how visual experience facilitates AT correspondences. Here we investigated one hypothesis: seeing the abstract shapes improve haptic exploration by (a) increasing effective haptic strategies and/or (b) decreasing ineffective haptic strategies. We analyzed five haptic strategies in video-recordings of 6- to 8-year-old children obtained in a previous study. We found the dominant strategy used to explore shapes differed based on visual experience. Effective strategies, which provide information about shape, were dominant in participants with prior visual experience, whereas ineffective strategies, which do not provide information about shape, were dominant in participants without prior visual experience. With prior visual experience, poking-an effective and efficient strategy-was dominant, whereas without prior visual experience, uncategorizable and ineffective strategies were dominant. These findings suggest that prior visual experience of abstract shapes in 6- to 8-year-olds can increase the effectiveness and efficiency of haptic exploration, potentially explaining why prior visual experience can increase the strength of AT sound-shape correspondences.

Crossmodal associations between naturally occurring tactile and sound textures

This study investigates the crossmodal associations between naturally occurring sound textures and tactile textures. Previous research has demonstrated the association between low-level sensory features of sound and touch, as well as higher-level, cognitively mediated associations involving language, emotions, and metaphors. However, stimuli like textures, which are found in both modalities have received less attention. In this study, we conducted two experiments: a free association task and a two alternate forced choice task using everyday tactile textures and sound textures selected from natural sound categories. The results revealed consistent crossmodal associations reported by participants between the textures of the two modalities. They tended to associate more sound textures (e.g., wood shavings and sandpaper) with tactile surfaces that were rated as harder, rougher, and intermediate on the sticky-slippery scale. While some participants based the auditory-tactile association on sensory features, others made the associations based on semantic relationships, co-occurrence in nature, and emotional mediation. Interestingly, the statistical features of the sound textures (mean, variance, kurtosis, power, autocorrelation, and correlation) did not show significant correlations with the crossmodal associations, indicating a higher-level association. This study provides insights into auditory-tactile associations by highlighting the role of sensory and emotional (or cognitive) factors in prompting these associations.

The Bouba-Kiki effect is predicted by sound properties but not speech properties

Humans robustly associate spiky shapes to words like "Kiki" and round shapes to words like "Bouba." According to a popular explanation, this is because the mouth assumes an angular shape while speaking "Kiki" and a rounded shape for "Bouba." Alternatively, this effect could reflect more general associations between shape and sound that are not specific to mouth shape or articulatory properties of speech. These possibilities can be distinguished using unpronounceable sounds: The mouth-shape hypothesis predicts no Bouba-Kiki effect for these sounds, whereas the generic shape-sound hypothesis predicts a systematic effect. Here, we show that the Bouba-Kiki effect is present for a variety of unpronounceable sounds ranging from reversed words and real object sounds (n = 45 participants) and even pure tones (n = 28). The effect was strongly correlated with the mean frequency of a sound across both spoken and reversed words. The effect was not systematically predicted by subjective ratings of pronounceability or with mouth aspect ratios measured from video. Thus, the Bouba-Kiki effect is explained using simple shape-sound associations rather than using speech properties.

Synesthetic Correspondence: An Overview

Intramodal and cross-modal perceptual grouping based on the spatial proximity and temporal closeness between multiple sensory stimuli, as an operational principle has built a coherent and meaningful representation of the multisensory event/object. To implement and investigate the cross-modal perceptual grouping, researchers have employed excellent paradigms of spatial/temporal ventriloquism and cross-modal dynamic capture and have revealed the conditional constraints as well as the functional facilitations among various correspondence of sensory properties, with featured behavioral evidence, computational framework as well as brain oscillation patterns. Typically, synesthetic correspondence as a special type of cross-modal correspondence can shape the efficiency and effect-size of cross-modal interaction. For example, factors such as pitch/loudness in the auditory dimension with size/brightness in the visual dimension could modulate the strength of the cross-modal temporal capture. The empirical behavioral findings, as well as psychophysical and neurophysiological evidence to address the cross-modal perceptual grouping and synesthetic correspondence, were summarized in this review. Finally, the potential applications (such as artificial synesthesia device) and how synesthetic correspondence interface with semantics (sensory linguistics), as well as the promising research questions in this field have been discussed.

Visual experience shapes the Bouba-Kiki effect and the size-weight illusion upon sight restoration from congenital blindness

The Bouba-Kiki effect is the systematic mapping between round/spiky shapes and speech sounds ("Bouba"/"Kiki"). In the size-weight illusion, participants judge the smaller of two equally-weighted objects as being heavier. Here we investigated the contribution of visual experience to the development of these phenomena. We compared three groups: early blind individuals (no visual experience), individuals treated for congenital cataracts years after birth (late visual experience), and typically sighted controls (visual experience from birth). We found that, in cataract-treated participants (tested visually/visuo-haptically), both phenomena are absent shortly after sight onset, just like in blind individuals (tested haptically). However, they emerge within months following surgery, becoming statistically indistinguishable from the sighted controls. This suggests a pivotal role of visual experience and refutes the existence of an early sensitive period: A short period of experience, even when gained only years after birth, is sufficient for participants to visually pick-up regularities in the environment, contributing to the development of these phenomena.

An fMRI study of visual geometric shapes processing

Cross-modal correspondence has been consistently evidenced between shapes and other sensory attributes. Especially, the curvature of shapes may arouse the affective account, which may contribute to understanding the mechanism of cross-modal integration. Hence, the current study used the functional magnetic resonance imaging (fMRI) technique to examine brain activity’s specificity when people view circular and angular shapes. The circular shapes consisted of a circle and an ellipse, while the angular shapes consisted of a triangle and a star. Results show that the brain areas activated by circular shapes mainly involved the sub-occipital lobe, fusiform gyrus, sub and middle occipital gyrus, and cerebellar VI. The brain areas activated by angular shapes mainly involve the cuneus, middle occipital gyrus, lingual gyrus, and calcarine gyrus. The brain activation patterns of circular shapes did not differ significantly from those of angular shapes. Such a null finding was unexpected when previous cross-modal correspondence of shape curvature was considered. The different brain regions detected by circular and angular shapes and the potential explanations were discussed in the paper.

Exploring Group Differences in the Crossmodal Correspondences

There has been a rapid growth of interest amongst researchers in the cross-modal correspondences in recent years. In part, this has resulted from the emerging realization of the important role that the correspondences can sometimes play in multisensory integration. In turn, this has led to an interest in the nature of any differences between individuals, or rather, between groups of individuals, in the strength and/or consensuality of cross-modal correspondences that may be observed in both neurotypically normal groups cross-culturally, developmentally, and across various special populations (including those who have lost a sense, as well as those with autistic tendencies). The hope is that our emerging understanding of such group differences may one day provide grounds for supporting the reality of the various different types of correspondence that have so far been proposed, namely structural, statistical, semantic, and hedonic (or emotionally mediated).

Making sense of sensory language: Acquisition of sensory knowledge by individuals with congenital sensory impairments

The present article provides a narrative review on how language communicates sensory information and how knowledge of sight and sound develops in individuals born deaf or blind. Studying knowledge of the perceptually inaccessible sensory domain for these populations offers a lens into how humans learn about that which they cannot perceive. We first review the linguistic strategies within language that communicate sensory information. Highlighting the power of language to shape knowledge, we next review the detailed knowledge of sensory information by individuals with congenital sensory impairments, limitations therein, and neural representations of imperceptible phenomena. We suggest that the acquisition of sensory knowledge is supported by language, experience with multiple perceptual domains, and cognitive and social abilities which mature over the first years of life, both in individuals with and without sensory impairment. We conclude by proposing a developmental trajectory for acquiring sensory knowledge in the absence of sensory perception.

A Cross-Modal and Cross-lingual Study of Iconicity in Language: Insights From Deep Learning

The present paper addresses the study of non-arbitrariness in language within a deep learning framework. We present a set of experiments aimed at assessing the pervasiveness of different forms of non-arbitrary phonological patterns across a set of typologically distant languages. Different sequence-processing neural networks are trained in a set of languages to associate the phonetic vectorization of a set of words to their sensory (Experiment 1), semantic (Experiment 2), and word-class representations (Experiment 3). The models are then tested, without further training, in a set of novel instances in a language belonging to a different language family, and their performance is compared with a randomized baseline. We show that the three cross-domain mappings can be successfully transferred across languages and language families, suggesting that the phonological structure of the lexicon is pervaded with language-invariant cues about the words' meaning and their syntactic classes.

The Bouba-Kiki Effect in Children With Childhood Apraxia of Speech

PURPOSE

The bouba-kiki (BK) effect refers to associations between visual shapes and auditory pseudonames. Thus, when tested, people tend to associate the pseudowords bouba and kiki with round or spiky shapes, respectively. This association requires cross-modal sensory integration. The ability to integrate information from different sensory modalities is crucial for speech development. A clinical population that may be impaired in cross-modal sensory integration is children with childhood apraxia of speech (CAS). The purpose of this study was to examine the involvement of cross-modal sensory integration in children with (CAS).

METHOD

The BK effect was assessed in participants with CAS (n = 18) and two control groups: One control group was composed of children with developmental language disorder (DLD), also termed specific language impairment (n = 15), and a second group included typically developing (TD) children (n = 22). The children were presented with 14 pairs of novel visual displays and nonwords. All the children were asked to state which shape and nonword correspond to one another. In addition, background cognitive (Leiter-3) and language measures (Hebrew PLS-4) were determined for all children.

RESULTS

Children in the CAS group were less successful in associating between visual shapes and corresponding auditory pseudonames (e.g., associating the spoken word "bouba" with a round shape; the spoken word "kiki" with a spiky shape). Thus, children with CAS demonstrated a statistically significant reduced BK effect compared with participants with TD and participants with DLD. No significant difference was found between the TD group and the DLD group.

CONCLUSIONS

The reduced BK effect in children with CAS supports the notion that cross-modal sensory integration may be altered in these children. Cross-modal sensory integration is the basis for speech production. Thus, difficulties in sensory integration may contribute to speech difficulties in CAS.

The bouba/kiki effect is robust across cultures and writing systems

The bouba/kiki effect-the association of the nonce word bouba with a round shape and kiki with a spiky shape-is a type of correspondence between speech sounds and visual properties with potentially deep implications for the evolution of spoken language. However, there is debate over the robustness of the effect across cultures and the influence of orthography. We report an online experiment that tested the bouba/kiki effect across speakers of 25 languages representing nine language families and 10 writing systems. Overall, we found strong evidence for the effect across languages, with bouba eliciting more congruent responses than kiki. Participants who spoke languages with Roman scripts were only marginally more likely to show the effect, and analysis of the orthographic shape of the words in different scripts showed that the effect was no stronger for scripts that use rounder forms for bouba and spikier forms for kiki. These results confirm that the bouba/kiki phenomenon is rooted in crossmodal correspondence between aspects of the voice and visual shape, largely independent of orthography. They provide the strongest demonstration to date that the bouba/kiki effect is robust across cultures and writing systems. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part II)'.

Touching and hearing the shapes: How auditory angular and curved sounds influence proficiency in recognising tactile angle and curve shapes when experienced and inexperienced in using haptic touch

This study investigated whether adding auditory angular and curved sounds to tactile angle and curve shapes – one unspecified sound to one unspecified shape – positively influences the accuracy and exploration time in recognising tactile angles and curves when experienced and inexperienced in using haptic touch. A within-participant experiment was conducted, with two groups of participants: experienced and inexperienced in using haptic touch, and with two conditions: congruous (e.g., angle shape and angular sound) and incongruous (e.g., angle shape and curved sound) tactile and auditory shape information. Adding congruous auditory angular and curved sounds to tactile angle and curve shapes positively influences the accuracy in recognising tactile angles and curves both when experienced and inexperienced in using haptic touch, and the exploration time on correct recognitions when experienced. People integrate tactile and auditory (angle; curve) shape information and this improves their proficiency in recognising tactile angles and curves.

The feeling of "kiki": Comparing developmental changes in sound-shape correspondence for audio-visual and audio-tactile stimuli

Sound-shape crossmodal correspondence, the naturally occurring associations between abstract visual shapes and nonsense sounds, is one aspect of multisensory processing that strengthens across early childhood. Little is known regarding whether school-aged children exhibit other variants of sound-shape correspondences such as audio-tactile (AT) associations between tactile shapes and nonsense sounds. Based on previous research in blind individuals suggesting the role of visual experience in establishing sound-shape correspondence, we hypothesized that children would show weaker AT association than adults and that children's AT association would be enhanced with visual experience of the shapes. In Experiment 1, we showed that, when asked to match shapes explored haptically via touch to nonsense words, 6- to 8-year-olds exhibited inconsistent AT associations, whereas older children and adults exhibited the expected AT associations, despite robust audio-visual (AV) associations found across all age groups in a related study. In Experiment 2, we confirmed the role of visual experience in enhancing AT association; here, 6- to 8-year-olds could exhibit the expected AT association if first exposed to the AV condition, whereas adults showed the expected AT association irrespective of whether the AV condition was tested first or second. Our finding suggests that AT sound-shape correspondence is weak early in development relative to AV sound-shape correspondence, paralleling previous findings on the development of other types of multisensory associations. The potential role of visual experience in the development of sound-shape correspondences in other senses is discussed.

Brain networks underlying the processing of sound symbolism related to softness perception

Unlike the assumption of modern linguistics, there is non-arbitrary association between sound and meaning in sound symbolic words. Neuroimaging studies have suggested the unique contribution of the superior temporal sulcus to the processing of sound symbolism. However, because these findings are limited to the mapping between sound symbolism and visually presented objects, the processing of sound symbolic information may also involve the sensory-modality dependent mechanisms. Here, we conducted a functional magnetic resonance imaging experiment to test whether the brain regions engaged in the tactile processing of object properties are also involved in mapping sound symbolic information with tactually perceived object properties. Thirty-two healthy subjects conducted a matching task in which they judged the congruency between softness perceived by touch and softness associated with sound symbolic words. Congruency effect was observed in the orbitofrontal cortex, inferior frontal gyrus, insula, medial superior frontal gyrus, cingulate gyrus, and cerebellum. This effect in the insula and medial superior frontal gyri was overlapped with softness-related activity that was separately measured in the same subjects in the tactile experiment. These results indicate that the insula and medial superior frontal gyrus play a role in processing sound symbolic information and relating it to the tactile softness information.

Virtual Haptic Perception as an Educational Assistive Technology: A Case Study in Inclusive Education

The traditional educational process of blind people is a complex practice that relies on the haptic perception (tactile perception) of physical models. However, physical models may be costly, inaccessible or may require a large storage space. To overcome these difficulties, in this article a virtual haptic perception approach to support the teaching and learning process of blind people is proposed. The proposed approach combines the use of virtual reality and haptic technologies. The research aim is to objectively evaluate the feasibility and effectiveness of using virtual haptic perception in the education of blind children. For this purpose, an experimental methodology was defined and used to teach maths, in particular fundamental three-dimensional (3D) shapes, to blind children. The results are analysed in terms of the participants' ability to explore and recognize virtual objects, and the knowledge gain after the virtual perception learning period. From this analysis it is concluded that haptic virtual perception is a valid and effective assistive technology for the education of blind children.

Neural mechanisms of visual sensitive periods in humans

Sensitive periods in brain development are phases of enhanced susceptibility to experience. Here we discuss research from human and non-human neuroscience studies which have demonstrated a) differences in the way infants vs. adults learn; b) how the brain adapts to atypical conditions, in particular a congenital vs. a late onset blindness (sensitive periods for atypical brain development); and c) the extent to which neural systems are capable of acquiring a typical brain organization after sight restoration following a congenital vs. late phase of pattern vision deprivation (sensitive periods for typical brain development). By integrating these three lines of research, we propose neural mechanisms characteristic of sensitive periods vs. adult neuroplasticity and learning.

A Protracted Sensitive Period Regulates the Development of Cross-Modal Sound-Shape Associations in Humans

Humans preferentially match arbitrary words containing higher- and lower-frequency phonemes to angular and smooth shapes, respectively. Here, we investigated the role of visual experience in the development of audiovisual and audiohaptic sound-shape associations (SSAs) using a unique set of five groups: individuals who had suffered a transient period of congenital blindness through congenital bilateral dense cataracts before undergoing cataract-reversal surgeries (CC group), individuals with a history of developmental cataracts (DC group), individuals with congenital permanent blindness (CB group), individuals with late permanent blindness (LB group), and controls with typical sight (TS group). Whereas the TS and LB groups showed highly robust SSAs, the CB, CC, and DC groups did not-in any of the modality combinations tested. These results provide evidence for a protracted sensitive period during which aberrant vision prevents SSA acquisition. Moreover, the finding of a systematic SSA in the LB group demonstrates that representations acquired during the sensitive period are resilient to loss despite dramatically changed experience.

Haptic sound-symbolism in young Spanish-speaking children

Haptic sound symbolism has been found in adults, between ideophones and various textures, between words and shape, and between written words and texture. However, associations between the sound of nonwords and features other than shape and in early stages of development have been less explored. The present study investigates the haptic manifestation of sound symbolism in an early developmental stage. We examined associations between nonwords and the rough-smooth tactile dimension in 3.5-to-4.5-year-old children. Two experiments were conducted: a pointing selection task and a naming task. Sound symbolic associations were found in the naming task, but not in the pointing task. On the other hand an unexpected bias towards smoothness was found in the pointing task. We explain these results by suggesting that the articulation of nonwords may facilitate or intensify sound symbolism, and that hedonic biases are manifested in the pointing task.

A Critical Analysis of Colour-Shape Correspondences: Examining the Replicability of Colour-Shape Associations

Research on the topic of colour-shape correspondences started in the early 20th century with the Bauhaus artist Wassily Kandinsky. However, more recently, the topic has been examined using the empirical framework of crossmodal correspondences research. The field remains one in which consistent results and generalisable hypotheses about the existence and nature of colour-shape correspondences are lacking. The replicability and consistency of findings concerning colour-shape correspondences are examined in three online colour-shape matching experiments using the same procedure and study design while varying the sets of shape stimuli that are evaluated. Participants matched one of 36 colours to each shape as well as made preference and arousal appraisal ratings for each of the shapes and colours. The complexities of analysing colour-shape correspondence data are discussed and illustrated by classifying and analysing shape and colours in a variety of different ways, including using continuous perceptual and objective measures. Significant colour-shape associations were found. However, as hypothesised, limited consistent results in regard to what perceptual shape characteristics predicted colour choices were documented across the three stimuli sets. This was the case both within and across different analysis methods. The factors that may be responsible for these inconsistencies are critically discussed. Intriguingly, however, evidence for emotional mediation, whereby shape and colour liking and arousal appraisals appear to influence the colour-shape correspondences made by participants, was found across all three experiments.

Sound symbolism in sighted and blind. The role of vision and orthography in sound-shape correspondences

Non-arbitrary sound-shape correspondences (SSC), such as the "bouba-kiki" effect, have been consistently observed across languages and together with other sound-symbolic phenomena challenge the classic linguistic dictum of the arbitrariness of the sign. Yet, it is unclear what makes a sound "round" or "spiky" to the human mind. Here we tested the hypothesis that visual experience is necessary for the emergence of SSC, supported by empirical evidence showing reduced SSC in visually impaired people. Results of two experiments comparing early blind and sighted individuals showed that SSC emerged strongly in both groups. Experiment 2, however, showed a partially different pattern of SSC in sighted and blind, that was mostly explained by a different effect of orthographic letter shape: The shape of written letters (spontaneously activated by spoken words) influenced SSC in the sighted, but not in the blind, who are exposed to an orthography (Braille) in which letters do not have spiky or round outlines. In sum, early blindness does not prevent the emergence of SSC, and differences between sighted and visually impaired people may be due the indirect influence (or lack thereof) of orthographic letter shape.

Multimodal Interaction of Contextual and Non-Contextual Sound and Haptics in Virtual Simulations

Touch plays a fundamental role in our daily interactions, allowing us to interact with and perceive objects and their spatial properties. Despite its importance in the real-world, touch is often ignored in virtual environments. However, accurately simulating the sense of touch is difficult, requiring the use of high-fidelity haptic devices that are cost-prohibitive. Lower fidelity consumer-level haptic devices are becoming more widespread, yet are generally limited in perceived fidelity and the range of motion (degrees of freedom) required to realistically simulate many tasks. Studies into sound and vision suggest that the presence or absence of sound can influence task performance. Here, we explore whether the presence or absence of contextually relevant sound cues influences the performance of a simple haptic drilling task. Although the results of this study do not show any statistically significant difference in task performance with general (task-irrelevant) sound, we discuss how this is a necessary step in understanding the role of sound on haptic perception.

Bouba or kiki with and without vision: Shape-audio regularities and mental images

95% of the world's population associate a rounded visual shape with the spoken word 'bouba', and an angular visual shape with the spoken word 'kiki', known as the bouba/kiki-effect. The bouba/kiki-effect occurs irrespective of familiarity with either the shape or word. This study investigated the bouba/kiki-effect when using haptic touch instead of vision, including the role of visual imagery. It also investigated whether the bouba/kiki shape-audio regularities are noticed at all, that is, whether they affect the bouba/kiki-effect itself and/or the recognition of individual bouba/kiki shapes, and finally what mental images they produce. Three experiments were conducted, with three groups of participants: blind, blindfold, and vision. In Experiment 1, the participants were asked to pick out the tactile/visual shape that they associated with the auditory bouba/kiki. Experiment 1 found that the participants who were blind did not show an instant bouba/kiki-effect (in Trial 1), whereas the blindfolded and the fully sighted did. It also found that the bouba/kiki shape-audio regularities affected the bouba/kiki-effect when using haptic touch: Those who were blind did show the bouba/kiki-effect from Trial 4, and those who were blindfolded no longer did. In Experiment 2, the participants were asked to name one tactile/visual shape and a segment of audio together as either 'bouba' or 'kiki'. Experiment 2 found that corresponding shape and audio improved the accuracy of both the blindfolded and the fully sighted, but not of those who were blind - they ignored the audio. Finally, in Experiment 3, the participants were asked to draw the shape that they associated with the auditory bouba/kiki. Experiment 3 found that their mental images, as depicted in their drawings, were not affected by whether they had experienced the bouba/kiki shapes by haptic touch or by vision. Regardless of their prior shape experience, that is, tactile or visual, their mental images included the most characteristic shape feature of bouba and kiki: curve and angle, respectively, and typically not the global shape. When taken together, these experiments suggest that the sensory regularities and mental images concerning bouba and kiki do not have to be based on, or even include visual information.

Bouba/Kiki in Touch: Associations Between Tactile Perceptual Qualities and Japanese Phonemes

Several studies have shown cross-modal associations between sounds and vision or gustation by asking participants to match pre-defined sound-symbolic words (SSWs), such as “bouba” or “kiki,” with visual or gustatory materials. Here, we conducted an explorative study on cross-modal associations of tactile sensations using spontaneous production of Japanese SSWs and semantic ratings. The Japanese language was selected, because it has a large number of SSWs that can represent a wide range of tactile perceptual spaces with fine resolution, and it shows strong associations between sound and touch. In the experiment, we used 120 everyday materials to cover basic material categories that could be associated with fundamental dimensions of tactile perception. Upon contact with these materials, participants expressed their tactile sensations by using Japanese SSWs, and at the same time, evaluated the tactile sensations by semantic differential scales using adjective pairs. Thanks to the variety of testing materials, we were able to demonstrate the existence of systematic associations between sounds and tactile fundamental perceptual dimensions in a more detailed and comprehensive way than ever done so before. In particular, we found that for vowels, positive tactile ratings were associated with the back vowel (/u/), while negative ratings were associated with the front vowels (/i/ and /e/). The central vowels (/o/ and /a/) were mainly associated with rough, hard, and dry feelings. Consonants were categorized based on vocal features and articulation. The category of the voiced consonants (e.g., /dz/ and /g/) corresponded to feelings of roughness, while that of voiceless consonants (e.g., /ʦ/, and /s/) corresponded to feelings of smoothness. The categories of the bilabial plosive (/p/ and /b/) and voiced alveolar nasal (/n/) consonants were mainly related to soft, sticky and wet feelings, while that of voiceless alveolar affricate (/ʦ/) and voiceless velar plosive (/k/) consonants were related to hard, slippery and dry feelings.

Angular Versus Curved Shapes: Correspondences and Emotional Processing

The present work aimed to systematically examine sensory and higher level correspondences to angular and curved shapes. Participants matched angular and curved abstract shapes to sensory experiences in five different modalities as well as to emotion, gender, and name attributes presented as written labels (Study 1) and real experiences (Study 2). The results demonstrated nonarbitrary mapping of angular and curved shapes to attributes from all basic sensory modalities (vision, audition, gustation, olfaction, and tactation) and higher level attributes (emotion, gender, and name). Participants associated curved shapes with sweet taste, quiet or calm sound, vanilla smell, green color, smooth texture, relieved emotion, female gender, and wide-vowel names. In contrast, they associated angular shapes with sour taste, loud or dynamic sound, spicy or citrus smell, red color, rough texture, excited or surprise emotion, male gender, and narrow-vowel names. These prevalent correspondences were robust across different shape pairs as well as all sensory and higher level attributes, presented as both verbal labels and real sensory experiences. The second goal of this research was to examine the relationship between the shape correspondences and individual differences in emotional processing, assessed by self-report and performance measures. The results suggest that heightened emotional ability is associated with making shape attributions that go along with the found prevalent trends.

Sound Properties Associated With Equiluminant Colours

There is a widespread tendency to associate certain properties of sound with those of colour (e.g., higher pitches with lighter colours). Yet it is an open question how sound influences chroma or hue when properly controlling for lightness. To examine this, we asked participants to adjust physically equiluminant colours until they ‘went best’ with certain sounds. For pure tones, complex sine waves and vocal timbres, increases in frequency were associated with increases in chroma. Increasing the loudness of pure tones also increased chroma. Hue associations varied depending on the type of stimuli. In stimuli that involved only limited bands of frequencies (pure tones, vocal timbres), frequency correlated with hue, such that low frequencies gave blue hues and progressed to yellow hues at 800 Hz. Increasing the loudness of a pure tone was also associated with a shift from blue to yellow. However, for complex sounds that share the same bandwidth of frequencies (100–3200 Hz) but that vary in terms of which frequencies have the most power, all stimuli were associated with yellow hues. This suggests that the presence of high frequencies (above 800 Hz) consistently yields yellow hues. Overall we conclude that while pitch–chroma associations appear to flexibly re-apply themselves across a variety of contexts, frequencies above 800 Hz appear to produce yellow hues irrespective of context. These findings reveal new sound–colour correspondences previously obscured through not controlling for lightness. Findings are discussed in relation to understanding the underlying rules of cross-modal correspondences, synaesthesia, and optimising the sensory substitution of visual information through sound.

When "Bouba" equals "Kiki": Cultural commonalities and cultural differences in sound-shape correspondences

It has been suggested that the Bouba/Kiki effect, in which meaningless speech sounds are systematically mapped onto rounded or angular shapes, reflects a universal crossmodal correspondence between audition and vision. Here, radial frequency (RF) patterns were adapted in order to compare the Bouba/Kiki effect in Eastern and Western participants demonstrating different perceptual styles. Three attributes of the RF patterns were manipulated: The frequency, amplitude, and spikiness of the sinusoidal modulations along the circumference of a circle. By testing participants in the US and Taiwan, both cultural commonalities and differences in sound-shape correspondence were revealed. RF patterns were more likely to be matched with “Kiki” than with “Bouba” when the frequency, amplitude, and spikiness increased. The responses from both groups of participants had a similar weighting on frequency; nevertheless, the North Americans had a higher weighting on amplitude, but a lower weighting on spikiness, than their Taiwanese counterparts. These novel results regarding cultural differences suggest that the Bouba/Kiki effect is partly tuned by differing perceptual experience. In addition, using the RF patterns in the Bouba/Kiki effect provides a “mid-level” linkage between visual and auditory processing, and a future understanding of sound-shape correspondences based on the mechanism of visual pattern processing.

When Sandpaper Is ‘Kiki’ and Satin Is ‘Bouba’: an Exploration of the Associations Between Words, Emotional States, and the Tactile Attributes of Everyday Materials

Over the last decade, scientists working on the topic of multisensory integration, as well as designers and marketers involved in trying to understand consumer behavior, have become increasingly interested in the non-arbitrary associations (e.g., sound symbolism) between different sensorial attributes of the stimuli they work with. Nevertheless, to date, little research in this area has investigated the presence of these crossmodal correspondences in the tactile evaluation of everyday materials. Here, we explore the presence and nature of the associations between tactile sensations, the sound of non-words, and people’s emotional states. Samples of cotton, satin, tinfoil, sandpaper, and abrasive sponge, were stroked along the participants’ forearm at the speed of 5 cm/s. Participants evaluated the materials along several dimensions, comprising scales anchored by pairs of non-words (e.g., Kiki/Bouba) and adjectives (e.g., ugly/beautiful). The results revealed that smoother textures were associated with non-words made up of round-shaped sounds (e.g., Maluma), whereas rougher textures were more strongly associated with sharp-transient sounds (e.g., Takete). The results also revealed the presence of a number of correspondences between tactile surfaces and adjectives related to visual and auditory attributes. For example, smooth textures were associated with features evoked by words such as ‘bright’ and ‘quiet’; by contrast, the rougher textures were associated with adjectives such as ‘dim’ and ‘loud’. The textures were also found to be associated with a number of emotional labels. Taken together, these results further our understanding of crossmodal correspondences involving the tactile modality and provide interesting insights in the applied field of design and marketing.