Drawing sounds: representing tones and chords spatially

Marble melancholy: using crossmodal correspondences of shapes, materials, and music to predict music-induced emotions

Introduction

Music is known to elicit strong emotions in listeners, and, if primed appropriately, can give rise to specific and observable crossmodal correspondences. This study aimed to assess two primary objectives: (1) identifying crossmodal correspondences emerging from music-induced emotions, and (2) examining the predictability of music-induced emotions based on the association of music with visual shapes and materials.

Methods

To achieve this, 176 participants were asked to associate visual shapes and materials with the emotion classes of the Geneva Music-Induced Affect Checklist scale (GEMIAC) elicited by a set of musical excerpts in an online experiment.

Results

Our findings reveal that music-induced emotions and their underlying core affect (i.e., valence and arousal) can be accurately predicted by the joint information of musical excerpt and features of visual shapes and materials associated with these music-induced emotions. Interestingly, valence and arousal induced by music have higher predictability than discrete GEMIAC emotions.

Discussion

These results demonstrate the relevance of crossmodal correspondences in studying music-induced emotions. The potential applications of these findings in the fields of sensory interactions design, multisensory experiences and art, as well as digital and sensory marketing are briefly discussed.

Cross-modal associations between paintings and sounds: Effects of embodiment

The present study investigated cross-modal associations between a series of paintings and sounds. We studied the effects of sound congruency (congruent vs. non-congruent sounds) and embodiment (embodied vs. synthetic sounds) on the evaluation of abstract and figurative paintings. Participants evaluated figurative and abstract paintings paired with congruent and non-congruent embodied and synthetic sounds. They also evaluated the perceived meaningfulness of the paintings, aesthetic value and immersive experience of the paintings. Embodied sounds (sounds associated with bodily sensations, bodily movements and touch) were more strongly associated with figurative paintings, while synthetic sounds (non-embodied sounds) were more strongly associated with abstract paintings. Sound congruency increased the perceived meaningfulness, immersive experience and aesthetic value of paintings. Sound embodiment increased immersive experience of paintings.

Cross-Modal Perceptual Organization in Works of Art

This study investigates the existence of cross-modal correspondences between a series of paintings by Kandinsky and a series of selections from Schönberg music. The experiment was conducted in two phases. In the first phase, by means of the Osgood semantic differential, the participants evaluated the perceptual characteristics first of visual stimuli (some pictures of Kandinsky's paintings, with varying perceptual characteristics and contents) and then of auditory stimuli (musical excerpts taken from the repertoire of Schönberg's piano works) relative to 11 pairs of adjectives tested on a continuous bipolar scale. In the second phase, participants were required to associate pictures and musical excerpts. The results of the semantic differential test show that certain paintings and musical excerpts were evaluated as semantically more similar, while others were evaluated as semantically more different. The results of the direct association between musical excerpts and paintings showed both attractions and repulsions among the stimuli. The overall results provide significant insights into the relationship between concrete and abstract concepts and into the process of perceptual grouping in cross-modal phenomena.

Your Face and Moves Seem Happier When I Smile

In this experiment, we replicated the effect of muscle engagement on perception such that the recognition of another's facial expressions was biased by the observer's facial muscular activity (Blaesi & Wilson, 2010). We extended this replication to show that such a modulatory effect is also observed for the recognition of dynamic bodily expressions. Via a multilab and within-subjects approach, we investigated the emotion recognition of point-light biological walkers, along with that of morphed face stimuli, while subjects were or were not holding a pen in their teeth. Under the "pen-in-the-teeth" condition, participants tended to lower their threshold of perception of happy expressions in facial stimuli compared to the "no-pen" condition, thus replicating the experiment by Blaesi and Wilson (2010). A similar effect was found for the biological motion stimuli such that participants lowered their threshold to perceive happy walkers in the pen-in-the-teeth condition compared to the no-pen condition. This pattern of results was also found in a second experiment in which the no-pen condition was replaced by a situation in which participants held a pen in their lips ("pen-in-lips" condition). These results suggested that facial muscular activity alters the recognition of not only facial expressions but also bodily expressions.

Reaching for the high note: judgments of auditory pitch are affected by kinesthetic position

Auditory pitch is represented on a vertical continuum (e.g., high vs. low). In three experiments, we examined whether the kinesthetic experience of moving in a particular direction (e.g., walking up vs. down stairs; reaching up vs. down) would affect judgments of auditory pitch. Participants listened to three tones twice each, once while moving upward and once while moving downward, and estimated the pitch of each tone. In all experiments, participants' judgments of the tones' pitch were biased in the direction of their movement. Moreover, this effect is not due to visibility of the movement or to using a numerical response method. Our results suggest that kinesthetic information from one's own bodily movements biases pitch estimation, and several possible mechanisms for the effect are discussed.

The Allocation of Valenced Percepts Onto 3D Space

Research on the metaphorical mapping of valenced concepts onto space indicates that positive, neutral, and negative concepts are mapped onto upward, midward, and downward locations, respectively. More recently, this type of research has been tested for the very first time in 3D physical space. The findings corroborate the mapping of valenced concepts onto the vertical space as described above but further show that positive and negative concepts are placed close to and away from the body; neutral concepts are placed midway. The current study aimed at investigating whether valenced perceptual stimuli are positioned onto 3D space akin to the way valenced concepts are positioned. By using a unique device known as the cognition cube, participants placed visual, auditory, tactile and olfactory stimuli on 3D space. The results mimicked the placing of valenced concepts onto 3D space; i.e., positive percepts were placed in upward and close-to-the-body locations and negative percepts were placed in downward and away-from-the-body locations; neutral percepts were placed midway. These pattern of results was more pronounced in the case of visual stimuli, followed by auditory, tactile, and olfactory stimuli. Significance Statement Just recently, a unique device called "the cognition cube" (CC) enabled to find that positive words are mapped onto upward and close-to-the-body locations and negative words are mapped onto downward and away-from-the-body locations; neutral words are placed midway. This way of placing words in relation to the body is consistent with an approach-avoidance effect such that "good" and "bad" things are kept close to and away from one's body. We demonstrate for the very first time that this same pattern emerges when visual, auditory, tactile, and olfactory perceptual stimuli are placed on 3D physical space. We believe these results are significant in that the CC can be used as a new tool to diagnose emotion-related disorders.

The strength of weak embodiment

While popular within some cognitive science approaches, the embodiment approach has still found resistance, particularly in light of evidence arguing against strong forms of embodiment. Among other things, the embodiment approach breaks away from the Cartesian ontology of the modulatory system. We claim that the advantages of the embodiment approach are: a) it grounds cognition into modal experience, b) it is harmonious with a materialist philosophy of mind (emergent materialism), and c) it is supported by experimental research in various fields. However, embodiment must still address abstractions, theoretical misunderstandings (representations vs non-representations) and neuroscientific findings that challenge the extension and relevance of sensorimotor properties into cognitive processes. While the strong version of embodiment is seriously challenged by conceptual and physiological setbacks, its weak version is supported by compelling evidence. We suggest future research focus on the psychophysiological bases of grounded cognition and redirect efforts towards the field of cross-modal correspondence.

An extended research of crossmodal correspondence between color and sound in psychology and cognitive ergonomics

Based on the existing research on sound symbolism and crossmodal correspondence, this study proposed an extended research on cross-modal correspondence between various sound attributes and color properties in a group of non-synesthetes. In Experiment 1, we assessed the associations between each property of sounds and colors. Twenty sounds with five auditory properties (pitch, roughness, sharpness, tempo and discontinuity), each varied in four levels, were used as the sound stimuli. Forty-nine colors with different hues, saturation and brightness were used to match to those sounds. Result revealed that besides pitch and tempo, roughness and sharpness also played roles in sound-color correspondence. Reaction times of sound-hue were a little longer than the reaction times of sound-lightness. In Experiment 2, a speeded target discrimination task was used to assess whether the associations between sound attributes and color properties could invoke natural cross-modal correspondence and improve participants' cognitive efficiency in cognitive tasks. Several typical sound-color pairings were selected according to the results of Experiment 1. Participants were divided into two groups (congruent and incongruent). In each trial participants had to judge whether the presented color could appropriately be associated with the sound stimuli. Result revealed that participants responded more quickly and accurately in the congruent group than in the incongruent group. It was also found that there was no significant difference in reaction times and error rates between sound-hue and sound-lightness. The results of Experiment 1 and 2 indicate the existence of a robust crossmodal correspondence between multiple attributes of sound and color, which also has strong influence on cognitive tasks. The inconsistency of the reaction times between sound-hue and sound-lightness in Experiment 1 and 2 is probably owing to the difference in experimental protocol, which indicates that the complexity of experiment design may be an important factor in crossmodal correspondence phenomena.