Pitch-based correspondences related to abstract concepts

Previous investigations have shown pitch-based correspondences with various perceptual and conceptual attributes. The present study reveals two novel pitch-based correspondences with highly abstract concepts. Three experiments with varying levels of implicitness of the association task showed that the concepts of future and in are associated with high-pitch sounds, while past and out are associated with low-pitch sounds. Hence, pitch-based correspondences can be observed even with temporal concepts that cannot be unambiguously represented in any perceptual format, at least, without spatial metaphorization. The correspondence effects were even more robust with the abstract temporal concepts of future/past than with more concrete spatial concepts of in/out. We propose that these effects might emerge from semantic multimodal abstraction processes mediated by affective dimensions of particular concepts.

Cross-modal congruency modulates evidence accumulation, not decision thresholds

Audiovisual cross-modal correspondences (CMCs) refer to the brain's inherent ability to subconsciously connect auditory and visual information. These correspondences reveal essential aspects of multisensory perception and influence behavioral performance, enhancing reaction times and accuracy. However, the impact of different types of CMCs-arising from statistical co-occurrences or shaped by semantic associations-on information processing and decision-making remains underexplored. This study utilizes the Implicit Association Test, where unisensory stimuli are sequentially presented and linked via CMCs within an experimental block by the specific response instructions (either congruent or incongruent). Behavioral data are integrated with EEG measurements through neurally informed drift-diffusion modeling to examine how neural activity across both auditory and visual trials is modulated by CMCs. Our findings reveal distinct neural components that differentiate between congruent and incongruent stimuli regardless of modality, offering new insights into the role of congruency in shaping multisensory perceptual decision-making. Two key neural stages were identified: an Early component enhancing sensory encoding in congruent trials and a Late component affecting evidence accumulation, particularly in incongruent trials. These results suggest that cross-modal congruency primarily influences the processing and accumulation of sensory information rather than altering decision thresholds.

Assessing the limits on size-pitch mapping reveals the interplay between top-down and bottom-up influences on relative crossmodal correspondences

Certain sensory dimensions, such as visual size and auditory pitch, are consistently associated, resulting in performance facilitation or inhibition. The mechanisms underlying these crossmodal correspondences are still the subject of debate: The relative or absolute nature of crossmodal mappings is connected to this debate, as an absolute mapping points to a bottom-up process, whereas a relative one is evidence of stronger top-down influences. Three experiments were conducted (including overall N = 207 participants), based on two different tasks, designed to explore a wide range of size-pitch crossmodal mappings. In Experiment 1, the participants were instructed to freely manipulate stimuli varing along a given dimension to 'match' the other. The results revealed evidence for a quasi-absolute mapping, but the correspondences shifted depending on the participants' auditory or visual attentional focus. In Experiment 2, the participants performed a visual speeded categorization task, involving a wide range of auditory task-irrelevant pitches, including the "preferred" ones, estimated on the basis of the results of Experiment 1. The results revealed a rather relative mapping, corroborating a top-down influence on the correspondence effect. Experiment 3 was designed to determine whether the relative mapping involved has boundary. The results confirmed that the larger the interval between pitches (i.e., more perceptually salient), the stronger the congruence effect, thus highlighting bottom-up facilitation. Taken together, these findings reveal that the size-pitch correspondences are sensitive to task-related top-down factors, as well as to stimulus-related bottom-up influences, ultimately revealing the adaptive nature of this kind of multisensory integration.

Acoustic features of instrumental movie soundtracks elicit distinct and mostly non-overlapping extra-musical meanings in the mind of the listener

Music can evoke powerful emotions in listeners. However, the role that instrumental music (music without any vocal part) plays in conveying extra-musical meaning, above and beyond emotions, is still a debated question. We conducted a study wherein participants (N = 121) listened to twenty 15-second-long excerpts of polyphonic instrumental soundtrack music and reported (i) perceived emotions (e.g., happiness, sadness) as well as (ii) movie scene properties imagined during listening (e.g., scene brightness, character role). We systematically investigated how acoustic features of instrumental soundtrack excerpts (e.g., tempo, loudness) contributed to mental imagery of movie scenes. We show distinct and mostly non-overlapping contributions of acoustic features to the imagination of properties of movie scene settings, characters, actions, and objects. Moreover, we find that negatively-valenced emotions fully mediate the relation between a subset of acoustic features and movie scene properties, providing evidence for the importance of emotional valence in evoking mental imagery. The data demonstrate the capacity of music to convey extra-musical semantic information through audition.

Crossmodal correspondence of elevation/pitch and size/pitch is driven by real-world features

Crossmodal correspondences are consistent associations between sensory features from different modalities, with some theories suggesting they may either reflect environmental correlations or stem from innate neural structures. This study investigates this question by examining whether retinotopic or representational features of stimuli induce crossmodal congruency effects. Participants completed an auditory pitch discrimination task paired with visual stimuli varying in their sensory (retinotopic) or representational (scene integrated) nature, for both the elevation/pitch and size/pitch correspondences. Results show that only representational visual stimuli produced crossmodal congruency effects on pitch discrimination. These results support an environmental statistics hypothesis, suggesting crossmodal correspondences rely on real-world features rather than on sensory representations.

What, if anything, can be considered an amodal sensory dimension?

The term 'amodal' is a key topic in several different research fields across experimental psychology and cognitive neuroscience, including in the areas of developmental and perception science. However, despite being regularly used in the literature, the term means something different to the researchers working in the different contexts. Many developmental scientists conceive of the term as referring to those perceptual qualities, such as, for example, the size and shape of an object, that can be picked up by multiple senses (e.g., vision and touch potentially providing information relevant to the same physical stimulus/property). However, the amodal label is also widely used in the case of those qualities that are not directly sensory, such as, for example, numerosity, rhythm, synchrony, etc. Cognitive neuroscientists, by contrast, tend to use the term amodal to refer to those central cognitive processes and brain areas that do not appear to be preferentially responsive to a particular sensory modality or to those symbolic or formal representations that essentially lack any modality and that are assumed to play a role in the higher processing of sensory information. Finally, perception scientists sometimes refer to the phenomenon of 'amodal completion', referring to the spontaneous completion of perceptual information that is missing when occluded objects are presented to observers. In this paper, we review the various different ways in which the term 'amodal' has been used in the literature and the evidence supporting the various uses of the term. Morever, we highlight some of the various properties that have been suggested to be 'amodal' over the years. Then, we try to address some of the questions that arise from the reviewed evidence, such as: Do different uses of the 'term' refer to different domains, for example, sensory information, perceptual processes, or perceptual representations? Are there any commonalities among the different uses of the term? To what extent is research on cross-modal associations (or correspondences) related to, or can shed light on, amodality? And how is the notion of amodal related to multisensory integration? Based on the reviewed evidence, it is argued that there is, as yet, no convincing empirical evidence to support the claim that amodal sensory qualities exist. We thus suggest that use of the term amodal would be more meaningful with respect to abstract cognition rather than necessarily sensory perception, the latter being more adequately explained/understood in terms of highly redundant cross-modal correspondences.

Sound symbolism in manual and vocal responses: phoneme-response interactions associated with grasping as well as vertical and size dimensions of keypresses

It has been shown that reading the vowel [i] and consonant [t] facilitates precision grip responses, while [ɑ] and [k] are associated with faster power grip responses. A similar effect has been observed when participants perform responses with small or large response keys. The present study investigated whether the vowels and consonants could produce different effects with the grip responses and keypresses when the speech units are read aloud (Experiment 1) or silently (Experiment 2). As a second objective, the study investigated whether the recently observed effect, in which the upper position of a visual stimulus is associated with faster vocalizations of the high vowel and the lower position is associated with the low vowel, can be observed in manual responses linking, for example, the [i] with responses of the upper key and [ɑ] with lower responses. Firstly, the study showed that when the consonants are overtly articulated, the interaction effect can be observed only with the grip responses, while the vowel production was shown to systematically influence small/large keypresses, as well as precision/power grip responses. Secondly, the vowel [i] and consonant [t] were associated with the upper responses, while [ɑ] and [k] were associated with the lower responses, particularly in the overt articulation task. The paper delves into the potential sound-symbolic implications of these phonetic elements, suggesting that their acoustic and articulatory characteristics might implicitly align them with specific response magnitudes, vertical positions, and grip types.

Should absolute pitch be considered as a unique kind of absolute sensory judgment in humans? A systematic and theoretical review of the literature

Absolute pitch is the name given to the rare ability to identify a musical note in an automatic and effortless manner without the need for a reference tone. Those individuals with absolute pitch can, for example, name the note they hear, identify all of the tones of a given chord, and/or name the pitches of everyday sounds, such as car horns or sirens. Hence, absolute pitch can be seen as providing a rare example of absolute sensory judgment in audition. Surprisingly, however, the intriguing question of whether such an ability presents unique features in the domain of sensory perception, or whether instead similar perceptual skills also exist in other sensory domains, has not been explicitly addressed previously. In this paper, this question is addressed by systematically reviewing research on absolute pitch using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) method. Thereafter, we compare absolute pitch with two rare types of sensory experience, namely synaesthesia and eidetic memory, to understand if and how these phenomena exhibit similar features to absolute pitch. Furthermore, a common absolute perceptual ability that has been often compared to absolute pitch, namely colour perception, is also discussed. Arguments are provided supporting the notion that none of the examined abilities can be considered like absolute pitch. Therefore, we conclude by suggesting that absolute pitch does indeed appear to constitute a unique kind of absolute sensory judgment in humans, and we discuss some open issues and novel directions for future research in absolute pitch.

High-Pitched Sound is Open and Low-Pitched Sound is Closed: Representing the Spatial Meaning of Pitch Height

Research shows that high- and low-pitch sounds can be associated with various meanings. For example, high-pitch sounds are associated with small concepts, whereas low-pitch sounds are associated with large concepts. This study presents three experiments revealing that high-pitch sounds are also associated with open concepts and opening hand actions, while low-pitch sounds are associated with closed concepts and closing hand actions. In Experiment 1, this sound-meaning correspondence effect was shown using the two-alternative forced-choice task, while Experiments 2 and 3 used reaction time tasks to show this interaction. In Experiment 2, high-pitch vocalizations were found to facilitate opening hand gestures, and low-pitch vocalizations were found to facilitate closing hand gestures, when performed simultaneously. In Experiment 3, high-pitched vocalizations were produced particularly rapidly when the visual target stimulus presented an open object, and low-pitched vocalizations were produced particularly rapidly when the target presented a closed object. These findings are discussed concerning the meaning of intonational cues. They are suggested to be based on cross-modally representing conceptual spatial knowledge in sensory, motor, and affective systems. Additionally, this pitch-opening effect might share cognitive processes with other pitch-meaning effects.

Humans (Homo sapiens) but not baboons (Papio papio) demonstrate crossmodal pitch-luminance correspondence

Humans spontaneously and consistently map information coming from different sensory modalities. Surprisingly, the phylogenetic origin of such cross-modal correspondences has been under-investigated. A notable exception is the study of Ludwig et al. (Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees [Pan troglodytes] and humans. Proceedings of the National Academy of Sciences, 108(51), 20661-20665) which reports that both humans and chimpanzees spontaneously map high-pitched sounds with bright objects and low-pitched sounds with dark objects. Our pre-registered study aimed to directly replicate this research on both humans and baboons (Papio papio), an old world monkey which is more phylogenetically distant from humans than chimpanzees. Following Ludwig et al. participants were presented with a visual classification task where they had to sort black and white square (low and high luminance), while background sounds (low or high-pitched tones) were playing. Whereas we replicated the finding that humans' performance on the visual task was affected by congruency between sound and luminance of the target, we did not find any of those effects on baboons' performance. These results question the presence of a shared cross-modal pitch-luminance mapping in other nonhuman primates.

Sensory translation between audition and vision

Across the millennia, and across a range of disciplines, there has been a widespread desire to connect, or translate between, the senses in a manner that is meaningful, rather than arbitrary. Early examples were often inspired by the vivid, yet mostly idiosyncratic, crossmodal matches expressed by synaesthetes, often exploited for aesthetic purposes by writers, artists, and composers. A separate approach comes from those academic commentators who have attempted to translate between structurally similar dimensions of perceptual experience (such as pitch and colour). However, neither approach has succeeded in delivering consensually agreed crossmodal matches. As such, an alternative approach to sensory translation is needed. In this narrative historical review, focusing on the translation between audition and vision, we attempt to shed light on the topic by addressing the following three questions: (1) How is the topic of sensory translation related to synaesthesia, multisensory integration, and crossmodal associations? (2) Are there common processing mechanisms across the senses that can help to guarantee the success of sensory translation, or, rather, is mapping among the senses mediated by allegedly universal (e.g., amodal) stimulus dimensions? (3) Is the term 'translation' in the context of cross-sensory mappings used metaphorically or literally? Given the general mechanisms and concepts discussed throughout the review, the answers we come to regarding the nature of audio-visual translation are likely to apply to the translation between other perhaps less-frequently studied modality pairings as well.

Does cross-modal correspondence modulate modality-specific perceptual processing? Study using timing judgment tasks

Cross-modal correspondences refer to associations between stimulus features across sensory modalities. Previous studies have shown that cross-modal correspondences modulate reaction times for detecting and identifying stimuli in one modality when uninformative stimuli from another modality are present. However, it is unclear whether such modulation reflects changes in modality-specific perceptual processing. We used two psychophysical timing judgment tasks to examine the effects of audiovisual correspondences on visual perceptual processing. In Experiment 1, we conducted a temporal order judgment (TOJ) task that asked participants to judge which of two visual stimuli presented with various stimulus onset asynchronies (SOAs) appeared first. In Experiment 2, we conducted a simultaneous judgment (SJ) task that asked participants to report whether the two visual stimuli were simultaneous or successive. We also presented an unrelated auditory stimulus, simultaneously or preceding the first visual stimulus, and manipulated the congruency between audiovisual stimuli. Experiment 1 indicated that the points of subjective simultaneity (PSSs) between the two visual stimuli estimated in the TOJ task shifted according to the audiovisual correspondence between the auditory pitch and visual features of vertical location and size. However, these audiovisual correspondences did not affect PSS estimated using the SJ task in Experiment 2. The different results of the two tasks can be explained through the response bias triggered by audiovisual correspondence that only the TOJ task included. We concluded that audiovisual correspondence would not modulate visual perceptual timing and that changes in modality-specific perceptual processing might not trigger the congruency effects reported in previous studies.

How many categories are there in crossmodal correspondences? A study based on exploratory factor analysis

Humans naturally associate stimulus features of one sensory modality with those of other modalities, such as associating bright light with high-pitched tones. This phenomenon is called crossmodal correspondence and is found between various stimulus features, and has been suggested to be categorized into several types. However, it is not yet clear whether there are differences in the underlying mechanism between the different kinds of correspondences. This study used exploratory factor analysis to address this question. Through an online experiment platform, we asked Japanese adult participants (Experiment 1: N = 178, Experiment 2: N = 160) to rate the degree of correspondence between two auditory and five visual features. The results of two experiments revealed that two factors underlie the subjective judgments of the audiovisual crossmodal correspondences: One factor was composed of correspondences whose auditory and visual features can be expressed in common Japanese terms, such as the loudness-size and pitch-vertical position correspondences, and another factor was composed of correspondences whose features have no linguistic similarities, such as pitch-brightness and pitch-shape correspondences. These results confirm that there are at least two types of crossmodal correspondences that are likely to differ in terms of language mediation.

Sensitive to music? Examining the crossmodal effect of audition on sweet taste sensitivity

Previous research has shown that music can influence taste perception. While most studies to date have focused on taste intensity ratings, less is known about the influence of musical stimuli on other parameters of taste function. In this within-subjects experiment (N = 73), we tested the effects of three sound conditions (High Sweetness soundtrack - HS; Low Sweetness soundtrack - LS; and Silence - S) on sweet taste sensitivity, namely, detection and recognition. Each participant tasted nine samples of sucrose solutions (from 0 g/L to 20 g/L) under each of the three sound conditions in counterbalanced order. We assessed the lower concentrations at which participants were able to detect (detection threshold) and correctly identify (recognition threshold) a taste sensation. Additionally, the intensity and hedonic ratings of samples above the recognition threshold (7.20 g/L) were analyzed. Affective variations (valence and arousal) in response to the sound conditions were also assessed. Although music did not lead to significant differences in mean detection and recognition thresholds, a larger proportion of sweet taste recognitions was observed at a near-threshold level (2.59 g/L) in the HS condition. The intensity and hedonic ratings of supra-threshold conditions were unaffected by the music condition. Significant differences in self-reported mood in response to the sound conditions were also observed. The present study suggests that the influence of music on the sweet taste perception of basic solutions may depend on the parameter under consideration.

Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration

Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the "recalibration" of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

Coloured hearing, colour music, colour organs, and the search for perceptually meaningful correspondences between colour and sound

There has long been interest in the nature of the relationship(s) between hue and pitch or, in other words, between colour and musical/pure tones, stretching back at least as far as Newton, Goethe, Helmholtz, and beyond. In this narrative historical review, we take a closer look at the motivations that have lain behind the various assertions that have been made in the literature concerning the analogies, and possible perceptual similarities, between colour and sound. During the last century, a number of experimental psychologists have also investigated the nature of the correspondence between these two primary dimensions of perceptual experience. The multitude of different crossmodal mappings that have been put forward over the centuries are summarized, and a distinction drawn between physical/structural and psychological correspondences. The latter being further sub-divided into perceptual and affective categories. Interest in physical correspondences has typically been motivated by the structural similarities (analogous mappings) between the organization of perceptible dimensions of auditory and visual experience. Emphasis has been placed both on the similarity in terms of the number of basic categories into which pitch and colour can be arranged and also on the fact that both can be conceptualized as circular dimensions. A distinction is drawn between those commentators who have argued for a dimensional alignment of pitch and hue (based on a structural mapping), and those who appear to have been motivated by the existence of specific correspondences between particular pairs of auditory and visual stimuli instead (often, as we will see, based on the idiosyncratic correspondences that have been reported by synaesthetes). Ultimately, though, the emotional-mediation account would currently appear to provide the most parsimonious account for whatever affinity the majority of people experience between musical sounds and colour.

Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Effects of pitch and musical sounds on body-representations when moving with sound

The effects of music on bodily movement and feelings, such as when people are dancing or engaged in physical activity, are well-documented—people may move in response to the sound cues, feel powerful, less tired. How sounds and bodily movements relate to create such effects? Here we deconstruct the problem and investigate how different auditory features affect people’s body-representation and feelings even when paired with the same movement. In three experiments, participants executed a simple arm raise synchronised with changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2) and within different frequency ranges (Experiment 3), while we recorded indirect and direct measures on their movement, body-representations and feelings. Changes in pitch influenced people’s general emotional state as well as the various bodily dimensions investigated—movement, proprioceptive awareness and feelings about one’s body and movement. Adding harmonic content amplified the differences between ascending and descending sounds, while shifting the absolute frequency range had a general effect on movement amplitude, bodily feelings and emotional state. These results provide new insights in the role of auditory and musical features in dance and exercise, and have implications for the design of sound-based applications supporting movement expression, physical activity, or rehabilitation.

High-pitch sounds small for domestic dogs: abstract crossmodal correspondences between auditory pitch and visual size

Humans possess intuitive associations linking certain non-redundant features of stimuli-e.g. high-pitched sounds with small object size (or similarly, low-pitched sounds with large object size). This phenomenon, known as crossmodal correspondence, has been identified in humans across multiple different senses. There is some evidence that non-human animals also form crossmodal correspondences, but the known examples are mostly limited to the associations between the pitch of vocalizations and the size of callers. To investigate whether domestic dogs, like humans, show abstract pitch-size association, we first trained dogs to approach and touch an object after hearing a sound emanating from it. Subsequently, we repeated the task but presented dogs with two objects differing in size, only one of which was playing a sound. The sound was either high or low pitched, thereby creating trials that were either congruent (high pitch from small object; low pitch from large objects) or incongruent (the reverse). We found that dogs reacted faster on congruent versus incongruent trials. Moreover, their accuracy was at chance on incongruent trials, but significantly above chance for congruent trials. Our results suggest that non-human animals show abstract pitch sound correspondences, indicating these correspondences may not be uniquely human but rather a sensory processing feature shared by other species.

High-pitch sounds small for domestic dogs: abstract crossmodal correspondences between auditory pitch and visual size

Humans possess intuitive associations linking certain non-redundant features of stimuli-e.g. high-pitched sounds with small object size (or similarly, low-pitched sounds with large object size). This phenomenon, known as crossmodal correspondence, has been identified in humans across multiple different senses. There is some evidence that non-human animals also form crossmodal correspondences, but the known examples are mostly limited to the associations between the pitch of vocalizations and the size of callers. To investigate whether domestic dogs, like humans, show abstract pitch-size association, we first trained dogs to approach and touch an object after hearing a sound emanating from it. Subsequently, we repeated the task but presented dogs with two objects differing in size, only one of which was playing a sound. The sound was either high or low pitched, thereby creating trials that were either congruent (high pitch from small object; low pitch from large objects) or incongruent (the reverse). We found that dogs reacted faster on congruent versus incongruent trials. Moreover, their accuracy was at chance on incongruent trials, but significantly above chance for congruent trials. Our results suggest that non-human animals show abstract pitch sound correspondences, indicating these correspondences may not be uniquely human but rather a sensory processing feature shared by other species.

Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

When exposed to complementary features of information across sensory modalities, our brains formulate cross-modal associations between features of stimuli presented separately to multiple modalities. For example, auditory pitch-visual size associations map high-pitch tones with small-size visual objects, and low-pitch tones with large-size visual objects. Preferential, or congruent, cross-modal associations have been shown to affect behavioural performance, i.e. choice accuracy and reaction time (RT) across multisensory decision-making paradigms. However, the neural mechanisms underpinning such influences in perceptual decision formation remain unclear. Here, we sought to identify when perceptual improvements from associative congruency emerge in the brain during decision formation. In particular, we asked whether such improvements represent ‘early’ sensory processing benefits, or ‘late’ post-sensory changes in decision dynamics. Using a modified version of the Implicit Association Test (IAT), coupled with electroencephalography (EEG), we measured the neural activity underlying the effect of auditory stimulus-driven pitch-size associations on perceptual decision formation. Behavioural results showed that participants responded significantly faster during trials when auditory pitch was congruent, rather than incongruent, with its associative visual size counterpart. We used multivariate Linear Discriminant Analysis (LDA) to characterise the spatiotemporal dynamics of EEG activity underpinning IAT performance. We found an ‘Early’ component (∼100–110 ms post-stimulus onset) coinciding with the time of maximal discrimination of the auditory stimuli, and a ‘Late’ component (∼330–340 ms post-stimulus onset) underlying IAT performance. To characterise the functional role of these components in decision formation, we incorporated a neurally-informed Hierarchical Drift Diffusion Model (HDDM), revealing that the Late component decreases response caution, requiring less sensory evidence to be accumulated, whereas the Early component increased the duration of sensory-encoding processes for incongruent trials. Overall, our results provide a mechanistic insight into the contribution of ‘early’ sensory processing, as well as ‘late’ post-sensory neural representations of associative congruency to perceptual decision formation.

Commercializing Sonic Seasoning in Multisensory Offline Experiential Events and Online Tasting Experiences

The term "sonic seasoning" refers to the deliberate pairing of sound/music with taste/flavour in order to enhance, or modify, the multisensory tasting experience. Although the recognition that people experience a multitude of crossmodal correspondences between stimuli in the auditory and chemical senses originally emerged from the psychophysics laboratory, the last decade has seen an explosion of interest in the use and application of sonic seasoning research findings, in a range of multisensory experiential events and online offerings. These marketing-led activations have included a variety of different approaches, from curating pre-composed music selections that have the appropriate sonic qualities (such as pitch or timbre), to the composition of bespoke music/soundscapes that match the specific taste/flavour of particular food or beverage products. Moreover, given that our experience of flavour often changes over time and frequently contains multiple distinct elements, there is also scope to more closely match the sonic seasoning to the temporal evolution of the various components (or notes) of the flavour experience. We review a number of case studies of the use of sonic seasoning, highlighting some of the challenges and opportunities associated with the various approaches, and consider the intriguing interplay between physical and digital (online) experiences. Taken together, the various examples reviewed here help to illustrate the growing commercial relevance of sonic seasoning research.

Musical Scents: On the Surprising Absence of Scented Musical/Auditory Events, Entertainments, and Experiences

The matching of scents with music is both one of the most natural (or intuitive) of crossmodal correspondences and, at the same time, one of the least frequently explored combinations of senses in an entertainment and multisensory experiential design context. This narrative review highlights the various occasions over the last century or two when scents and sounds have coincided, and the various motivations behind those who have chosen to bring these senses together: This has included everything from the masking of malodour to the matching of the semantic meaning or arousal potential of the two senses, through to the longstanding and recently-reemerging interest in the crossmodal correspondences (now that they have been distinguished from the superficially similar phenomenon of synaesthesia, with which they were previously often confused). As such, there exist a number of ways in which these two senses can be incorporated into meaningful multisensory experiences that can potentially resonate with the public. Having explored the deliberate combination of scent and music (or sound) in everything from "scent-sory" marketing through to fragrant discos and olfactory storytelling, I end by summarizing some of the opportunities around translating such unusual multisensory experiences from the public to the private sphere. This will likely be via the widespread dissemination of sensory apps that promise to convert (or translate) from one sense (likely scent) to another (e.g., music), as has, for example already started to occur in the world of music selections to match the flavour of specific wines.

The Lightness/Pitch Crossmodal Correspondence Modulates the Rubin Face/Vase Perception

We examine whether crossmodal correspondences (CMCs) modulate perceptual disambiguation by considering the influence of lightness/pitch congruency on the perceptual resolution of the Rubin face/vase (RFV). We randomly paired a black-and-white RFV (black faces and white vase, or vice versa) with either a high or low pitch tone and found that CMC congruency biases the dominant visual percept. The perceptual option that was CMC-congruent with the tone (white/high pitch or black/low pitch) was reported significantly more often than the perceptual option CMC-incongruent with the tone (white/low pitch or black/high pitch). However, the effect was only observed for stimuli presented for longer and not shorter durations suggesting a perceptual effect rather than a response bias, and moreover, we infer an effect on perceptual reversals rather than initial percepts. We found that the CMC congruency effect for longer-duration stimuli only occurred after prior exposure to the stimuli of several minutes, suggesting that the CMC congruency develops over time. These findings extend the observed effects of CMCs from relatively low-level feature-based effects to higher-level object-based perceptual effects (specifically, resolving ambiguity) and demonstrate that an entirely new category of crossmodal factors (CMC congruency) influence perceptual disambiguation in bistability.

Blending Emotions and Cross-Modality in Sonic Seasoning: Towards Greater Applicability in the Design of Multisensory Food Experiences

Sonic seasoning refers to the way in which music can influence multisensory tasting experiences. To date, the majority of the research on sonic seasoning has been conducted in Europe or the USA, typically in a within-participants experimental context. In the present study, we assessed the applicability of sonic seasoning in a large-scale between-participants setting in Asia. A sample of 1611 participants tasted one sample of chocolate while listening to a song that evoked a specific combination of cross-modal and emotional consequences. The results revealed that the music’s emotional character had a more prominent effect than its cross-modally corresponding attributes on the multisensory tasting experience. Participants expressed a higher buying intention for the chocolate and rated it as having a softer texture when listening to mainly positive (as compared to mainly negative) music. The chocolates were rated as having a more intense flavor amongst those participants listening to ‘softer’ as compared to ‘harder’ music. Therefore, the present study demonstrates that music is capable of triggering a combination of specific cross-modal and emotional effects in the multisensory tasting experience of a chocolate.

Olfactory-colour crossmodal correspondences in art, science, and design

The last few years have seen a rapid growth of interest amongst researchers in the crossmodal correspondences. One of the correspondences that has long intrigued artists is the putative association between colours and odours. While traditionally conceptualised in terms of synaesthesia, over the last quarter century or so, at least 20 published peer-reviewed articles have assessed the consistent, and non-random, nature of the colours that people intuitively associate with specific (both familiar and unfamiliar) odours in a non-food context. Having demonstrated such consistent mappings amongst the general (i.e. non-synaesthetic) population, researchers have now started to investigate whether they are shared cross-culturally, and to document their developmental acquisition. Over the years, several different explanations have been put forward by researchers for the existence of crossmodal correspondences, including the statistical, semantic, structural, and emotional-mediation accounts. While several of these approaches would appear to have some explanatory validity as far as the odour-colour correspondences are concerned, contemporary researchers have focussed on learned associations as the dominant explanatory framework. The nature of the colour-odour associations that have been reported to date appear to depend on the familiarity of the odour and the ease of source naming, and hence the kind of association/representation that is accessed. While the bidirectionality of odour-colour correspondences has not yet been rigorously assessed, many designers are nevertheless already starting to build on odour-colour crossmodal correspondences in their packaging/labelling/branding work.

Temperature-Based Crossmodal Correspondences: Causes and Consequences

The last few years have seen an explosive growth of research interest in the crossmodal correspondences, the sometimes surprising associations that people experience between stimuli, attributes, or perceptual dimensions, such as between auditory pitch and visual size, or elevation. To date, the majority of this research has tended to focus on audiovisual correspondences. However, a variety of crossmodal correspondences have also been demonstrated with tactile stimuli, involving everything from felt shape to texture, and from weight through to temperature. In this review, I take a closer look at temperature-based correspondences. The empirical research not only supports the existence of robust crossmodal correspondences between temperature and colour (as captured by everyday phrases such as 'red hot') but also between temperature and auditory pitch. Importantly, such correspondences have (on occasion) been shown to influence everything from our thermal comfort in coloured environments through to our response to the thermal and chemical warmth associated with stimulation of the chemical senses, as when eating, drinking, and sniffing olfactory stimuli. Temperature-based correspondences are considered in terms of the four main classes of correspondence that have been identified to date, namely statistical, structural, semantic, and affective. The hope is that gaining a better understanding of temperature-based crossmodal correspondences may one day also potentially help in the design of more intuitive sensory-substitution devices, and support the delivery of immersive virtual and augmented reality experiences.

Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as "maluma" and "takete" with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.

Crossmodal Visuospatial Effects on Auditory Perception of Musical Contour

The crossmodal correspondence between auditory pitch and visuospatial elevation (in which high- and low-pitched tones are associated with high and low spatial elevation respectively) has been proposed as the basis for Western musical notation. One implication of this is that music perception engages visuospatial processes and may not be exclusively auditory. Here, we investigated how music perception is influenced by concurrent visual stimuli. Participants listened to unfamiliar five-note musical phrases with four kinds of pitch contour (rising, falling, rising-falling, or falling-rising), accompanied by incidental visual contours that were either congruent (e.g., auditory rising/visual rising) or incongruent (e.g., auditory rising/visual falling) and judged whether the final note of the musical phrase was higher or lower in pitch than the first. Response times for the auditory judgment were significantly slower for incongruent compared to congruent trials, i.e., there was a congruency effect, even though the visual contours were incidental to the auditory task. These results suggest that music perception, although generally regarded as an auditory experience, may actually be multisensory in nature.

Assessing the Role of Emotional Mediation in Explaining Crossmodal Correspondences Involving Musical Stimuli

A wide variety of crossmodal correspondences, defined as the often surprising connections that people appear to experience between simple features, attributes, or dimensions of experience, either physically present or else merely imagined, in different sensory modalities, have been demonstrated in recent years. However, a number of crossmodal correspondences have also been documented between more complex (i.e., multi-component) stimuli, such as, for example, pieces of music and paintings. In this review, the extensive evidence supporting the emotional mediation account of the crossmodal correspondences between musical stimuli (mostly pre-recorded short classical music excerpts) and visual stimuli, including colour patches through to, on occasion, paintings, is critically evaluated. According to the emotional mediation account, it is the emotional associations that people have with stimuli that constitutes one of the fundamental bases on which crossmodal associations are established. Taken together, the literature that has been published to date supports emotional mediation as one of the key factors underlying the crossmodal correspondences involving emotionally-valenced stimuli, both simple and complex.

Heaviness-brightness correspondence and stimulus-response compatibility

Cross-sensory correspondences can reflect crosstalk between aligned conceptual feature dimensions, though uncertainty remains regarding the identities of all the dimensions involved. It is unclear, for example, if heaviness contributes to correspondences separately from size. Taking steps to dissociate variations in heaviness from variations in size, the question was asked if a heaviness-brightness correspondence will induce a congruity effect during the speeded brightness classification of simple visual stimuli. Participants classified the stimuli according to whether they were brighter or darker than the mid-gray background against which they appeared. They registered their speeded decisions by manipulating (e.g., tapping) the object they were holding in either their left or right hand (e.g., left for bright, right for dark). With these two otherwise identical objects contrasting in their weight, stimuli were classified more quickly when the relative heaviness of the object needing to be manipulated corresponded with the brightness of the stimulus being classified (e.g., the heavier object for a darker stimulus). This novel congruity effect, in the guise of a stimulus-response (S-R) compatibility effect, was induced when heaviness was isolated as an enduring feature of the object needing to be manipulated. It was also undiminished when participants completed a concurrent verbal memory load task, countering claims that the heaviness-brightness correspondence is verbally mediated. Heaviness, alongside size, appears to contribute to cross-sensory correspondences in its own right and in a manner confirming the far-reaching influence of correspondences, extending here to the fluency with which people communicate simple ideas by manipulating a hand-held object.

Coarse to Fine Audio-Visual Size Correspondences Develop During Primary School Age

Developmental studies have shown that children can associate visual size with non-visual and apparently unrelated stimuli, such as pure tone frequencies. Most research to date has focused on audio-visual size associations by showing that children can associate low pure tone frequencies with large objects, and high pure tone frequencies with small objects. Researchers relate these findings to coarser association, i.e., less precise associations for which binary categories of stimuli are used such as in the case of low versus high frequencies and large versus small visual stimuli. This study investigates how finer, more precise, crossmodal audio-visual associations develop during primary school age (from 6 to 11 years old). To unveil such patterns, we took advantage of a range of auditory pure tones and tested how primary school children match sounds with visually presented shapes. We tested 66 children (6-11 years old) in an audio-visual matching task involving a range of pure tone frequencies. Visual stimuli were circles or angles of different sizes. We asked participants to indicate the shape matching the sound. All children associated large objects/angles with low pitch, and small objects/angles with high pitch sounds. Interestingly, older children made greater use of intermediate visual sizes to provide their responses. Indeed, audio-visual associations for finer differences between stimulus features such as size and pure tone frequencies, may develop later depending on the maturation of supramodal size perception processes. Considering our results, we suggest that audio-visual size correspondences can be used for educational purposes by aiding the discrimination of sizes, including angles of different aperture. Moreover, their use should be shaped according to children's specific developmental stage.

A Sweet Voice: The Influence of Cross-Modal Correspondences Between Taste and Vocal Pitch on Advertising Effectiveness

We have seen a rapid growth of interest in cross-modal correspondences between sound and taste over recent years. People consistently associate higher-pitched sounds with sweet/sour foods, while lower-pitched sounds tend to be associated with bitter foods. The human voice is key in broadcast advertising, and the role of voice in communication generally is partly characterized by acoustic parameters of pitch. However, it remains unknown whether voice pitch and taste interactively influence consumer behavior. Since consumers prefer congruent sensory information, it is plausible that voice pitch and taste interactively influence consumers’ responses to advertising stimuli. Based on the cross-modal correspondence phenomenon, this study aimed to elucidate the role played by voice pitch–taste correspondences in advertising effectiveness. Participants listened to voiceover advertisements (at a higher or lower pitch than the original narrator’s voice) for three food products with distinct tastes (sweet, sour, and bitter) and rated their buying intention (an indicator of advertising effectiveness). The results show that the participants were likely to exhibit greater buying intention toward both sweet and sour food when they listened to higher-pitched (vs lower-pitched) voiceover advertisements. The influence of a higher pitch on sweet and sour food preferences was observed in only two of the three studies: studies 1 and 2 for sour food, and studies 2 and 3 for sweet food. These findings emphasize the role that voice pitch–taste correspondence plays in preference formation, and advance the applicability of cross-modal correspondences to business.

Not Just Another Pint! The Role of Emotion Induced by Music on the Consumer's Tasting Experience

We introduce a novel methodology to assess the influence of the emotion induced by listening to music on the consumer's multisensory tasting experience. These crossmodal effects were analyzed when two contrasting music tracks (positive vs negative emotion) were presented to consumers while tasting beer. The results suggest that the emotional reactions triggered by the music influenced specific aspects of the multisensory tasting experience. Participants liked the beer more, and rated it as tasting sweeter, when listening to music associated with positive emotion. The same beer was rated as more bitter, with higher alcohol content, and as having more body, when the participants listened to music associated with negative emotion. Moreover, participants were willing to pay 7-8% more for the beer that was tasted while they listened to positive music. This novel methodology was subsequently replicated with two different styles of beer. These results are discussed along with practical implications concerning the way in which music can add significant value to how a consumer responds to a brand.

Taste the Bass: Low Frequencies Increase the Perception of Body and Aromatic Intensity in Red Wine

Associations between heaviness and bass/low-pitched sounds reverberate throughout music, philosophy, literature, and language. Given that recent research into the field of cross-modal correspondences has revealed a number of robust relationships between sound and flavour, this exploratory study was designed to investigate the effects of lower frequency sound (10 Hz to 200 Hz) on the perception of the mouthfeel character of palate weight/body. This is supported by an overview of relevant cross-modal studies and cultural production. Wines were the tastants - a New Zealand Pinot Noir and a Spanish Garnacha - which were tasted in silence and with a 100 Hz (bass) and a higher 1000 Hz sine wave tone. Aromatic intensity was included as an additional character given suggestions that pitch may influence the perception of aromas, which might presumably affect the perception of wine body. Intensity of acidity and liking were also evaluated. The results revealed that the Pinot Noir wine was rated as significantly fuller-bodied when tasted with a bass frequency than in silence or with a higher frequency sound. The low frequency stimulus also resulted in the Garnacha wine being rated as significantly more aromatically intense than when tasted in the presence of the higher frequency auditory stimulus. Acidity was rated considerably higher with the higher frequency in both wines by those with high wine familiarity and the Pinot Noir significantly better liked than the Garnacha. Possible reasons as to why the tones used in this study affected perception of the two wines differently are discussed. Practical application of the findings are also proposed.