Crossmodal association between visual and acoustic cues in a tortoise (Testudo hermanni)

Assessing the limits on size-pitch mapping reveals the interplay between top-down and bottom-up influences on relative crossmodal correspondences

Certain sensory dimensions, such as visual size and auditory pitch, are consistently associated, resulting in performance facilitation or inhibition. The mechanisms underlying these crossmodal correspondences are still the subject of debate: The relative or absolute nature of crossmodal mappings is connected to this debate, as an absolute mapping points to a bottom-up process, whereas a relative one is evidence of stronger top-down influences. Three experiments were conducted (including overall N = 207 participants), based on two different tasks, designed to explore a wide range of size-pitch crossmodal mappings. In Experiment 1, the participants were instructed to freely manipulate stimuli varing along a given dimension to 'match' the other. The results revealed evidence for a quasi-absolute mapping, but the correspondences shifted depending on the participants' auditory or visual attentional focus. In Experiment 2, the participants performed a visual speeded categorization task, involving a wide range of auditory task-irrelevant pitches, including the "preferred" ones, estimated on the basis of the results of Experiment 1. The results revealed a rather relative mapping, corroborating a top-down influence on the correspondence effect. Experiment 3 was designed to determine whether the relative mapping involved has boundary. The results confirmed that the larger the interval between pitches (i.e., more perceptually salient), the stronger the congruence effect, thus highlighting bottom-up facilitation. Taken together, these findings reveal that the size-pitch correspondences are sensitive to task-related top-down factors, as well as to stimulus-related bottom-up influences, ultimately revealing the adaptive nature of this kind of multisensory integration.

Crossmodal correspondence of elevation/pitch and size/pitch is driven by real-world features

Crossmodal correspondences are consistent associations between sensory features from different modalities, with some theories suggesting they may either reflect environmental correlations or stem from innate neural structures. This study investigates this question by examining whether retinotopic or representational features of stimuli induce crossmodal congruency effects. Participants completed an auditory pitch discrimination task paired with visual stimuli varying in their sensory (retinotopic) or representational (scene integrated) nature, for both the elevation/pitch and size/pitch correspondences. Results show that only representational visual stimuli produced crossmodal congruency effects on pitch discrimination. These results support an environmental statistics hypothesis, suggesting crossmodal correspondences rely on real-world features rather than on sensory representations.

What, if anything, can be considered an amodal sensory dimension?

The term 'amodal' is a key topic in several different research fields across experimental psychology and cognitive neuroscience, including in the areas of developmental and perception science. However, despite being regularly used in the literature, the term means something different to the researchers working in the different contexts. Many developmental scientists conceive of the term as referring to those perceptual qualities, such as, for example, the size and shape of an object, that can be picked up by multiple senses (e.g., vision and touch potentially providing information relevant to the same physical stimulus/property). However, the amodal label is also widely used in the case of those qualities that are not directly sensory, such as, for example, numerosity, rhythm, synchrony, etc. Cognitive neuroscientists, by contrast, tend to use the term amodal to refer to those central cognitive processes and brain areas that do not appear to be preferentially responsive to a particular sensory modality or to those symbolic or formal representations that essentially lack any modality and that are assumed to play a role in the higher processing of sensory information. Finally, perception scientists sometimes refer to the phenomenon of 'amodal completion', referring to the spontaneous completion of perceptual information that is missing when occluded objects are presented to observers. In this paper, we review the various different ways in which the term 'amodal' has been used in the literature and the evidence supporting the various uses of the term. Morever, we highlight some of the various properties that have been suggested to be 'amodal' over the years. Then, we try to address some of the questions that arise from the reviewed evidence, such as: Do different uses of the 'term' refer to different domains, for example, sensory information, perceptual processes, or perceptual representations? Are there any commonalities among the different uses of the term? To what extent is research on cross-modal associations (or correspondences) related to, or can shed light on, amodality? And how is the notion of amodal related to multisensory integration? Based on the reviewed evidence, it is argued that there is, as yet, no convincing empirical evidence to support the claim that amodal sensory qualities exist. We thus suggest that use of the term amodal would be more meaningful with respect to abstract cognition rather than necessarily sensory perception, the latter being more adequately explained/understood in terms of highly redundant cross-modal correspondences.