Temporal ventriloquism effect in European starlings: Evidence for two parallel processing pathways

Reality and illusion: the assessment of angular separation of multi-modal signallers in a duetting bird

The spatial distribution of cooperating individuals plays a strategic role in territorial interactions of many group-living animals, and can indicate group cohesion. Vocalizations are commonly used to judge the distribution of signallers, but the spatial resolution of sounds is poor. Many species therefore accompany calls with movement; however, little is known about the role of audio-visual perception in natural interactions. We studied the effect of angular separation on the efficacy of multimodal duets in the Australian magpie-lark, Grallina cyanoleuca. We tested specifically whether conspicuous wing movements, which typically accompany duets, affect responses to auditory angular separation. Multimodal playbacks of duets using robotic models and speakers showed that birds relied primarily on acoustic cues when visual and auditory angular separations were congruent, but used both modalities to judge separation between the signallers when modalities were spatially incongruent. The visual component modified the effect of acoustic separation: robotic models that were apart weakened the response when speakers were together, while models that were together strengthened responses when speakers were apart. Our results show that responses are stronger when signallers are together, and suggest that males were are able to bind information cross-modally on the senders' spatial location, which is consistent with a multisensory illusion.

Adding colour-realistic video images to audio playbacks increases stimulus engagement but does not enhance vocal learning in zebra finches

Bird song and human speech are learned early in life and for both cases engagement with live social tutors generally leads to better learning outcomes than passive audio-only exposure. Real-world tutor–tutee relations are normally not uni- but multimodal and observations suggest that visual cues related to sound production might enhance vocal learning. We tested this hypothesis by pairing appropriate, colour-realistic, high frame-rate videos of a singing adult male zebra finch tutor with song playbacks and presenting these stimuli to juvenile zebra finches (Taeniopygia guttata). Juveniles exposed to song playbacks combined with video presentation of a singing bird approached the stimulus more often and spent more time close to it than juveniles exposed to audio playback only or audio playback combined with pixelated and time-reversed videos. However, higher engagement with the realistic audio–visual stimuli was not predictive of better song learning. Thus, although multimodality increased stimulus engagement and biologically relevant video content was more salient than colour and movement equivalent videos, the higher engagement with the realistic audio–visual stimuli did not lead to enhanced vocal learning. Whether the lack of three-dimensionality of a video tutor and/or the lack of meaningful social interaction make them less suitable for facilitating song learning than audio–visual exposure to a live tutor remains to be tested.

Violation of the Unity Assumption Disrupts Temporal Ventriloquism Effect in Starlings

When stimuli from different sensory modalities are received, they may be combined by the brain to form a multisensory percept. One key mechanism for multisensory binding is the unity assumption under which multisensory stimuli that share certain physical properties like temporal and/or spatial correspondence are grouped together as deriving from one object. In humans, evidence for a role of the unity assumption has been found in spatial tasks and also in temporal tasks using stimuli that share physical properties (speech-related stimuli, musical and synesthetically congruent stimuli). In our study, we investigate the role of the unity assumption in an animal model in a temporal order judgment task. When subjects are asked to indicate which of two spatially separated visual stimuli appeared first in time, performance improves when the visual stimuli are paired (in time) with spatially non-informative acoustic cues, a phenomenon known as the temporal ventriloquism effect. Here, we show that European starlings perform better when one singleton acoustic cue is paired with the first visual stimulus as compared to pairing with the second visual stimulus. This shows, in combination with our previous study, that a non-informative singleton acoustic cue, when temporally paired with the first visual stimulus, triggers alerting while, when temporally pairing with the second visual stimulus, it prevents a temporal ventriloquism effect because the unity assumption is violated. Thus, the unity assumption influences sensory perception not only in humans but also in an animal model. The importance of the unity assumption in this task supports the idea that the temporal ventriloquism effect, similar to the spatial ventriloquism effect, is based on multisensory binding and integration but not on alerting effects.