Multisensory integration and cross-modal learning in synaesthesia: A unifying model

Multistage maturation optimizes vision

Late development of color vision improves object recognition.

Cognitive and brain correlates of acquired number-colour synaesthetic-like associations

Synaesthesia is a condition in which one sensory dimension triggers another sensation. The exact contribution of genetic and environmental factors in synaesthesia is not yet fully understood. Most synaesthesia phenotypes involve associations in which the synaesthetic inducer constitutes some form of linguistic/conceptual information acquired during the course of development (e.g., digits, letters, and names of months). To study the role of learning in synaesthesia, we induced synaesthesia by training and examined the behavioural and brain correlates of number-colour associations. We took advantage of the well-known behavioural and neural signature of numbers and used number symbols as inducers. Short (2 weeks) and long (4 weeks) training protocols were conducted with two different groups. Task-related BOLD response was acquired while participants performed Stroop tasks requiring naming colours while ignoring the stimuli (i.e., number symbols, dots, words). If the arbitrary association involving number-colour is automatic, the irrelevant dimension (i.e., numbers) would interfere with the colour response. In addition, if number-colour associations are transferred to linguistic and non-symbolic representations, the passive viewing of stimuli (i.e., words and dots) would disrupt colour naming. Behavioural findings showed automatic associations as both training protocols elicited reliable congruency effects for all stimulus dimensions. Congruency effects following both training protocols produced reliable brain activations in various cortical sites involved in number and in cognitive control. The behavioural and brain patterns reported here support the role of learning in the brain correlates of developmental synaesthesia and provide the first evidence that automatic associations involving different magnitude dimensions can be acquired.

Do the colors of your letters depend on your language? Language-dependent and universal influences on grapheme-color synesthesia in seven languages

Grapheme-color synesthetes experience graphemes as having a consistent color (e.g., "N is turquoise"). Synesthetes' specific associations (which letter is which color) are often influenced by linguistic properties such as phonetic similarity, color terms ("Y is yellow"), and semantic associations ("D is for dog and dogs are brown"). However, most studies of synesthesia use only English-speaking synesthetes. Here, we measure the effect of color terms, semantic associations, and non-linguistic shape-color associations on synesthetic associations in Dutch, English, Greek, Japanese, Korean, Russian, and Spanish. The effect size of linguistic influences (color terms, semantic associations) differed significantly between languages. In contrast, the effect size of non-linguistic influences (shape-color associations), which we predicted to be universal, indeed did not differ between languages. We conclude that language matters (outcomes are influenced by the synesthete's language) and that synesthesia offers an exceptional opportunity to study influences on letter representations in different languages.

Synesthesia as (Multimodal) Mental Imagery

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one's mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory stimulation.

Synesthesia and the Perception of Unseen Realities

Exceptional experiences (ExE) incorporate a range of phenomena including subjective paranormal and transpersonal experiences. Synesthesia and synesthetic experiences are discussed as important variables in understanding the etiologies of ExE. The neural and psychological correlates of synesthetic experiences (associated with hyperconnectivity) are discussed in relation to ExE. It is argued that synesthetic processes enable both the detection and conscious perception of information from a range of sources that are usually unseen or inaccessible, including abstract, unlanguaged, preconscious, and potentially other nonlocal sources.

Substantiating synesthesia: a novel aid in a case of grapheme-colour synesthesia and concomitant dyscalculia

Here we present the case of SP, a 21-year-old female with life-long dyscalculia. SP was subsequently diagnosed with grapheme-color synesthesia, a diagnosis that serendipitously catalyzed our development of a novel aid:The digit-color calculator (DCC). The DCC substantiates SP's color concurrents, dramatically ameliorating her difficulties with basic calculations. We envisage the DCC and its analogues may assist others in educational settings, particularly if they experience difficulties with the acquisition of literacy and numeracy. Further devices that leverage synesthesia may also have the potential to improve the quality of life for others with trait synesthesia regardless of concomitant disorder.

Visual body form and orientation cues do not modulate visuo-tactile temporal integration

Body ownership relies on spatiotemporal correlations between multisensory signals and visual cues specifying oneself such as body form and orientation. The mechanism for the integration of bodily signals remains unclear. One approach to model multisensory integration that has been influential in the multisensory literature is Bayesian causal inference. This specifies that the brain integrates spatial and temporal signals coming from different modalities when it infers a common cause for inputs. As an example, the rubber hand illusion shows that visual form and orientation cues can promote the inference of a common cause (one's body) leading to spatial integration shown by a proprioceptive drift of the perceived location of the real hand towards the rubber hand. Recent studies investigating the effect of visual cues on temporal integration, however, have led to conflicting findings. These could be due to task differences, variation in ecological validity of stimuli and/or small samples. In this pre-registered study, we investigated the influence of visual information on temporal integration using a visuo-tactile temporal order judgement task with realistic stimuli and a sufficiently large sample determined by Bayesian analysis. Participants viewed videos of a touch being applied to plausible or implausible visual stimuli for one's hand (hand oriented plausibly, hand rotated 180 degrees, or a sponge) while also being touched at varying stimulus onset asynchronies. Participants judged which stimulus came first: viewed or felt touch. Results show that visual cues do not modulate visuo-tactile temporal order judgements. This is not in line with the idea that bodily signals indicating oneself influence the integration of multisensory signals in the temporal domain. The current study emphasises the importance of rigour in our methodologies and analyses to advance the understanding of how properties of multisensory events affect the encoding of temporal information in the brain.

Investigating genetic links between grapheme-colour synaesthesia and neuropsychiatric traits

Synaesthesia is a neurological phenomenon affecting perception, where triggering stimuli (e.g. letters and numbers) elicit unusual secondary sensory experiences (e.g. colours). Family-based studies point to a role for genetic factors in the development of this trait. However, the contributions of common genomic variation to synaesthesia have not yet been investigated. Here, we present the SynGenes cohort, the largest genotyped collection of unrelated people with grapheme-colour synaesthesia (n = 723). Synaesthesia has been associated with a range of other neuropsychological traits, including enhanced memory and mental imagery, as well as greater sensory sensitivity. Motivated by the prior literature on putative trait overlaps, we investigated polygenic scores derived from published genome-wide scans of schizophrenia and autism spectrum disorder (ASD), comparing our SynGenes cohort to 2181 non-synaesthetic controls. We found a very slight association between schizophrenia polygenic scores and synaesthesia (Nagelkerke's R2 = 0.0047, empirical p = 0.0027) and no significant association for scores related to ASD (Nagelkerke's R2 = 0.00092, empirical p = 0.54) or body mass index (R2 = 0.00058, empirical p = 0.60), included as a negative control. As sample sizes for studying common genomic variation continue to increase, genetic investigations of the kind reported here may yield novel insights into the shared biology between synaesthesia and other traits, to complement findings from neuropsychology and brain imaging. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Learning in colour: children with grapheme-colour synaesthesia show cognitive benefits in vocabulary and self-evaluated reading

Cognitive benefits associated with grapheme-colour synaesthesia in adults are well documented, but far less is known about whether such benefits might arise in synaesthetes as children. One previous study on a very small group of randomly sampled child synaesthetes found cognitive benefits in short-term memory and processing speed (the ability to quickly scan an array of images and discriminate between them), but was inconclusive for a test of receptive vocabulary. Using a stratified population sample (Growing Up in Scotland Project, Edinburgh, UK: Scottish Executive, 2007), we investigated the performance of a large cohort of child grapheme-colour synaesthetes using four literacy measures taken at age 10 years. These were three verbal comprehension measures (expressive vocabulary, receptive vocabulary and sentence comprehension) and one measure of academic self-concept in reading (plus one measure of academic self-concept in numeracy as a comparison). After controlling for demographic differences between groups, synaesthetes showed significantly enhanced performance for expressive and receptive vocabulary compared to their peers, but no benefits in sentence comprehension. Child synaesthetes also reported significantly higher academic self-concept for reading, but not for numeracy. Finally, we found that synaesthetes made significantly more progress than controls across the primary school years, although they began school with no a priori advantage. Our study provides powerful new evidence that children with grapheme-colour synaesthesia show vocabulary and literacy differences, which we contextualize within a theory of synaesthetic development. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Do synaesthesia and mental imagery tap into similar cross-modal processes?

Synaesthesia has previously been linked with imagery abilities, although an understanding of a causal role for mental imagery in broader synaesthetic experiences remains elusive. This can be partly attributed to our relatively poor understanding of imagery in sensory domains beyond vision. Investigations into the neural and behavioural underpinnings of mental imagery have nevertheless identified an important role for imagery in perception, particularly in mediating cross-modal interactions. However, the phenomenology of synaesthesia gives rise to the assumption that associated cross-modal interactions may be encapsulated and specific to synaesthesia. As such, evidence for a link between imagery and perception may not generalize to synaesthesia. Here, we present results that challenge this idea: first, we found enhanced somatosensory imagery evoked by visual stimuli of body parts in mirror-touch synaesthetes, relative to other synaesthetes or controls. Moreover, this enhanced imagery generalized to tactile object properties not directly linked to their synaesthetic associations. Second, we report evidence that concurrent experience evoked in grapheme-colour synaesthesia was sufficient to trigger visual-to-tactile correspondences that are common to all. Together, these findings show that enhanced mental imagery is a consistent hallmark of synaesthesia, and suggest the intriguing possibility that imagery may facilitate the cross-modal interactions that underpin synaesthesic experiences. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Echoes from the past: synaesthetic colour associations reflect childhood gender stereotypes

Grapheme-colour synaesthesia is a neurological phenomenon in which linguistic symbols evoke consistent colour sensations. Synaesthesia is believed to be influenced by both genetic and environmental factors, but how these factors interact to create specific associations in specific individuals is poorly understood. In this paper, we show that a grapheme-colour association in adult synaesthetes can be traced to a particular environmental effect at a particular moment in childhood. We propose a model in which specific grapheme-colour associations are 'locked in' during development in children predisposed to become synaesthetes, whereas grapheme-colour associations remain flexible in non-synaesthetes. We exploit Western gender-colour stereotypes to test our model: we found that young girls in general tend to associate their first initial with the colour pink. Consistent with our model, adult female synaesthetes are influenced by their childhood environment: they associate their first initial with pink. Adult female non-synaesthetes do not show this bias. Instead, in our study, non-synaesthetes tended to associate their first initial with their current favourite colour. The results thus support the 'locking in' model of synaesthesia, suggesting that synaesthetic associations can be used as a 'time capsule', revealing childhood influences on adult linguistic associations. Grapheme-colour synaesthesia may thus offer an extraordinary opportunity to study linguistic development. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Synaesthesia: a distinct entity that is an emergent feature of adaptive neurocognitive differences

In this article, I argue that synaesthesia is not on a continuum with neurotypical cognition. Synaesthesia is special: its phenomenology is different; it has distinct causal mechanisms; and is likely to be associated with a distinct neurocognitive profile. However, not all synaesthetes are the same, and there are quantifiable differences between them. In particular, the number of types of synaesthesia that a person possesses is a hitherto underappreciated variable that predicts cognitive differences along a number of dimensions (mental imagery, sensory sensitivity, attention to detail). Together with enhanced memory, this may constitute a common core of abilities that may go some way to explaining why synaesthesia might have evolved. I argue that the direct benefits of synaesthesia are generally limited (i.e. the synaesthetic associations do not convey novel information about the world) but, nevertheless, synaesthesia may develop due to other adaptive functions (e.g. perceptual ability, memory) that necessitate changes to design features of the brain. The article concludes by suggesting that synaesthesia forces us to reconsider what we mean by a 'normal' mind/brain. There may be multiple 'normal' neurodevelopmental trajectories that can sculpt very different ways of experiencing the world, of which synaesthesia is but one. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Synaesthetic colour associations for Japanese Kanji characters: from the perspective of grapheme learning

One of the fundamental questions about grapheme-colour synaesthesia is how specific associations between the graphemes and colours are formed. We addressed this question by focusing on the determinants of synaesthetic colours for Japanese Kanji characters (logographic characters) using a psycholinguistic approach. Study 1 explored the influence meaning has on synaesthetic colours for Kanji characters representing abstract meanings by examining synaesthetic colours for antonym pairs (i.e. characters with meanings opposed to each other) in Japanese synaesthetes. Results showed that semantic relations influenced the grapheme-colour associations for characters representing abstract meanings in the early stages of learning abstract Kanji, while the influence was reduced in the grapheme-colour associations for those learned later. Study 2 examined the effect that learning new sounds or meanings of graphemes has on synaesthetic colours for those graphemes. Japanese synaesthetes were taught new sounds or new meanings for familiar Kanji characters. Results indicated that acquiring new information for graphemes slightly but significantly reduced the test-retest grapheme-colour association consistency, suggesting that synaesthetic colours can be modulated to reflect the synaesthete's latest knowledge about graphemes. Implications of these findings are discussed from the perspective of the relationship between synaesthesia and grapheme learning. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Distinct colours in the 'synaesthetic colour palette'

In grapheme-colour synaesthesia, particular linguistic elements evoke particular colour sensations. Interestingly, when asked, non-synaesthetes can also associate colours to letters, and previous studies show that specific letter-to-colour associations have similar biases to those of synaesthetes. However, it is an open question whether the colours reported by synaesthetes and non-synaesthetes differ overall: is there a 'synaesthetic colour palette'? In this study, we visualize the overall distribution in colour space of colour concurrents in grapheme-colour synaesthetes, and colour associations in non-synaesthetic controls. We confirm the existence of a synaesthetic colour palette: colour concurrents in synaesthetes are different from colour associations in non-synaesthetes. We quantify three factors that distinguish the colour palette of synaesthetes and non-synaesthetes: synaesthetes have an increased over-representation of 'pure' (unmixed) hues, an increased presence of 'warm' (yellow, orange, brown) colours, and an increased presence of achromatic (grey, white, black) colours. Furthermore, we demonstrate that differences in the synaesthetic colour palette can be used to train a machine learning algorithm to reliably classify single subjects as synaesthetes versus non-synaesthetes without using test-retest consistency data. As far as we know, this is the first time an individual could be 'diagnosed' as a synaesthete, based only on his or her colours evoked by letters. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Classic psychedelics: the special role of the visual system

Here, we briefly overview the various aspects of classic serotonergic hallucinogens reported by a number of studies. One of the key hypotheses of our paper is that the visual effects of psychedelics might play a key role in resetting fears. Namely, we especially focus on visual processes because they are among the most prominent features of hallucinogen-induced hallucinations. We hypothesize that our brain has an ancient visual-based (preverbal) intrinsic cognitive process that, during the transient inhibition of top-down convergent and abstract thinking (mediated by the prefrontal cortex) by psychedelics, can neutralize emotional fears of unconscious and conscious life experiences from the past. In these processes, the decreased functional integrity of the self-referencing processes of the default mode network, the modified multisensory integration (linked to bodily self-consciousness and self-awareness), and the modified amygdala activity may also play key roles. Moreover, the emotional reset (elimination of stress-related emotions) by psychedelics may induce psychological changes and overwrite the stress-related neuroepigenetic information of past unconscious and conscious emotional fears.

A possible key role of vision in the development of schizophrenia

Based on a brief overview of the various aspects of schizophrenia reported by numerous studies, here we hypothesize that schizophrenia may originate (and in part be performed) from visual areas. In other words, it seems that a normal visual system or at least an evanescent visual perception may be an essential prerequisite for the development of schizophrenia as well as of various types of hallucinations. Our study focuses on auditory and visual hallucinations, as they are the most prominent features of schizophrenic hallucinations (and also the most studied types of hallucinations). Here, we evaluate the possible key role of the visual system in the development of schizophrenia.

A colorful advantage in iconic memory

Synesthesia is a benign neurodevelopmental condition in which stimulation of one sensory modality evokes experiences in a second, unstimulated modality (Simner and Hubbard, 2013). In grapheme-color synesthesia (GCS), which is experienced by 1-2% of adults, synesthetes reliably and involuntarily experience specific colors when viewing blackand-white graphemes. Previous case-studies have identified synesthetes with spectacular memory (Luria, 1968; Smilek, Dixon, Cudahy, & Merikle, 2001) and group studies have found advantages for synesthetes compared to nonsynesthetes in long-term memory (Rothen, Meier, & Ward, 2012). Here, we tested whether similar advantages are also present in earlier stages of memory. We tested visual iconic memory-the sensory store for vision-which has a very large capacity, but decays approximately 1000 ms after stimulus offset (Chow, 1985; Sergent et al., 2013; Sperling, 1960). We tested 20 synesthetes and 20 nonsynesthetes in a direct replication of the Sperling (1960) Partial Report Paradigm using letters (Experiment 1) and non-alphanumeric symbols (Experiment 2) as stimuli. Overall, synesthetes had a greater iconic memory capacity than nonsynesthetes when presented with synesthesia-inducing letter stimuli. This advantage was reduced when they were presented with non-synesthesia inducing symbol stimuli. Critically this advantage was most prominent when memory was stressed by asking participants to remember large arrays. Our results demonstrate that synesthetic memory advantages extend to the earliest stages of memory, and suggest that advantages in later stages of memory may arise from these earlier advantages.

Rare variants in axonogenesis genes connect three families with sound-color synesthesia

Synesthesia is a rare nonpathological phenomenon where stimulation of one sense automatically provokes a secondary perception in another. Hypothesized to result from differences in cortical wiring during development, synesthetes show atypical structural and functional neural connectivity, but the underlying molecular mechanisms are unknown. The trait also appears to be more common among people with autism spectrum disorder and savant abilities. Previous linkage studies searching for shared loci of large effect size across multiple families have had limited success. To address the critical lack of candidate genes, we applied whole-exome sequencing to three families with sound-color (auditory-visual) synesthesia affecting multiple relatives across three or more generations. We identified rare genetic variants that fully cosegregate with synesthesia in each family, uncovering 37 genes of interest. Consistent with reports indicating genetic heterogeneity, no variants were shared across families. Gene ontology analyses highlighted six genes-COL4A1, ITGA2, MYO10, ROBO3, SLC9A6, and SLIT2-associated with axonogenesis and expressed during early childhood when synesthetic associations are formed. These results are consistent with neuroimaging-based hypotheses about the role of hyperconnectivity in the etiology of synesthesia and offer a potential entry point into the neurobiology that organizes our sensory experiences.

Synaesthesia lost and found: two cases of person- and music-colour synaesthesia

Synaesthesia is a developmental condition involving cross-communication between sensory modalities or substreams whereby an inducer (e.g. a sound) automatically evokes a concurrent percept in another modality (e.g. a colour). Whether this condition arises due to atypical structural connectivity (e.g., between normally unconnected cortical areas) or altered neurochemistry remains a central question. We report the exceptional cases of two synaesthetes - subjects AB and CD - both of whom experience coloured auras around individuals, as well as coloured perceptions in response to music. Both subjects have, in recent years, suffered a complete loss or reduction of their synaesthetic experiences, one (AB) through successive head traumas, including a lightning strike, followed by a number of medications, and the other (CD) while taking anxiolytic medications. Using semi-structured interviews and data from the Synaesthesia Battery and a colourpicker task, we characterize the phenomenological characteristics of their pre-loss synaesthesia, as well as the subsequent restoration of each subject's synaesthetic experiences (in the months post-trauma for AB, and after cessation of medication for CD). Even after years of suppression, the patterns of associations were highly consistent with those experienced pre-injury. The phenomenological experience of synaesthesia can, thus, like most conscious experiences, be modulated by pharmacologically diverse medications or head injury. However, the underlying neural substrates mediating specific synaesthetic pairings appear remarkably 'hard-wired' and can persist over very long periods even under conditions that alter or completely suppress the conscious synaesthetic experience itself.

Synesthesia strengthens sound-symbolic cross-modal correspondences

Synesthesia is a phenomenon in which an experience in one domain is accompanied by an involuntary secondary experience in another, unrelated domain; in classical synesthesia, these associations are arbitrary and idiosyncratic. Cross-modal correspondences refer to universal associations between seemingly unrelated sensory features, e.g., auditory pitch and visual size. Some argue that these phenomena form a continuum, with classical synesthesia being an exaggeration of universal cross-modal correspondences, whereas others contend that the two are quite different, since cross-modal correspondences are non-arbitrary, non-idiosyncratic, and do not involve secondary experiences. Here, we used the implicit association test to compare synesthetes' and non-synesthetes' sensitivity to cross-modal correspondences. We tested the associations between auditory pitch and visual elevation, auditory pitch and visual size, and sound-symbolic correspondences between auditory pseudowords and visual shapes. Synesthetes were more sensitive than non-synesthetes to cross-modal correspondences involving sound-symbolic, but not low-level sensory, associations. We conclude that synesthesia heightens universally experienced cross-modal correspondences, but only when these involve sound symbolism. This is only partly consistent with the idea of a continuum between synesthesia and cross-modal correspondences, but accords with the idea that synesthesia is a high-level, post-perceptual phenomenon, with spillover of the abilities of synesthetes into domains outside their synesthesias. To our knowledge, this is the first demonstration that synesthetes, relative to non-synesthetes, experience stronger cross-modal correspondences outside their synesthetic domains.

The Brain's Tendency to Bind Audiovisual Signals Is Stable but Not General

Previous studies have shown a surprising amount of between-subjects variability in the strength of interactions between sensory modalities. For the same set of stimuli, some subjects exhibit strong interactions, whereas others exhibit weak interactions. To date, little is known about what underlies this variability. Sensory integration in the brain could be governed by a global mechanism or by task-specific mechanisms that could be either stable or variable across time. We used a rigorous quantitative tool (Bayesian causal inference) to investigate whether integration (i.e., binding) tendencies generalize across tasks and are stable across time. We report for the first time that individuals' binding tendencies are stable across time but are task-specific. These results provide evidence against the hypothesis that sensory integration is governed by a single, global parameter in the brain.

The Merit of Synesthesia for Consciousness Research

Synesthesia is a phenomenon in which additional perceptual experiences are elicited by sensory stimuli or cognitive concepts. Synesthetes possess a unique type of phenomenal experiences not directly triggered by sensory stimulation. Therefore, for better understanding of consciousness it is relevant to identify the mental and physiological processes that subserve synesthetic experience. In the present work we suggest several reasons why synesthesia has merit for research on consciousness. We first review the research on the dynamic and rapidly growing field of the studies of synesthesia. We particularly draw attention to the role of semantics in synesthesia, which is important for establishing synesthetic associations in the brain. We then propose that the interplay between semantics and sensory input in synesthesia can be helpful for the study of the neural correlates of consciousness, especially when making use of ambiguous stimuli for inducing synesthesia. Finally, synesthesia-related alterations of brain networks and functional connectivity can be of merit for the study of consciousness.