Consistency and strength of grapheme-color associations are separable aspects of synesthetic experience

Using immersive virtual reality to recreate the synaesthetic experience

Synaesthesia is a condition where people experience unusual sensory or cognitive sensations in response to apparently unrelated stimuli. This paper presents two experiments which aimed to examine whether Virtual Reality (VR) technology can be used to recreate the synaesthetic experience. There is a lack of research in this area, with most studies focussing primarily on synaesthetic colors. Experiment 1 aimed to build on previous research by using not only a traditional color-picker but also VR to capture a more nuanced picture of synaesthetic perception. A multiple case study design was used to examine the experiences of six participants in detail. Data gathering took place via Zoom. During the initial data-gathering session, participants used a color-picker to provide grapheme-color associations. After this session, some of the participants' synaesthetic experiences were recreated using a VR-by-proxy approach. Results indicated that VR is capable of capturing elements of synaesthetic perception that other methods have been unable to, such as texture, small degrees of movement, and 3D structure. Experiment 2 expanded upon these findings by moving beyond the VR-by-proxy approach and asking three participants to recreate their own audiovisual synaesthetic associations in the VR environment. Inductive Thematic Analysis was used to analyze the results of this experiment. The potential of expanding this technique to other forms of perceptual diversity is discussed.

Perceptual Dissimilarity Analysis Distinguishes Grapheme-Color Synesthetes from Nonsynesthetes

Synesthetes can be distinguished from nonsynesthetes on a variety of experimental tasks because their concurrent synesthetic experiences can affect task performance if these experiences match or conflict with some aspect of the stimulus. Here, we tested grapheme-color synesthetes and nonsynesthetic control participants using a novel perceptual similarity task to assess whether synesthetes' concurrent color experiences influence perceived grapheme similarity. Participants iteratively arranged graphemes and, separately, their associated synesthetic colors in a display, such that similar items were placed close together and dissimilar items further apart. The resulting relative inter-item distances were used to calculate the pair-wise (dis)similarity between items in the set, and thence to create separate perceptual representational dissimilarity matrices (RDMs) for graphemes and colors, on an individual basis. On the assumption that synesthetes' similarity judgments for graphemes would be influenced by their concurrent color experiences, we predicted that grapheme and color RDMs would be more strongly correlated for synesthetes than nonsynesthetes. We found that the mean grapheme-color RDM correlation was indeed significantly higher in synesthetes than nonsynesthetes; in addition, synesthetes' grapheme-color RDM correlations were more likely to be individually statistically significant, even after correction for multiple tests, than those of nonsynesthetes. Importantly, synesthetes' grapheme-color RDM correlations were scaled with the consistency of their grapheme-color associations as measured by their Synesthesia Battery (SB) scores. By contrast, the relationship between SB scores and grapheme-color RDM correlations for nonsynesthetes was not significant. Thus, dissimilarity analysis quantitatively distinguished synesthetes from nonsynesthetes, in a way that meaningfully reflects a key aspect of synesthetic experience.