A putative model of multisensory object representation

Dissociable identity- and modality-specific neural representations as revealed by cross-modal nonspatial inhibition of return

There are ongoing debates on whether object concepts are coded as supramodal identity-based or modality-specific representations in the human brain. In this fMRI study, we adopted a cross-modal "prime-neutral cue-target" semantic priming paradigm, in which the prime-target relationship was manipulated along both the identity and the modality dimensions. The prime and the target could refer to either the same or different semantic identities, and could be delivered via either the same or different sensory modalities. By calculating the main effects and interactions of this 2 (identity cue validity: "Identity_Cued" vs. "Identity_Uncued") × 2 (modality cue validity: "Modality_Cued" vs. "Modality_Uncued") factorial design, we aimed at dissociating three neural networks involved in creating novel identity-specific representations independent of sensory modality, in creating modality-specific representations independent of semantic identity, and in evaluating changes of an object along both the identity and the modality dimensions, respectively. Our results suggested that bilateral lateral occipital cortex was involved in creating a new supramodal semantic representation irrespective of the input modality, left dorsal premotor cortex, and left intraparietal sulcus were involved in creating a new modality-specific representation irrespective of its semantic identity, and bilateral superior temporal sulcus was involved in creating a representation when the identity and modality properties were both cued or both uncued. In addition, right inferior frontal gyrus showed enhanced neural activity only when both the identity and the modality of the target were new, indicating its functional role in novelty detection.

Haptic shape processing in visual cortex

Humans typically rely upon vision to identify object shape, but we can also recognize shape via touch (haptics). Our haptic shape recognition ability raises an intriguing question: To what extent do visual cortical shape recognition mechanisms support haptic object recognition? We addressed this question using a haptic fMRI repetition design, which allowed us to identify neuronal populations sensitive to the shape of objects that were touched but not seen. In addition to the expected shape-selective fMRI responses in dorsal frontoparietal areas, we observed widespread shape-selective responses in the ventral visual cortical pathway, including primary visual cortex. Our results indicate that shape processing via touch engages many of the same neural mechanisms as visual object recognition. The shape-specific repetition effects we observed in primary visual cortex show that visual sensory areas are engaged during the haptic exploration of object shape, even in the absence of concurrent shape-related visual input. Our results complement related findings in visually deprived individuals and highlight the fundamental role of the visual system in the processing of object shape.

Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation

Sensory substitution devices (SSDs) have come a long way since first developed for visual rehabilitation. They have produced exciting experimental results, and have furthered our understanding of the human brain. Unfortunately, they are still not used for practical visual rehabilitation, and are currently considered as reserved primarily for experiments in controlled settings. Over the past decade, our understanding of the neural mechanisms behind visual restoration has changed as a result of converging evidence, much of which was gathered with SSDs. This evidence suggests that the brain is more than a pure sensory-machine but rather is a highly flexible task-machine, i.e., brain regions can maintain or regain their function in vision even with input from other senses. This complements a recent set of more promising behavioral achievements using SSDs and new promising technologies and tools. All these changes strongly suggest that the time has come to revive the focus on practical visual rehabilitation with SSDs and we chart several key steps in this direction such as training protocols and self-train tools.

Disintegration of multisensory signals from the real hand reduces default limb self-attribution: an fMRI study

The perception of our limbs in space is built upon the integration of visual, tactile, and proprioceptive signals. Accumulating evidence suggests that these signals are combined in areas of premotor, parietal, and cerebellar cortices. However, it remains to be determined whether neuronal populations in these areas integrate hand signals according to basic temporal and spatial congruence principles of multisensory integration. Here, we developed a setup based on advanced 3D video technology that allowed us to manipulate the spatiotemporal relationships of visuotactile (VT) stimuli delivered on a healthy human participant's real hand during fMRI and investigate the ensuing neural and perceptual correlates. Our experiments revealed two novel findings. First, we found responses in premotor, parietal, and cerebellar regions that were dependent upon the spatial and temporal congruence of VT stimuli. This multisensory integration effect required a simultaneous match between the seen and felt postures of the hand, which suggests that congruent visuoproprioceptive signals from the upper limb are essential for successful VT integration. Second, we observed that multisensory conflicts significantly disrupted the default feeling of ownership of the seen real limb, as indexed by complementary subjective, psychophysiological, and BOLD measures. The degree to which self-attribution was impaired could be predicted from the attenuation of neural responses in key multisensory areas. These results elucidate the neural bases of the integration of multisensory hand signals according to basic spatiotemporal principles and demonstrate that the disintegration of these signals leads to "disownership" of the seen real hand.

Crossmodal interactions of haptic and visual texture information in early sensory cortex

Both visual and haptic information add to the perception of surface texture. While prior studies have reported crossmodal interactions of both sensory modalities at the behavioral level, neuroimaging studies primarily investigated texture perception in separate visual and haptic paradigms. These experimental designs, however, only allowed to identify overlap in both sensory processing streams but no interaction of visual and haptic texture processing. By varying texture characteristics in a bimodal task, the current study investigated how these crossmodal interactions are reflected at the cortical level. We used fMRI to compare cortical activation in response to matching versus non-matching visual-haptic texture information. We expected that passive simultaneous presentation of matching visual-haptic input would be sufficient to induce BOLD responses graded with varying texture characteristics. Since no cognitive evaluation of the stimuli was required, we expected to find changes primarily at a rather early processing stage. Our results confirmed our assumptions by showing crossmodal interactions of visual-haptic texture information in early somatosensory and visual cortex. However, the nature of the crossmodal effects was slightly different in both sensory cortices. In early visual cortex, matching visual-haptic information increased the average activation level and induced parametric BOLD signal variations with varying texture characteristics. In early somatosensory cortex only the latter was true. These results challenge the notion that visual and haptic texture information is processed independently and indicate a crossmodal interaction of sensory information already at an early cortical processing stage.

tDCS Modulation of Visually Induced Analgesia

Multisensory interactions can produce analgesic effects. In particular, viewing one's own body reduces pain levels, perhaps because of changes in connectivity between visual areas specialized for body representation, and sensory areas underlying pain perception. We tested the causal role of the extrastriate visual cortex in triggering visually induced analgesia by modulating the excitability of this region with transcranial direct current stimulation (tDCS). Anodal, cathodal, or sham tDCS (2 mA, 10 min) was administered to 24 healthy participants over the right occipital or over the centro-parietal areas thought to be involved in the sensory processing of pain. Participants were required to rate the intensity of painful electrical stimuli while viewing either their left hand or an object occluding the left hand, both before and immediately after tDCS. We found that the analgesic effect of viewing the body was enhanced selectively by anodal stimulation of the occipital cortex. The effect was specific for the polarity and the site of stimulation. The present results indicate that visually induced analgesia may depend on neural signals from the extrastriate visual cortex.

Transfer of object category knowledge across visual and haptic modalities: experimental and computational studies

We study people's abilities to transfer object category knowledge across visual and haptic domains. If a person learns to categorize objects based on inputs from one sensory modality, can the person categorize these same objects when the objects are perceived through another modality? Can the person categorize novel objects from the same categories when these objects are, again, perceived through another modality? Our work makes three contributions. First, by fabricating Fribbles (3-D, multi-part objects with a categorical structure), we developed visual-haptic stimuli that are highly complex and realistic, and thus more ecologically valid than objects that are typically used in haptic or visual-haptic experiments. Based on these stimuli, we developed the See and Grasp data set, a data set containing both visual and haptic features of the Fribbles, and are making this data set freely available on the world wide web. Second, complementary to previous research such as studies asking if people transfer knowledge of object identity across visual and haptic domains, we conducted an experiment evaluating whether people transfer object category knowledge across these domains. Our data clearly indicate that we do. Third, we developed a computational model that learns multisensory representations of prototypical 3-D shape. Similar to previous work, the model uses shape primitives to represent parts, and spatial relations among primitives to represent multi-part objects. However, it is distinct in its use of a Bayesian inference algorithm allowing it to acquire multisensory representations, and sensory-specific forward models allowing it to predict visual or haptic features from multisensory representations. The model provides an excellent qualitative account of our experimental data, thereby illustrating the potential importance of multisensory representations and sensory-specific forward models to multisensory perception.

Haptic categorical perception of shape

Categorization and categorical perception have been extensively studied, mainly in vision and audition. In the haptic domain, our ability to categorize objects has also been demonstrated in earlier studies. Here we show for the first time that categorical perception also occurs in haptic shape perception. We generated a continuum of complex shapes by morphing between two volumetric objects. Using similarity ratings and multidimensional scaling we ensured that participants could haptically discriminate all objects equally. Next, we performed classification and discrimination tasks. After a short training with the two shape categories, both tasks revealed categorical perception effects. Training leads to between-category expansion resulting in higher discriminability of physical differences between pairs of stimuli straddling the category boundary. Thus, even brief training can alter haptic representations of shape. This suggests that the weights attached to various haptic shape features can be changed dynamically in response to top-down information about class membership.

Drawing enhances cross-modal memory plasticity in the human brain: a case study in a totally blind adult

In a memory-guided drawing task under blindfolded conditions, we have recently used functional Magnetic Resonance Imaging (fMRI) to demonstrate that the primary visual cortex (V1) may operate as the visuo-spatial buffer, or “sketchpad,” for working memory. The results implied, however, a modality-independent or amodal form of its operation. In the present study, to validate the role of V1 in non-visual memory, we eliminated not only the visual input but all levels of visual processing by replicating the paradigm in a congenitally blind individual. Our novel Cognitive-Kinesthetic method was used to train this totally blind subject to draw complex images guided solely by tactile memory. Control tasks of tactile exploration and memorization of the image to be drawn, and memory-free scribbling were also included. FMRI was run before training and after training. Remarkably, V1 of this congenitally blind individual, which before training exhibited noisy, immature, and non-specific responses, after training produced full-fledged response time-courses specific to the tactile-memory drawing task. The results reveal the operation of a rapid training-based plasticity mechanism that recruits the resources of V1 in the process of learning to draw. The learning paradigm allowed us to investigate for the first time the evolution of plastic re-assignment in V1 in a congenitally blind subject. These findings are consistent with a non-visual memory involvement of V1, and specifically imply that the observed cortical reorganization can be empowered by the process of learning to draw.

Cortical free-association dynamics: distinct phases of a latching network

A Potts associative memory network has been proposed as a simplified model of macroscopic cortical dynamics, in which each Potts unit stands for a patch of cortex, which can be activated in one of S local attractor states. The internal neuronal dynamics of the patch is not described by the model, rather it is subsumed into an effective description in terms of graded Potts units, with adaptation effects both specific to each attractor state and generic to the patch. If each unit, or patch, receives effective (tensor) connections from C other units, the network has been shown to be able to store a large number p of global patterns, or network attractors, each with a fraction a of the units active, where the critical load p_{c} scales roughly like p_{c}≈CS^{2}/aln(1/a) (if the patterns are randomly correlated). Interestingly, after retrieving an externally cued attractor, the network can continue jumping, or latching, from attractor to attractor, driven by adaptation effects. The occurrence and duration of latching dynamics is found through simulations to depend critically on the strength of local attractor states, expressed in the Potts model by a parameter w. Here we describe with simulations and then analytically the boundaries between distinct phases of no latching, of transient and sustained latching, deriving a phase diagram in the plane w-T, where T parametrizes thermal noise effects. Implications for real cortical dynamics are briefly reviewed in the conclusions.

Time course of information processing in visual and haptic object classification

Vision identifies objects rapidly and efficiently. In contrast, object recognition by touch is much slower. Furthermore, haptics usually serially accumulates information from different parts of objects, whereas vision typically processes object information in parallel. Is haptic object identification slower simply due to sequential information acquisition and the resulting memory load or due to more fundamental processing differences between the senses? To compare the time course of visual and haptic object recognition, we slowed visual processing using a novel, restricted viewing technique. In an electroencephalographic (EEG) experiment, participants discriminated familiar, nameable from unfamiliar, unnamable objects both visually and haptically. Analyses focused on the evoked and total fronto-central theta-band (5-7 Hz; a marker of working memory) and the occipital upper alpha-band (10-12 Hz; a marker of perceptual processing) locked to the onset of classification. Decreases in total upper alpha-band activity for haptic identification of objects indicate a likely processing role of multisensory extrastriate areas. Long-latency modulations of alpha-band activity differentiated between familiar and unfamiliar objects in haptics but not in vision. In contrast, theta-band activity showed a general increase over time for the slowed-down visual recognition task only. We conclude that haptic object recognition relies on common representations with vision but also that there are fundamental differences between the senses that do not merely arise from differences in their speed of processing.

Touching motion: rTMS on the human middle temporal complex interferes with tactile speed perception

Brain functional and psychophysical studies have clearly demonstrated that visual motion perception relies on the activity of the middle temporal complex (hMT+). However, recent studies have shown that hMT+ seems to be also activated during tactile motion perception, suggesting that this visual extrastriate area is involved in the processing and integration of motion, irrespective of the sensorial modality. In the present study, we used repetitive transcranial magnetic stimulation (rTMS) to assess whether hMT+ plays a causal role in tactile motion processing. Blindfolded participants detected changes in the speed of a grid of tactile moving points with their finger (i.e. tactile modality). The experiment included three different conditions: a control condition with no TMS and two TMS conditions, i.e. hMT+-rTMS and posterior parietal cortex (PPC)-rTMS. Accuracies were significantly impaired during hMT+-rTMS but not in the other two conditions (No-rTMS or PPC-rTMS), moreover, thresholds for detecting speed changes were significantly higher in the hMT+-rTMS with respect to the control TMS conditions. These findings provide stronger evidence that the activity of the hMT+ area is involved in tactile speed processing, which may be consistent with the hypothesis of a supramodal role for that cortical region in motion processing.

Categorizing natural objects: a comparison of the visual and the haptic modalities

Although the hands are the most important tool for humans to manipulate objects, only little is known about haptic processing of natural objects. Here, we selected a unique set of natural objects, namely seashells, which vary along a variety of object features, while others are shared across all stimuli. To correctly interact with objects, they have to be identified or categorized. For both processes, measuring similarities between objects is crucial. Our goal is to better understand the haptic similarity percept by comparing it to the visual similarity percept. First, direct similarity measures were analyzed using multidimensional scaling techniques to visualize the perceptual spaces of both modalities. We find that the visual and the haptic modality form almost identical perceptual spaces. Next, we performed three different categorization tasks. All tasks exhibit a highly accurate processing of complex shapes of the haptic modality. Moreover, we find that objects grouped into the same category form regions within the perceptual space. Hence, in both modalities, perceived similarity constitutes the basis for categorizing objects. Moreover, both modalities focus on shape to form categories. Taken together, our results lead to the assumption that the same cognitive processes link haptic and visual similarity perception and the resulting categorization behavior.

Object and spatial imagery dimensions in visuo-haptic representations

Visual imagery comprises object and spatial dimensions. Both types of imagery encode shape but a key difference is that object imagers are more likely to encode surface properties than spatial imagers. Since visual and haptic object representations share many characteristics, we investigated whether haptic and multisensory representations also share an object-spatial continuum. Experiment 1 involved two tasks in both visual and haptic within-modal conditions, one requiring discrimination of shape across changes in texture, the other discrimination of texture across changes in shape. In both modalities, spatial imagers could ignore changes in texture but not shape, whereas object imagers could ignore changes in shape but not texture. Experiment 2 re-analyzed a cross-modal version of the shape discrimination task from an earlier study. We found that spatial imagers could discriminate shape across changes in texture but object imagers could not and that the more one preferred object imagery, the more texture changes impaired discrimination. These findings are the first evidence that object and spatial dimensions of imagery can be observed in haptic and multisensory representations.

Similarity and categorization: from vision to touch

Even though human perceptual development relies on combining multiple modalities, most categorization studies so far have focused on the visual modality. To better understand the mechanisms underlying multisensory categorization, we analyzed visual and haptic perceptual spaces and compared them with human categorization behavior. As stimuli we used a three-dimensional object space of complex, parametrically-defined objects. First, we gathered similarity ratings for all objects and analyzed the perceptual spaces of both modalities using multidimensional scaling analysis. Next, we performed three different categorization tasks which are representative of every-day learning scenarios: in a fully unconstrained task, objects were freely categorized, in a semi-constrained task, exactly three groups had to be created, whereas in a constrained task, participants received three prototype objects and had to assign all other objects accordingly. We found that the haptic modality was on par with the visual modality both in recovering the topology of the physical space and in solving the categorization tasks. We also found that within-category similarity was consistently higher than across-category similarity for all categorization tasks and thus show how perceptual spaces based on similarity can explain visual and haptic object categorization. Our results suggest that both modalities employ similar processes in forming categories of complex objects.

Recruitment of occipital cortex during sensory substitution training linked to subjective experience of seeing in people with blindness

Over three months of intensive training with a tactile stimulation device, 18 blind and 10 blindfolded seeing subjects improved in their ability to identify geometric figures by touch. Seven blind subjects spontaneously reported ‘visual qualia’, the subjective sensation of seeing flashes of light congruent with tactile stimuli. In the latter subjects tactile stimulation evoked activation of occipital cortex on electroencephalography (EEG). None of the blind subjects who failed to experience visual qualia, despite identical tactile stimulation training, showed EEG recruitment of occipital cortex. None of the blindfolded seeing humans reported visual-like sensations during tactile stimulation. These findings support the notion that the conscious experience of seeing is linked to the activation of occipital brain regions in people with blindness. Moreover, the findings indicate that provision of visual information can be achieved through non-visual sensory modalities which may help to minimize the disability of blind individuals, affording them some degree of object recognition and navigation aid.

Cross-modal Processing in the Occipito-temporal Cortex: A TMS Study of the Müller-Lyer Illusion

The Müller-Lyer illusion occurs both in vision and in touch, and transfers cross-modally from vision to haptics [Mancini, F., Bricolo, E., & Vallar, G. Multisensory integration in the Müller-Lyer illusion: From vision to haptics. Quarterly Journal of Experimental Psychology, 63, 818–830, 2010]. Recent evidence suggests that the neural underpinnings of the Müller-Lyer illusion in the visual modality involve the bilateral lateral occipital complex (LOC) and right superior parietal cortex (SPC). Conversely, the neural correlates of the haptic and cross-modal illusions have never been investigated previously. Here we used repetitive TMS (rTMS) to address the causal role of the regions activated by the visual illusion in the generation of the visual, haptic, and cross-modal visuo-haptic illusory effects, investigating putative modality-specific versus cross-modal underlying processes. rTMS was administered to the right and the left hemisphere, over occipito-temporal cortex or SPC. rTMS over left and right occipito-temporal cortex impaired both unisensory (visual, haptic) and cross-modal processing of the illusion in a similar fashion. Conversely, rTMS interference over left and right SPC did not affect the illusion in any modality. These results demonstrate the causal involvement of bilateral occipito-temporal cortex in the representation of the visual, haptic, and cross-modal Müller-Lyer illusion, in favor of the hypothesis of shared underlying processes. This indicates that occipito-temporal cortex plays a cross-modal role in perception both of illusory and nonillusory shapes.

Smoking experience modulates the cortical integration of vision and haptics

Human neuroplasticity of multisensory integration has been studied mainly in the context of natural or artificial training situations in healthy subjects. However, regular smokers also offer the opportunity to assess the impact of intensive daily multisensory interactions with smoking-related objects on the neural correlates of crossmodal object processing. The present functional magnetic resonance imaging study revealed that smokers show a comparable visuo-haptic integration pattern for both smoking paraphernalia and control objects in the left lateral occipital complex, a region playing a crucial role in crossmodal object recognition. Moreover, the degree of nicotine dependence correlated positively with the magnitude of visuo-haptic integration in the left lateral occipital complex (LOC) for smoking-associated but not for control objects. In contrast, in the left LOC non-smokers displayed a visuo-haptic integration pattern for control objects, but not for smoking paraphernalia. This suggests that prolonged smoking-related multisensory experiences in smokers facilitate the merging of visual and haptic inputs in the lateral occipital complex for the respective stimuli. Studying clinical populations who engage in compulsive activities may represent an ecologically valid approach to investigating the neuroplasticity of multisensory integration.

Dual pathways for haptic and visual perception of spatial and texture information

Segregation of information flow along a dorsally directed pathway for processing object location and a ventrally directed pathway for processing object identity is well established in the visual and auditory systems, but is less clear in the somatosensory system. We hypothesized that segregation of location vs. identity information in touch would be evident if texture is the relevant property for stimulus identity, given the salience of texture for touch. Here, we used functional magnetic resonance imaging (fMRI) to investigate whether the pathways for haptic and visual processing of location and texture are segregated, and the extent of bisensory convergence. Haptic texture-selectivity was found in the parietal operculum and posterior visual cortex bilaterally, and in parts of left inferior frontal cortex. There was bisensory texture-selectivity at some of these sites in posterior visual and left inferior frontal cortex. Connectivity analyses demonstrated, in each modality, flow of information from unisensory non-selective areas to modality-specific texture-selective areas and further to bisensory texture-selective areas. Location-selectivity was mostly bisensory, occurring in dorsal areas, including the frontal eye fields and multiple regions around the intraparietal sulcus bilaterally. Many of these regions received input from unisensory areas in both modalities. Together with earlier studies, the activation and connectivity analyses of the present study establish that somatosensory processing flows into segregated pathways for location and object identity information. The location-selective somatosensory pathway converges with its visual counterpart in dorsal frontoparietal cortex, while the texture-selective somatosensory pathway runs through the parietal operculum before converging with its visual counterpart in visual and frontal cortex. Both segregation of sensory processing according to object property and multisensory convergence appear to be universal organizing principles.

Shape from sound: evidence for a shape operator in the lateral occipital cortex

A recent view of cortical functional specialization suggests that the primary organizing principle of the cortex is based on task requirements, rather than sensory modality. Consistent with this view, recent evidence suggests that a region of the lateral occipitotemporal cortex (LO) may process object shape information regardless of the modality of sensory input. There is considerable evidence that area LO is involved in processing visual and haptic shape information. However, sound can also carry acoustic cues to an object's shape, for example, when a sound is produced by an object's impact with a surface. Thus, the current study used auditory stimuli that were created from recordings of objects impacting a hard surface to test the hypothesis that area LO is also involved in auditory shape processing. The objects were of two shapes, rods and balls, and of two materials, metal and wood. Subjects were required to categorize the impact sounds in one of three tasks, (1) by the shape of the object while ignoring material, (2) by the material of the object while ignoring shape, or (3) by using all the information available. Area LO was more strongly recruited when subjects discriminated impact sounds based on the shape of the object that made them, compared to when subjects discriminated those same sounds based on material. The current findings suggest that activation in area LO is shape selective regardless of sensory input modality, and are consistent with an emerging theory of perceptual functional specialization of the brain that is task-based rather than sensory modality-based.

Visual stability

Our vision remains stable even though the movements of our eyes, head and bodies create a motion pattern on the retina. One of the most important, yet basic, feats of the visual system is to correctly determine whether this retinal motion is owing to real movement in the world or rather our own self-movement. This problem has occupied many great thinkers, such as Descartes and Helmholtz, at least since the time of Alhazen. This theme issue brings together leading researchers from animal neurophysiology, clinical neurology, psychophysics and cognitive neuroscience to summarize the state of the art in the study of visual stability. Recently, there has been significant progress in understanding the limits of visual stability in humans and in identifying many of the brain circuits involved in maintaining a stable percept of the world. Clinical studies and new experimental methods, such as transcranial magnetic stimulation, now make it possible to test the causal role of different brain regions in creating visual stability and also allow us to measure the consequences when the mechanisms of visual stability break down.

Tactile shape discrimination recruits human lateral occipital complex during early perceptual processing

Neuroimaging studies investigating somatosensory-based object recognition in humans have revealed activity in the lateral occipital complex, a cluster of regions primarily associated with visual object recognition. To date, determining whether this activity occurs during or subsequent to recognition per se, has been difficult to assess due to the low temporal resolution of the hemodynamic response. To more finely measure the timing of somatosensory object recognition processes we employed high density EEG using a modified version of a paradigm previously applied to neuroimaging experiments. Simple geometric shapes were presented to the right index finger of 10 participants while the ongoing EEG was measured time locked to the stimulus. In the condition of primary interest participants discriminated the shape of the stimulus. In the alternate condition they judged stimulus duration. Using traditional event-related potential analysis techniques we found significantly greater amplitudes in the evoked potentials of the shape discrimination condition between 140 and 160 ms, a timeframe in which LOC mediated perceptual processes are believed to occur during visual object recognition. Scalp voltage topography and source analysis procedures indicated the lateral occipital complex as the likely source behind this effect. This finding supports a multisensory role for the lateral occipital complex during object recognition.

Prior visual experience, and perception and memory of shape in people with total blindness

The aim of this study was to explore the role of prior visual experience for tactile differentiation of object shapes. The study investigated whether people who lost their vision later in life were able to identify and recognize object shapes more accurately and faster than those who were blind from their birth. Four experiments were conducted. The first two were concerned with tactile shape differentiation, the second two with shape recognition. The hypotheses were only partially confirmed. The ‘late’ blind participants distinguished shapes more accurately than the congenitally blind (particularly in ‘simple’ perception tasks). This finding may suggest that people who have prior visual experience use an allocentric strategy when visualizing object shapes in their imagery. The ‘late’ blind participants performed the tasks more slowly than those who were congenitally blind. This may be explained by the complexity of the task, the time needed to create an allocentric representation, and discrepancy in the tactile experiences between the congenitally and late blind groups. A number of implications for further research are outlined.

Unimodal and crossmodal working memory representations of visual and kinesthetic movement trajectories

The present study investigated whether visual and kinesthetic stimuli are stored as multisensory or modality-specific representations in unimodal and crossmodal working memory tasks. To this end, angle-shaped movement trajectories were presented to 16 subjects in delayed matching-to-sample tasks either visually or kinesthetically during encoding and recognition. During the retention interval, a secondary visual or kinesthetic interference task was inserted either immediately or with a delay after encoding. The modality of the interference task interacted significantly with the encoding modality. After visual encoding, memory was more impaired by a visual than by a kinesthetic secondary task, while after kinesthetic encoding the pattern was reversed. The time when the secondary task had to be performed interacted with the encoding modality as well. For visual encoding, memory was more impaired, when the secondary task had to be performed at the beginning of the retention interval. In contrast, memory after kinesthetic encoding was more affected, when the secondary task was introduced later in the retention interval. The findings suggest that working memory traces are maintained in a modality-specific format characterized by distinct consolidation processes that take longer after kinesthetic than after visual encoding.

Separate mechanisms for audio-tactile pitch and loudness interactions

A major goal in perceptual neuroscience is to understand how signals from different sensory modalities are combined to produce stable and coherent representations. We previously investigated interactions between audition and touch, motivated by the fact that both modalities are sensitive to environmental oscillations. In our earlier study, we characterized the effect of auditory distractors on tactile frequency and intensity perception. Here, we describe the converse experiments examining the effect of tactile distractors on auditory processing. Because the two studies employ the same psychophysical paradigm, we combined their results for a comprehensive view of how auditory and tactile signals interact and how these interactions depend on the perceptual task. Together, our results show that temporal frequency representations are perceptually linked regardless of the attended modality. In contrast, audio-tactile loudness interactions depend on the attended modality: Tactile distractors influence judgments of auditory intensity, but judgments of tactile intensity are impervious to auditory distraction. Lastly, we show that audio-tactile loudness interactions depend critically on stimulus timing, while pitch interactions do not. These results reveal that auditory and tactile inputs are combined differently depending on the perceptual task. That distinct rules govern the integration of auditory and tactile signals in pitch and loudness perception implies that the two are mediated by separate neural mechanisms. These findings underscore the complexity and specificity of multisensory interactions.

Effects of vision and haptics on categorizing common objects

Most research on object recognition and categorization centers on vision. However, these phenomena are likely influenced by the commonly used modality of touch. The present study tested this notion by having participants explore three-dimensional objects using vision and haptics in naming and sorting tasks. Results showed greater difficulty naming (recognizing) and sorting (categorizing) objects haptically. For both conditions, error increased from the concrete attribute of size to the more abstract quality of predation, providing behavioral evidence for shared object representation in vision and haptics.

Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience

The occipital and parietal lobes contain regions that are recruited for both visual and haptic object processing. The purpose of the present study was to characterize the underlying neural mechanisms for bimodal integration of vision and haptics in these visuo-haptic object-selective brain regions to find out whether these brain regions are sites of neuronal or areal convergence. Our sensory conditions consisted of visual-only (V), haptic-only (H), and visuo-haptic (VH), which allowed us to evaluate integration using the superadditivity metric. We also presented each stimulus condition at two different levels of signal-to-noise ratio or salience. The salience manipulation allowed us to assess integration using the rule of inverse effectiveness. We were able to localize previously described visuo-haptic object-selective regions in the lateral occipital cortex (lateral occipital tactile-visual area) and the intraparietal sulcus, and also localized a new region in the left anterior fusiform gyrus. There was no evidence of superadditivity with the VH stimulus at either level of salience in any of the regions. There was, however, a strong effect of salience on multisensory enhancement: the response to the VH stimulus was more enhanced at higher salience across all regions. In other words, the regions showed enhanced integration of the VH stimulus with increasing effectiveness of the unisensory stimuli. We called the effect "enhanced effectiveness." The presence of enhanced effectiveness in visuo-haptic object-selective brain regions demonstrates neuronal convergence of visual and haptic sensory inputs for the purpose of processing object shape.

Object familiarity modulates effective connectivity during haptic shape perception

In the preceding paper (Lacey, S., Flueckiger, P., Stilla, R., Lava, M., Sathian, K., 2009a. Object familiarity modulates involvement of visual imagery in haptic shape perception), we showed that the activations evoked by visual imagery overlapped more extensively, and their magnitudes were more correlated, with those evoked during haptic shape perception of familiar, compared to unfamiliar, objects. Here we used task-specific analyses of functional and effective connectivity to provide convergent evidence. These analyses showed that the visual imagery and familiar haptic shape tasks activated similar networks, whereas the unfamiliar haptic shape task activated a different network. Multivariate Granger causality analyses of effective connectivity, in both a conventional form and one purged of zero-lag correlations, showed that the visual imagery and familiar haptic shape networks involved top-down paths from prefrontal cortex into the lateral occipital complex (LOC), whereas the unfamiliar haptic shape network was characterized by bottom-up, somatosensory inputs into the LOC. We conclude that shape representations in the LOC are flexibly accessible, either top-down or bottom-up, according to task demands, and that visual imagery is more involved in LOC activation during haptic shape perception when objects are familiar, compared to unfamiliar.

Object familiarity modulates the relationship between visual object imagery and haptic shape perception

Although visual cortical engagement in haptic shape perception is well established, its relationship with visual imagery remains controversial. We addressed this using functional magnetic resonance imaging during separate visual object imagery and haptic shape perception tasks. Two experiments were conducted. In the first experiment, the haptic shape task employed unfamiliar, meaningless objects, whereas familiar objects were used in the second experiment. The activations evoked by visual object imagery overlapped more extensively, and their magnitudes were more correlated, with those evoked during haptic shape perception of familiar, compared to unfamiliar, objects. In the companion paper (Deshpande et al., this issue), we used task-specific functional and effective connectivity analyses to provide convergent evidence: these analyses showed that the neural networks underlying visual imagery were similar to those underlying haptic shape perception of familiar, but not unfamiliar, objects. We conclude that visual object imagery is more closely linked to haptic shape perception when objects are familiar, compared to when they are unfamiliar.