Transfer between vision and haptics: Memory for 2-D patterns and 3-D objects

A comparison of vibrotactile patterns in an early warning system for obstacle detection using a haptic vest

Technical devices can enhance safety by warning people of unrecognized obstacles, particularly in traffic, wilderness, and industrial settings. This study aims to identify the most effective vibrotactile stimuli for localization tasks by developing and evaluating various types of vibrotactile alerts presented through a tactile vest with visual patterns. The study design involved comparing the time and consistency of interpreting visual stimuli and subsequent tactile stimuli. The tactile stimuli included: a 'point' vibration on the left or right side of the back, a 'column' pattern of five vibrations on one side of the back, and a 'wave' pattern of vibrations running along the back from left to right or vice versa. The results indicated that reaction times to visual stimuli were significantly shorter than to vibration stimuli, suggesting that visual stimuli are suitable for alert systems with low cognitive load. The 'point' and 'column' patterns were recognized significantly faster and more clearly than the'wave' pattern. Consequently, the haptic vest was classified as a potentially effective low cognitive load device in localization performance. The findings could inform the design of early warning systems for obstacle detection in real traffic situations.

The importance of multisensory-motor learning on subsequent visual recognition

Speed of visual object recognition is facilitated after active manual exploration of objects relative to passive visual processing alone. Manual exploration allows viewers to select important information about object structure that may facilitate recognition. Viewpoints where the objects' axis of elongation is perpendicular or parallel to the line of sight are selected more during exploration, recognized faster than other viewpoints, and afford the most information about structure when object movement is controlled by the viewer. Prior work used virtual object exploration in active and passive viewing conditions, limiting multisensory structural object information. Adding multisensory information to encoding may change accuracy of overall recognition, viewpoint selection, and viewpoint recognition. We tested whether the known active advantage for object recognition would change when real objects were studied, affording visual and haptic information. Participants interacted with 3D novel objects during manual exploration or passive viewing of another's object interactions. Object recognition was tested using several viewpoints of rendered objects. We found that manually explored objects were recognized more accurately than objects studied through passive exploration and that recognition of viewpoints differed from previous work.

Developmental changes in the visual, haptic, and bimodal perception of geometric angles

Geometrical knowledge is typically taught to children through a combination of vision and repetitive drawing (i.e. haptics), yet our understanding of how different spatial senses contribute to geometric perception during childhood is poor. Studies of line orientation suggest a dominant role of vision affecting the calibration of haptics during development; however, the associated multisensory interactions underpinning angle perception are unknown. Here we examined visual, haptic, and bimodal perception of angles across three age groups of children: 6 to 8 years, 8 to 10 years, and 10 to 12 years, with age categories also representing their class (grade) in primary school. All participants first learned an angular shape, presented dynamically, in one of three sensory tracing conditions: visual only, haptic only, or bimodal exploration. At test, which was visual only, participants selected a target angle from four possible alternatives with distractor angle sizes varying relative to the target angle size. We found a clear improvement in accuracy of angle perception with development for all learning modalities. Angle perception in the youngest group was equally poor (but above chance) for all modalities; however, for the two older child groups, visual learning was better than haptics. Haptic perception did not improve to the level of vision with age (even in a comparison adult group), and we found no specific benefit for bimodal learning over visual learning in any age group, including adults. Our results support a developmental increment in both spatial accuracy and precision in all modalities, which was greater in vision than in haptics, and are consistent with previous accounts of cross-sensory calibration in the perception of geometric forms.

The Effect of Exercise Intensity on Affective and Repetition Priming in Middle-Aged Adults

Previous research has shown that physical exercise improves memory. In the present study, we investigated the possible effects of the intensity of physical exercise as a function of the affective valence of words on implicit memory. In the study, 79 young adult volunteers were randomly assigned to perform moderate- (50% VO2max) or high-intensity exercise (80% VO2max) on a stationary bike. Once the required exercise intensity was achieved, participants performed an affective and repetition priming task concurrently with the physical exercise. Both groups showed similar repetition priming. The moderate-intensity exercise group showed affective priming with positive words, while affective priming was not found in the high-intensity exercise group. Facilitation occurred in both groups when a negative target word was preceded by a positive prime word. Our results suggest that the positive effect of physical exercise on memory is modulated by the affective valence of the stimuli. It seems that moderate-intensity exercise is more beneficial for implicit memory than high-intensity exercise.

The Nature of Haptic Working Memory Capacity and Its Relation to Visual Working Memory

I conducted three experiments to investigate haptic working memory capacity using a haptic change detection task with 2D stimuli. I adopted a single-task paradigm comprising haptic single-feature (orientation or texture) and haptic multifeature (orientation and texture) conditions in Experiment 1 and a dual-task paradigm with a primary haptic orientation or texture change detection task and a concurrent secondary visual shape or colour change detection task in Experiments 2-3. I observed that in the single-task paradigm, haptic change detection capacity was higher for single features than it was for multiple features. In haptic working memory, unlike in visual working memory, features of two different dimensions within an object cannot be integrated. In the dual-task paradigm, interference was observed when the concurrent visual shape change detection task was combined with the haptic orientation change detection task although interference was not observed when the concurrent visual colour change detection task was combined with it. In addition, the concurrent visual shape or colour change detection task did not interfere with the capacity for haptic texture memory, which was higher than that for haptic orientation memory. These findings demonstrate that geometric properties perhaps retained a common storage system shared between haptic working memory and visual working memory; however, haptic texture might be retained in an independent stable storage system that is haptic-specific.

“The Theater of the Mind”: The Effect of Radio Exposure on TV Advertising

Contemporary society requires communication strategies that integrate different media channels in order to improve advertising performance. Currently, there are not many scientific research studies of the various mass media, comparing the results of audiovisual advertising to purely audio or visual messages aimed at detecting the best combination of media, especially from a neurophysiological perspective. This study aims to investigate the effects of previous exposure to an advertisement via radio on the consumers’ response to the same advertisement shown on television (TV) or as a banner on a website. A total of seventy participants in a between-subjects experiment watched several television commercials during the advertising break of a documentary or saw some banners during a web surfing task. Half were first exposed to the same advertisements via radio. The results have shown that participants who previously listened to the radio advertisements spent a longer time looking at the brand and had a higher engagement when watching the same advertisements on television. Moreover, they had a different kind of visual attention to the website banners. This pattern of results indicates the effect of mere exposure—that is, the exposure to a radio advertisement enhances the effectiveness of the same advertisement via television or web, offering useful insights for media planning campaigns. Even if mere exposure has been extensively studied, cross-media research is scarcely explored, whereas this study detected the effects of mere exposure in a cross-media communication strategy, showing that it can be measured through psychophysiological methods.

The Recognition of Solid Object Shape: The Importance of Inhomogeneity

A single experiment evaluated the haptic-visual cross-modal matching of solid object shape. One set of randomly shaped artificial objects was used (sinusoidally modulated spheres, SMS) as well as two sets of naturally shaped objects (bell peppers, Capsicum annuum and sweet potatoes, Ipomoea batatas). A total of 66 adults participated in the study. The participants' task was to haptically explore a single object on any particular trial and subsequently indicate which of 12 simultaneously visible objects possessed the same shape. The participants' performance for the natural objects was 60.9 and 78.7 percent correct for the bell peppers and sweet potatoes, respectively. The analogous performance for the SMS objects, while better than chance, was far worse (18.6 percent correct). All of these types of stimulus objects possess a rich geometrical structure (e.g., they all possess multiple elliptic, hyperbolic, and parabolic surface regions). Nevertheless, these three types of stimulus objects are perceived differently: Individual members of sweet potatoes and bell peppers are largely identifiable to human participants, while the individual SMS objects are not. Analyses of differential geometry indicate that these natural objects (e.g., bell peppers and sweet potatoes) possess heterogeneous spatial configurations of distinctly curved surface regions, and this heterogeneity is lacking in SMS objects. The current results therefore suggest that increases in surface structure heterogeneity facilitate human object recognition.

Integration of prior knowledge during haptic exploration depends on information type

When haptically exploring softness, humans use higher peak forces when indenting harder versus softer objects. Here, we investigated the influence of different channels and types of prior knowledge on initial peak forces. Participants explored two stimuli (hard vs. soft) and judged which was softer. In Experiment 1 participants received either semantic (the words "hard" and "soft"), visual (video of indentation), or prior information from recurring presentation (blocks of harder or softer pairs only). In a control condition no prior information was given (randomized presentation). In the recurring condition participants used higher initial forces when exploring harder stimuli. No effects were found in control and semantic conditions. With visual prior information, participants used less force for harder objects. We speculate that these findings reflect differences between implicit knowledge induced by recurring presentation and explicit knowledge induced by visual and semantic information. To test this hypothesis, we investigated whether explicit prior information interferes with implicit information in Experiment 2. Two groups of participants discriminated softness of harder or softer stimuli in two conditions (blocked and randomized). The interference group received additional explicit information during the blocked condition; the implicit-only group did not. Implicit prior information was only used for force adaptation when no additional explicit information was given, whereas explicit interfered with movement adaptation. The integration of prior knowledge only seems possible when implicit prior knowledge is induced-not with explicit knowledge.

Visuo-Haptic Exploration for Multimodal Memory

When faced with a novel object, we explore it to understand its shape. This way we combine information coming from different senses, as touch, proprioception and vision, together with the motor information embedded in our motor execution plan. The exploration process provides a structure and constrains this rich flow of inputs, supporting the formation of a unified percept and the memorization of the object features. However, how the exploration strategies are planned is still an open question. In particular, is the exploration strategy used to memorize an object different from the exploration strategy adopted in a recall task? To address this question we used iCube, a sensorized cube which measures its orientation in space and the location of the contacts on its faces. Participants were required to explore the cube faces where little pins were positioned in varying number. Participants had to explore the cube twice and individuate potential differences between the two presentations, which could be performed either haptically alone, or with also vision available. The haptic and visuo-haptic (VH) exploratory strategies changed significantly when finalized to memorize the structure of the object with respect to when the same object was explored to recall and compare it with its memorized instance. These findings indicate that exploratory strategies are adapted not only to the property of the object to be analyzed but also to the prospective use of the resulting representation, be it memorization or recall. The results are discussed in light of the possibility of a systematic modeling of natural VH exploration strategies.

Representations of microgeometric tactile information during object recognition

Object recognition through tactile perception involves two elements: the shape of the object (macrogeometric properties) and the material of the object (microgeometric properties). Here we sought to determine the characteristics of microgeometric tactile representations regarding object recognition through tactile perception. Participants were directed to recognize objects with different surface materials using either tactile information or visual information. With a quantitative analysis of the cognitive process regarding object recognition, Experiment 1 confirmed the same eight concepts (composed of rules defining distinct cognitive processes) commonly generated in both tactile and visual perceptions to accomplish the task, although an additional concept was generated during the visual task. Experiment 2 focused only on tactile perception. Three tactile objects with different surface materials (plastic, cloth and sandpaper) were used for the object recognition task. The participants answered a questionnaire regarding the process leading to their answers (which was designed based on the results obtained in Experiment 1) and to provide ratings on the vividness, familiarity and affective valence. We used these experimental data to investigate whether changes in material attributes (tactile information) change the characteristics of tactile representation. The observation showed that differences in tactile information resulted in differences in cognitive processes, vividness, familiarity and emotionality. These two experiments collectively indicated that microgeometric tactile information contributes to object recognition by recruiting various cognitive processes including episodic memory and emotion, similar to the case of object recognition by visual information.

Visuo-haptic integration in object identification using novel objects

Although some studies have shown that haptic and visual identification seem to rely on similar processes, few studies have directly compared the two. We investigated haptic and visual object identification by asking participants to learn to recognize (Experiments 1, and 3), or to match (Experiment 2) novel objects that varied only in shape. Participants explored objects haptically, visually, or bimodally, and were then asked to identify objects haptically and/or visually. We demonstrated that patterns of identification errors were similar across identification modality, independently of learning and testing condition, suggesting that the haptic and visual representations in memory were similar. We also demonstrated that identification performance depended on both learning and testing conditions: visual identification surpassed haptic identification only when participants explored the objects visually or bimodally. When participants explored the objects haptically, haptic and visual identification were equivalent. Interestingly, when participants were simultaneously presented with two objects (one was presented haptically, and one was presented visually), object similarity only influenced performance when participants were asked to indicate whether the two objects were the same, or when participants had learned about the objects visually-without any haptic input. The results suggest that haptic and visual object representations rely on similar processes, that they may be shared, and that visual processing may not always lead to the best performance.

Transfer of Spatial Context from Visual to Haptic Search

Under incidental learning conditions, spatial layouts can be acquired implicitly and facilitate visual search (contextual-cueing effect). We examined whether the contextual-cueing effect is specific to the visual modality or transfers to the haptic modality. The participants performed 320 (experiment 1) or 192 (experiment 2) visual search trials based on a typical contextual-cueing paradigm, followed by haptic search trials in which half of the trials had layouts used in the previous visual search trials. The visual contextual-cueing effect was obtained in the learning phase. More importantly, the effect was transferred from visual to haptic searches; there was greater facilitation of haptic search trials when the spatial layout was the same as in the previous visual search trials, compared with trials in which the spatial layout differed from those in the visual search. This suggests the commonality of spatial memory to allocate focused attention in both visual and haptic modalities.

A Prime Example of the Maluma/Takete Effect? Testing for Sound Symbolic Priming

Certain nonwords, like maluma and takete, are associated with roundness and sharpness, respectively. However, this has typically been demonstrated using explicit tasks. We investigated whether this association would be detectable using a more implicit measure-a sequential priming task. We began with a replication of the standard Maluma/Takete effect (Experiments 1a and 1b) before examining whether round and sharp nonword primes facilitated the categorization of congruent shapes (Experiment 2). We found modest evidence of a priming effect in response accuracy. We next examined whether nonword primes affected categorization of ambiguous shapes, using visual (Experiment 3) and auditory primes (Experiment 4). We found that ambiguous shapes were categorized as round (sharp) more often following the presentation of a round (sharp) nonword. This suggests that phonemes may activate related shape information which then affects the processing of shapes, and that this association emerges even when participants are not explicitly searching for it.

Beyond visual conventions: Rethinking the design of tactile diagrams

This article reviews the debate over the comprehension of two-dimensional microcapsule tactile pictures portraying three-dimensional information through the use of perspective and visual drawing conventions. It is proposed that three-dimensional haptic object recognition and its relationship to visual object recognition can inform the design of two-dimensional tactual displays. Experimental work is discussed which validates the use of such information in informing the design of 2-D tactile pictures.

Tactual, Visual, and Cross-Modal Transfer of Texture in 5- and 8-Year-Old Children

Children's tactual, visual, and cross-modal transfer abilities for texture were investigated in a delayed matching-to-sample paradigm. Transfer performance from vision to touch was found to increase between 5 and 8 years of age, whereas transfer performance from touch to vision did not vary with age and matched touch-to-touch performance. Asymmetrical cross-modal abilities were observed at the age of 8 years, vision-to-touch transfer performance being higher than touch-to-vision transfer performance (experiment 2). This developmental pattern could not be attributed to limitations in the tactual or visual discriminability of the textures or to differences in tactual or visual memory between the two age groups (experiment 1). It is suggested that the increase with age in vision-to-touch performance may be related to the intervention of more efficient top – down perceptual processes in the older children.

Aging and the Visual, Haptic, and Cross-Modal Perception of Natural Object Shape

One hundred observers participated in two experiments designed to investigate aging and the perception of natural object shape. In the experiments, younger and older observers performed either a same/different shape discrimination task (experiment 1) or a cross-modal matching task (experiment 2). Quantitative effects of age were found in both experiments. The effect of age in experiment 1 was limited to cross-modal shape discrimination: there was no effect of age upon unimodal (ie within a single perceptual modality) shape discrimination. The effect of age in experiment 2 was eliminated when the older observers were either given an unlimited amount of time to perform the task or when the number of response alternatives was decreased. Overall, the results of the experiments reveal that older observers can effectively perceive 3-D shape from both vision and haptics.

Crossmodal enhancement in the LOC for visuohaptic object recognition over development

Research has provided strong evidence of multisensory convergence of visual and haptic information within the visual cortex. These studies implement crossmodal matching paradigms to examine how systems use information from different sensory modalities for object recognition. Developmentally, behavioral evidence of visuohaptic crossmodal processing has suggested that communication within sensory systems develops earlier than across systems; nonetheless, it is unknown how the neural mechanisms driving these behavioral effects develop. To address this gap in knowledge, BOLD functional Magnetic Resonance Imaging (fMRI) was measured during delayed match-to-sample tasks that examined intramodal (visual-to-visual, haptic-to-haptic) and crossmodal (visual-to-haptic, haptic-to-visual) novel object recognition in children aged 7-8.5 years and adults. Tasks were further divided into sample encoding and test matching phases to dissociate the relative contributions of each. Results of crossmodal and intramodal object recognition revealed the network of known visuohaptic multisensory substrates, including the lateral occipital complex (LOC) and the intraparietal sulcus (IPS). Critically, both adults and children showed crossmodal enhancement within the LOC, suggesting a sensitivity to changes in sensory modality during recognition. These groups showed similar regions of activation, although children generally exhibited more widespread activity during sample encoding and weaker BOLD signal change during test matching than adults. Results further provided evidence of a bilateral region in the occipitotemporal cortex that was haptic-preferring in both age groups. This region abutted the bimodal LOtv, and was consistent with a medial to lateral organization that transitioned from a visual to haptic bias within the LOC. These findings converge with existing evidence of visuohaptic processing in the LOC in adults, and extend our knowledge of crossmodal processing in adults and children.

Visuo-haptic multisensory object recognition, categorization, and representation

Visual and haptic unisensory object processing show many similarities in terms of categorization, recognition, and representation. In this review, we discuss how these similarities contribute to multisensory object processing. In particular, we show that similar unisensory visual and haptic representations lead to a shared multisensory representation underlying both cross-modal object recognition and view-independence. This shared representation suggests a common neural substrate and we review several candidate brain regions, previously thought to be specialized for aspects of visual processing, that are now known also to be involved in analogous haptic tasks. Finally, we lay out the evidence for a model of multisensory object recognition in which top-down and bottom-up pathways to the object-selective lateral occipital complex are modulated by object familiarity and individual differences in object and spatial imagery.

Conceptual priming with pictures and environmental sounds

A series of experiments was conducted to examine conceptual priming within and across modalities with pictures and environmental sounds. In Experiment 1, we developed a new multimodal stimulus set consisting of two picture and sound exemplars that represented 80 object items. In Experiments 2, we investigated whether categorization of the stimulus items would be facilitated by picture and environmental sound primes that were derived from different exemplars of the target items; and in Experiments 3 and 4, we tested the additional influence on priming when trials were consolidated within a target modality and the inter stimulus interval was lengthened. The results demonstrated that target categorization was facilitated by the advanced presentation of conceptually related exemplars, but there were differences in effectiveness when pictures and sounds appeared as primes.

Cross-modal transitivity in a California sea lion (Zalophus californianus)

The ability of an experimentally experienced female California sea lion to form transitive relations across sensory modalities was tested using a matching-to-sample procedure. The subject was trained by trial-and-error, using differential reinforcement, to relate an acoustic sample stimulus to one member from each of two previously established visual classes. Once the two auditory-visual relations were formed, she was tested to determine whether untrained transitive relations would emerge between each of the acoustic stimuli and the remaining stimuli of each 10-member visual class. During testing, the sea lion demonstrated immediate transfer by responding correctly on 89% of the 18 novel transfer trials compared to 88% on familiar baseline trials. We then repeated this training and transfer procedure twice more with new auditory-visual pairings with similar positive results. Finally, the six explicitly trained auditory-visual relations and the 56 derived auditory-visual relations were intermixed in a single session, and the subject's performance remained stable at high levels. This sea lion's transfer performance indicates that a nonhuman animal is capable of forming new associations through cross-modal transitivity.

The perceptual nature of audiovisual interactions for semantic knowledge in young and elderly adults

Audiovisual interactions for familiar objects are at the core of perception. The nature of these interactions depends on the amodal--sensory abstracted--or modal--sensory-dependent--approach of knowledge. According to these approaches, the interactions should be respectively semantic and indirect or perceptual and direct. This issue is therefore a central question to memory and perception, yet the nature of these interactions remains unexplored in young and elderly adults. We used a cross-modal priming paradigm combined with a visual masking procedure of half of the auditory primes. The data demonstrated similar results in the young and elderly adult groups. The mask interfered with the priming effect in the semantically congruent condition, whereas the mask facilitated the processing of the visual target in the semantically incongruent condition. These findings indicate that audiovisual interactions are perceptual, and support the grounded cognition theory.

Tactual perception: a review of experimental variables and procedures

This paper reviews the literature on tactual perception. Throughout this review, we will highlight some of the most relevant aspects in the touch literature: type of stimuli; type of participants; type of tactile exploration; and finally, the interaction between touch and other senses. Regarding type of stimuli, we will analyse studies with abstract stimuli such as vibrations, with two- and three-dimensional stimuli, and also concrete stimuli, considering the relation between familiar and unfamiliar stimuli and the haptic perception of faces. Under the "type of participants" topic, we separated studies with blind participants, studies with children and adults, and also performed an overview of sex differences in performance. The type of tactile exploration is explored considering conditions of active and passive touch, the relevance of movement in touch and the relation between haptic exploration and time. Finally, interactions between touch and vision, touch and smell and touch and taste are explored in the last topic. The review ends with an overall conclusion on the state of the art for the tactual perception literature. With this work, we intend to present an organised overview of the main variables in touch experiments, compiling aspects reported in the tactual literature, and attempting to provide both a summary of previous findings, and a guide to the design of future works on tactual perception and memory, through a presentation of implications from previous studies.

Haptic Object Recognition: How Important are Depth Cues and Plane Orientation?

Raised-line drawings of familiar objects are very difficult to identify with active touch only. In contrast, haptically explored real 3-D objects are usually recognised efficiently, albeit slower and less accurately than with vision. Real 3-D objects have more depth information than outline drawings, but also extra information about identity (eg texture, hardness, temperature). Previous studies have not manipulated the availability of depth information in haptic object recognition whilst controlling for other information sources, so the importance of depth cues has not been assessed. In the present experiments, people named plastic small-scale models of familiar objects. Five versions of bilaterally symmetrical objects were produced. Versions varied only in the amount of depth information: minimal for cookie-cutter and filled-in outlines, partial for squashed and half objects, and full for 3-D models. Recognition was faster and much more accurate when more depth information was available, whether exploration was with both hands or just one finger. Novices found it almost impossible to recognise objects explored with two hand-held probes whereas experts succeeded using probes regardless of the amount of depth information. Surprisingly, plane misorientation did not impair recognition. Unlike with vision, depth information, but not object orientation, is extremely important for haptic object recognition.

Unimodal and crossmodal working memory representations of visual and kinesthetic movement trajectories

The present study investigated whether visual and kinesthetic stimuli are stored as multisensory or modality-specific representations in unimodal and crossmodal working memory tasks. To this end, angle-shaped movement trajectories were presented to 16 subjects in delayed matching-to-sample tasks either visually or kinesthetically during encoding and recognition. During the retention interval, a secondary visual or kinesthetic interference task was inserted either immediately or with a delay after encoding. The modality of the interference task interacted significantly with the encoding modality. After visual encoding, memory was more impaired by a visual than by a kinesthetic secondary task, while after kinesthetic encoding the pattern was reversed. The time when the secondary task had to be performed interacted with the encoding modality as well. For visual encoding, memory was more impaired, when the secondary task had to be performed at the beginning of the retention interval. In contrast, memory after kinesthetic encoding was more affected, when the secondary task was introduced later in the retention interval. The findings suggest that working memory traces are maintained in a modality-specific format characterized by distinct consolidation processes that take longer after kinesthetic than after visual encoding.

Perception of Direction for Applied Tangential Skin Displacement: Effects of Speed, Displacement, and Repetition

A variety of tasks could benefit from the availability of direction cues that do not rely on vision or sound. The application of tangential skin displacement at the fingertip has been found to be a reliable means of communicating direction and has potential to be rendered by a compact device. Our lab has conducted experiments exploring the use of this type of tactile stimulus to communicate direction. Each subject pressed his/her right index fingertip against a 7 mm rounded rubber cylinder that moved at constant speed, applying shear force to deform the skin of the fingerpad. A range of displacements (0.05-1 mm) and speeds (0.5-4 mm/s) were tested. Subjects were asked to respond with the direction of the skin stretch, choosing from four directions, each separated by 90 degrees. Direction detection accuracy was found to depend upon both the speed and total displacement of the stimulus, with higher speeds and larger displacements resulting in greater accuracy. Accuracy rates greater than 95 percent were observed with as little as 0.2 mm of tangential displacement and at speeds as slow as 1 mm/s. Results were analyzed for direction dependence and temporal trends. Subjects responded most accurately to stimuli in the proximal and distal directions, and least accurately to stimuli in the ulnar direction. Subject performance decreased slightly with prolonged testing but there was no statistically significant learning trend. A second experiment was conducted to evaluate priming effects and the benefit of repeated stimuli. It was found that repeated stimuli do not improve direction communication, but subject responses were found to have a priming effect on future performance. This preliminary information will inform the design and use of a tactile display suitable for use in hand-held electronics.

Cross-Modal Transfer in Visual and Haptic Face Recognition

We report four psychophysical experiments investigating cross-modal transfer in visual and haptic face recognition. We found surprisingly good haptic performance and cross-modal transfer for both modalities. Interestingly, transfer was asymmetric depending on which modality was learned first. These findings are discussed in relation to haptic object processing and face processing.

Multisensory visual-tactile object related network in humans: insights gained using a novel crossmodal adaptation approach

Neuroimaging techniques have provided ample evidence for multisensory integration in humans. However, it is not clear whether this integration occurs at the neuronal level or whether it reflects areal convergence without such integration. To examine this issue as regards visuo-tactile object integration we used the repetition suppression effect, also known as the fMRI-based adaptation paradigm (fMR-A). Under some assumptions, fMR-A can tag specific neuronal populations within an area and investigate their characteristics. This technique has been used extensively in unisensory studies. Here we applied it for the first time to study multisensory integration and identified a network of occipital (LOtv and calcarine sulcus), parietal (aIPS), and prefrontal (precentral sulcus and the insula) areas all showing a clear crossmodal repetition suppression effect. These results provide a crucial first insight into the neuronal basis of visuo-haptic integration of objects in humans and highlight the power of using fMR-A to study multisensory integration using non-invasinve neuroimaging techniques.

A putative model of multisensory object representation

This review surveys the recent literature on visuo-haptic convergence in the perception of object form, with particular reference to the lateral occipital complex (LOC) and the intraparietal sulcus (IPS) and discusses how visual imagery or multisensory representations might underlie this convergence. Drawing on a recent distinction between object- and spatially-based visual imagery, we propose a putative model in which LOtv, a subregion of LOC, contains a modality-independent representation of geometric shape that can be accessed either bottom-up from direct sensory inputs or top-down from frontoparietal regions. We suggest that such access is modulated by object familiarity: spatial imagery may be more important for unfamiliar objects and involve IPS foci in facilitating somatosensory inputs to the LOC; by contrast, object imagery may be more critical for familiar objects, being reflected in prefrontal drive to the LOC.

Learning to perceive differences in solid shape through vision and touch

A single experiment was designed to investigate perceptual learning and the discrimination of 3-D object shape. Ninety-six observers were presented with naturally shaped solid objects either visually, haptically, or across the modalities of vision and touch. The observers' task was to judge whether the two sequentially presented objects on any given trial possessed the same or different 3-D shapes. The results of the experiment revealed that significant perceptual learning occurred in all modality conditions, both unimodal and cross-modal. The amount of the observers' perceptual learning, as indexed by increases in hit rate and d', was similar for all of the modality conditions. The observers' hit rates were highest for the unimodal conditions and lowest in the cross-modal conditions. Lengthening the inter-stimulus interval from 3 to 15 s led to increases in hit rates and decreases in response bias. The results also revealed the existence of an asymmetry between two otherwise equivalent cross-modal conditions: in particular, the observers' perceptual sensitivity was higher for the vision-haptic condition and lower for the haptic-vision condition. In general, the results indicate that effective cross-modal shape comparisons can be made between the modalities of vision and active touch, but that complete information transfer does not occur.

Selective attention modulates visual and haptic repetition priming: effects in aging and Alzheimer's disease

In two experiments, we examined the effect of selective attention at encoding on repetition priming in normal aging and Alzheimer's disease (AD) patients for objects presented visually (experiment 1) or haptically (experiment 2). We used a repetition priming paradigm combined with a selective attention procedure at encoding. Reliable priming was found for both young adults and healthy older participants for visually presented pictures (experiment 1) as well as for haptically presented objects (experiment 2). However, this was only found for attended and not for unattended stimuli. The results suggest that independently of the perceptual modality, repetition priming requires attention at encoding and that perceptual facilitation is maintained in normal aging. However, AD patients did not show priming for attended stimuli, or for unattended visual or haptic objects. These findings suggest an early deficit of selective attention in AD. Results are discussed from a cognitive neuroscience approach.

Vision and touch: multiple or multisensory representations of objects?

The relationship between visually and haptically derived representations of objects is an important question in multisensory processing and, increasingly, in mental representation. We review evidence for the format and properties of these representations, and address possible theoretical models. We explore the relevance of visual imagery processes and highlight areas for further research, including the neglected question of asymmetric performance in the visuo-haptic cross-modal memory paradigm. We conclude that the weight of evidence suggests the existence of a multisensory representation, spatial in format, and flexibly accessible by both bottom-up and top-down inputs, although efficient comparison between modality-specific representations cannot entirely be ruled out.

Multisensory Identification of Natural Objects in a Two-Way Crossmodal Priming Paradigm

Abstract. The question of how vision and audition interact in natural object identification is currently a matter of debate. We developed a large set of auditory and visual stimuli representing natural objects in order to facilitate research in the field of multisensory processing. Normative data was obtained for 270 brief environmental sounds and 320 visual object stimuli. Each stimulus was named, categorized, and rated with regard to familiarity and emotional valence by N = 56 participants (Study 1). This multimodal stimulus set was employed in two subsequent crossmodal priming experiments that used semantically congruent and incongruent stimulus pairs in a S1-S2 paradigm. Task-relevant targets were either auditory (Study 2) or visual stimuli (Study 3). The behavioral data of both experiments expressed a crossmodal priming effect with shorter reaction times for congruent as compared to incongruent stimulus pairs. The observed facilitation effect suggests that object identification in one modality is influenced by input from another modality. This result implicates that congruent visual and auditory stimulus pairs were perceived as the same object and demonstrates a first validation of the multimodal stimulus set.

Memory for curvature of objects: haptic touch vs. vision

The present study examined the role of vision and haptics in memory for stimulus objects that vary along the dimension of curvature. Experiment 1 measured haptic-haptic (T-T) and haptic-visual (T-V) discrimination of curvature in a short-term memory paradigm, using 30-second retention intervals containing five different interpolated tasks. Results showed poorest performance when the interpolated tasks required spatial processing or movement, thereby suggesting that haptic information about shape is encoded in a spatial-motor representation. Experiment 2 compared visual-visual (V-V) and visual-haptic (V-T) short-term memory, again using 30-second delay intervals. The results of the ANOVA failed to show a significant effect of intervening activity. Intra-modal visual performance and cross-modal performance were similar. Comparing the four modality conditions (inter-modal V-T, T-V; intra-modal V-V, T-T, by combining the data of Experiments 1 and 2), in a global analysis, showed a reliable interaction between intervening activity and experiment (modality). Although there appears to be a general tendency for spatial and movement activities to exert the most deleterious effects overall, the patterns are not identical when the initial stimulus is encoded haptically (Experiment 1) and visually (Experiment 2).

Central visual persistences: II. Effects of hand and head rotations

In an earlier paper, kinesthetic effects on central visual persistences (CPs) were reported, including the ability to move these images by hand following eye closure. While all CPs could be translated anywhere within the frontal field, the present report documents a more selective influence of manual rotations on CPs in the same subjects. When common objects or figures drawn on cards were rotated (while holding one end of the object or one corner of a card between thumb and forefinger), it was found that CPs of larger objects rotated with the hand. By contrast, CPs of smaller objects, parts of objects, and textures remained stable in space as the hand rotated. It is proposed that CPs of smaller stimuli and textures are represented mainly by the ventral stream (temporal cortex) while larger CPs, which rotate, are represented mainly by the dorsal stream (parietal cortex). A second discovery was that CPs of small objects (but not of line segments or textures) could be rotated when the thumb and fingers surrounded the edges of the object. It is proposed that neuronal convergence of visual and tactile information about shape increases parietal responses to small objects, so that their CPs will rotate. Experiments with CPs offer new tools to infer visual coding differences between ventral and dorsal streams in man.

Mental representation in visual/haptic crossmodal memory: evidence from interference effects

Two experiments used visual-, verbal-, and haptic-interference tasks during encoding (Experiment 1) and retrieval (Experiment 2) to examine mental representation of familiar and unfamiliar objects in visual/haptic crossmodal memory. Three competing theories are discussed, which variously suggest that these representations are: (a) visual; (b) dual-code-visual for unfamiliar objects but visual and verbal for familiar objects; or (c) amodal. The results suggest that representations of unfamiliar objects are primarily visual but that crossmodal memory for familiar objects may rely on a network of different representations. The pattern of verbal-interference effects suggests that verbal strategies facilitate encoding of unfamiliar objects regardless of modality, but only haptic recognition regardless of familiarity. The results raise further research questions about all three theoretical approaches.