Investigating the MESA (multipoint electrotactile speech aid): the transmission of segmental features of speech

Improved tactile speech perception using audio-to-tactile sensory substitution with formant frequency focusing

Haptic hearing aids, which provide speech information through tactile stimulation, could substantially improve outcomes for both cochlear implant users and for those unable to access cochlear implants. Recent advances in wide-band haptic actuator technology have made new audio-to-tactile conversion strategies viable for wearable devices. One such strategy filters the audio into eight frequency bands, which are evenly distributed across the speech frequency range. The amplitude envelopes from the eight bands modulate the amplitudes of eight low-frequency tones, which are delivered through vibration to a single site on the wrist. This tactile vocoder strategy effectively transfers some phonemic information, but vowels and obstruent consonants are poorly portrayed. In 20 participants with normal touch perception, we tested (1) whether focusing the audio filters of the tactile vocoder more densely around the first and second formant frequencies improved tactile vowel discrimination, and (2) whether focusing filters at mid-to-high frequencies improved obstruent consonant discrimination. The obstruent-focused approach was found to be ineffective. However, the formant-focused approach improved vowel discrimination by 8%, without changing overall consonant discrimination. The formant-focused tactile vocoder strategy, which can readily be implemented in real time on a compact device, could substantially improve speech perception for haptic hearing aid users.

Electrotactile Communication via Matrix Electrode Placed on the Torso Using Fast Calibration, and Static vs. Dynamic Encoding

Electrotactile stimulation is a technology that reproducibly elicits tactile sensations and can be used as an alternative channel to communicate information to the user. The presented work is a part of an effort to develop this technology into an unobtrusive communication tool for first responders. In this study, the aim was to compare the success rate (SR) between discriminating stimulation at six spatial locations (static encoding) and recognizing six spatio-temporal patterns where pads are activated sequentially in a predetermined order (dynamic encoding). Additionally, a procedure for a fast amplitude calibration, that includes a semi-automated initialization and an optional manual adjustment, was employed and evaluated. Twenty subjects, including twelve first responders, participated in the study. The electrode comprising the 3 × 2 matrix of pads was placed on the lateral torso. The results showed that high SRs could be achieved for both types of message encoding after a short learning phase; however, the dynamic approach led to a statistically significant improvement in messages recognition (SR of 93.3%), compared to static stimulation (SR of 83.3%). The proposed calibration procedure was also effective since in 83.8% of the cases the subjects did not need to adjust the stimulation amplitude manually.

Congruent aero-tactile stimuli bias perception of voicing continua

Multimodal integration is the formation of a coherent percept from different sensory inputs such as vision, audition, and somatosensation. Most research on multimodal integration in speech perception has focused on audio-visual integration. In recent years, audio-tactile integration has also been investigated, and it has been established that puffs of air applied to the skin and timed with listening tasks shift the perception of voicing by naive listeners. The current study has replicated and extended these findings by testing the effect of air puffs on gradations of voice onset time along a continuum rather than the voiced and voiceless endpoints of the original work. Three continua were tested: bilabial (“pa/ba”), velar (“ka/ga”), and a vowel continuum (“head/hid”) used as a control. The presence of air puffs was found to significantly increase the likelihood of choosing voiceless responses for the two VOT continua but had no effect on choices for the vowel continuum. Analysis of response times revealed that the presence of air puffs lengthened responses for intermediate (ambiguous) stimuli and shortened them for endpoint (non-ambiguous) stimuli. The slowest response times were observed for the intermediate steps for all three continua, but for the bilabial continuum this effect interacted with the presence of air puffs: responses were slower in the presence of air puffs, and faster in their absence. This suggests that during integration auditory and aero-tactile inputs are weighted differently by the perceptual system, with the latter exerting greater influence in those cases where the auditory cues for voicing are ambiguous.

Electro-Haptic Stimulation: A New Approach for Improving Cochlear-Implant Listening

Cochlear implants (CIs) have been remarkably successful at restoring speech perception for severely to profoundly deaf individuals. Despite their success, several limitations remain, particularly in CI users' ability to understand speech in noisy environments, locate sound sources, and enjoy music. A new multimodal approach has been proposed that uses haptic stimulation to provide sound information that is poorly transmitted by the implant. This augmenting of the electrical CI signal with haptic stimulation (electro-haptic stimulation; EHS) has been shown to improve speech-in-noise performance and sound localization in CI users. There is also evidence that it could enhance music perception. We review the evidence of EHS enhancement of CI listening and discuss key areas where further research is required. These include understanding the neural basis of EHS enhancement, understanding the effectiveness of EHS across different clinical populations, and the optimization of signal-processing strategies. We also discuss the significant potential for a new generation of haptic neuroprosthetic devices to aid those who cannot access hearing-assistive technology, either because of biomedical or healthcare-access issues. While significant further research and development is required, we conclude that EHS represents a promising new approach that could, in the near future, offer a non-invasive, inexpensive means of substantially improving clinical outcomes for hearing-impaired individuals.

Using haptic stimulation to enhance auditory perception in hearing-impaired listeners

INTRODUCTION

Hearing-assistive devices, such as hearing aids and cochlear implants, transform the lives of hearing-impaired people. However, users often struggle to locate and segregate sounds. This leads to impaired threat detection and an inability to understand speech in noisy environments. Recent evidence suggests that segregation and localization can be improved by providing missing sound-information through haptic stimulation.

AREAS COVERED

This article reviews the evidence that haptic stimulation can effectively provide sound information. It then discusses the research and development required for this approach to be implemented in a clinically viable device. This includes discussion of what sound information should be provided and how that information can be extracted and delivered.

EXPERT OPINION

Although this research area has only recently emerged, it builds on a significant body of work showing that sound information can be effectively transferred through haptic stimulation. Current evidence suggests that haptic stimulation is highly effective at providing missing sound-information to cochlear implant users. However, a great deal of work remains to implement this approach in an effective wearable device. If successful, such a device could offer an inexpensive, noninvasive means of improving educational, work, and social experiences for hearing-impaired individuals, including those without access to hearing-assistive devices.

Electro-Haptic Enhancement of Spatial Hearing in Cochlear Implant Users

Cochlear implants (CIs) have enabled hundreds of thousands of profoundly hearing-impaired people to perceive sounds by electrically stimulating the auditory nerve. However, CI users are often very poor at locating sounds, which leads to impaired sound segregation and threat detection. We provided missing spatial hearing cues through haptic stimulation to augment the electrical CI signal. We found that this "electro-haptic" stimulation dramatically improved sound localisation. Furthermore, participants were able to effectively integrate spatial information transmitted through these two senses, performing better with combined audio and haptic stimulation than with either alone. Our haptic signal was presented to the wrists and could readily be delivered by a low-cost wearable device. This approach could provide a non-invasive means of improving outcomes for the vast majority of CI users who have only one implant, without the expense and risk of a second implantation.

Cross-modal effects in speech perception

Speech research during recent years has moved progressively away from its traditional focus on audition toward a more multisensory approach. In addition to audition and vision, many somatosenses including proprioception, pressure, vibration and aerotactile sensation are all highly relevant modalities for experiencing and/or conveying speech. In this article, we review both long-standing cross-modal effects stemming from decades of audiovisual speech research as well as new findings related to somatosensory effects. Cross-modal effects in speech perception to date are found to be constrained by temporal congruence and signal relevance, but appear to be unconstrained by spatial congruence. Far from taking place in a one-, two- or even three-dimensional space, the literature reveals that speech occupies a highly multidimensional sensory space. We argue that future research in cross-modal effects should expand to consider each of these modalities both separately and in combination with other modalities in speech.

Mechanics of finger-tip electronics

Tactile sensors and electrotactile stimulators can provide important links between humans and virtual environments, through the sensation of touch. Soft materials, such as low modulus silicones, are attractive as platforms and support matrices for arrays sensors and actuators that laminate directly onto the fingertips. Analytic models for the mechanics of three dimensional, form-fitting finger cuffs based on such designs are presented here, along with quantitative validation using the finite element method. The results indicate that the maximum strains in the silicone and the embedded devices are inversely proportional to the square root of radius of curvature of the cuff. These and other findings can be useful in formulating designs for these and related classes of body-worn, three dimensional devices.

Aerotactile integration from distal skin stimuli

Tactile sensations at extreme distal body locations can integrate with auditory information to alter speech perception among uninformed and untrained listeners. Inaudible air puffs were applied to participants' ankles, simultaneously with audible syllables having aspirated and unaspirated stop onsets. Syllables heard simultaneously with air puffs were more likely to be heard as aspirated. These results demonstrate that event-appropriate information from distal parts of the body integrates in speech perception, even without frequent or robust location-specific experience. In addition, overall performance was significantly better for those with hair on their ankles, which suggests that the presence of hair may help establish signal relevance, and so aid in multi-modal speech perception.

Perceptual grouping over time within and across auditory and tactile modalities

In auditory scene analysis, population separation and temporal coherence have been proposed to explain how auditory features are grouped together and streamed over time. The present study investigated whether these two theories can be applied to tactile streaming and whether temporal coherence theory can be applied to crossmodal streaming. The results show that synchrony detection between two tones/taps at different frequencies/locations became difficult when one of the tones/taps was embedded in a perceptual stream. While the taps applied to the same location were streamed over time, the taps applied to different locations were not. This observation suggests that tactile stream formation can be explained by population-separation theory. On the other hand, temporally coherent auditory stimuli at different frequencies were streamed over time, but temporally coherent tactile stimuli applied to different locations were not. When there was within-modality streaming, temporally coherent auditory stimuli and tactile stimuli were not streamed over time, either. This observation suggests the limitation of temporal coherence theory when it is applied to perceptual grouping over time.

Aero-tactile integration in speech perception

Visual information from a speaker's face can enhance or interfere with accurate auditory perception. This integration of information across auditory and visual streams has been observed in functional imaging studies, and has typically been attributed to the frequency and robustness with which perceivers jointly encounter event-specific information from these two modalities. Adding the tactile modality has long been considered a crucial next step in understanding multisensory integration. However, previous studies have found an influence of tactile input on speech perception only under limited circumstances, either where perceivers were aware of the task or where they had received training to establish a cross-modal mapping. Here we show that perceivers integrate naturalistic tactile information during auditory speech perception without previous training. Drawing on the observation that some speech sounds produce tiny bursts of aspiration (such as English 'p'), we applied slight, inaudible air puffs on participants' skin at one of two locations: the right hand or the neck. Syllables heard simultaneously with cutaneous air puffs were more likely to be heard as aspirated (for example, causing participants to mishear 'b' as 'p'). These results demonstrate that perceivers integrate event-relevant tactile information in auditory perception in much the same way as they do visual information.

Tactile enhancement of auditory and visual speech perception in untrained perceivers

A single pool of untrained subjects was tested for interactions across two bimodal perception conditions: audio-tactile, in which subjects heard and felt speech, and visual-tactile, in which subjects saw and felt speech. Identifications of English obstruent consonants were compared in bimodal and no-tactile baseline conditions. Results indicate that tactile information enhances speech perception by about 10 percent, regardless of which other mode (auditory or visual) is active. However, within-subject analysis indicates that individual subjects who benefit more from tactile information in one cross-modal condition tend to benefit less from tactile information in the other.

Language beyond action

The discovery of mirror neurons in macaques and of a similar system in humans has provided a new and fertile neurobiological ground for rooting a variety of cognitive faculties. Automatic sensorimotor resonance has been invoked as the key elementary process accounting for disparate (dys)functions, like imitation, ideomotor apraxia, autism, and schizophrenia. In this paper, we provide a critical appraisal of three of these claims that deal with the relationship between language and the motor system. Does language comprehension require the motor system? Was there an evolutionary switch from manual gestures to speech as the primary mode of language? Is human communication explained by automatic sensorimotor resonances? A positive answer to these questions would open the tantalizing possibility of bringing language and human communication within the fold of the motor system. We argue that the available empirical evidence does not appear to support these claims, and their theoretical scope fails to account for some crucial features of the phenomena they are supposed to explain. Without denying the enormous importance of the discovery of mirror neurons, we highlight the limits of their explanatory power for understanding language and communication.

Speech perception

This chapter focuses on one of the first steps in comprehending spoken language: How do listeners extract the most fundamental linguistic elements-consonants and vowels, or the distinctive features which compose them-from the acoustic signal? We begin by describing three major theoretical perspectives on the perception of speech. Then we review several lines of research that are relevant to distinguishing these perspectives. The research topics surveyed include categorical perception, phonetic context effects, learning of speech and related nonspeech categories, and the relation between speech perception and production. Finally, we describe challenges facing each of the major theoretical perspectives on speech perception.

The perception of tactile distance: influences of body site, space, and time

Vibrotactile prostheses for deaf or blind persons have been applied to any number of different locations on the body, including the finger, wrist, forearm, abdomen, back, and nape of the neck. The discriminability of patterns presented by such devices can be affected by the acuity of the site of application and the resolution of the display. In addition, the mutual influences among stimuli close together in both space and time can affect percepts within a broad range of parameters. For example, consideration must be given to a variety of tactile illusions often associated with the spatial separations and the range of temporal intervals typically used in cutaneous communication displays. Experiments are reported in which magnitude estimates and cross-modality matches of perceived extent produced by pairs of vibrotactile taps presented to separate loci were obtained on three different body sites. Perceived distance was directly related both to the timing between the taps and to their physical separation. The findings show a consistent relationship to cortical magnification across body sites.

The use of tactile supplements in lipreading Swedish and English: a single-subject study

The speech perception skills of GS, a Swedish adult deaf man who has used a "natural" tactile supplement to lipreading for over 45 years, were tested in two languages: Swedish and English. Two different tactile supplements to lipreading were investigated. In the first,"Tactiling," GS detected the vibrations accompanying speech by placing his thumb directly on the speaker's throat. In the second, a simple tactile aid consisting of a throat microphone, amplifier, and a hand-held bone vibrator was used. Both supplements led to improved lipreading of materials ranging in complexity from consonants in [aCa] nonsense syllables to Speech Tracking. Analysis of GS's results indicated that the tactile signal assisted him in identifying vowel duration, consonant voicing, and some manner of articulation categories. GS's tracking rate in Swedish was around 40 words per minute when the materials were presented via lipreading alone. When the lipreading signal was supplemented by tactile cues, his tracking rates were in the range of 60-65 words per minute. Although GS's tracking rates for English materials were around half those achieved in Swedish, his performance showed a similar pattern in that the use of tactile cues led to improvements of around 40% over lipreading alone.

A longitudinal evaluation of the speech perception capabilities of children using multichannel tactile vocoders

Thirty children (mean age 6:11, range 4:3 to 11:0, SD = 2:3) with profound hearing impairments were followed longitudinally over a 3-year period and evaluated every 6 months with a battery of speech perception tests. The battery spanned several levels of perception, from pattern perception to open-set word recognition. The children were all enrolled in a single full-day educational program that used multichannel tactile aids in addition to hearing aids. Testing was conducted in Auditory alone (A), Tactile plus Auditory (TA), Tactile alone (T), and in one instance, Tactile plus Auditory plus Vision (TAV) conditions because the primary interest of the work was the relationship between auditory and tactile training on perception. Results indicated that children's performance improved with age, with the oldest children achieving open-set speech recognition in the TA condition. Performance in the TA condition generally exceeded that in both A and T conditions. Outcomes were compared to those from two studies in the literature for children of similar age with cochlear implants and tactile aids on the same tests. Results suggest that performance of children who had cochlear implants for an average of 21 months was similar to TA and TAV performance of children in the present study who had tactile experience over a similar period.

FFT-based digital tactile vocoder system for real-time use

A microprocessor-based real-time digital vibrotactile vocoder system has been developed to train the deaf and for artificial hearing research. The system is composed of a microcomputer module with a digital signal processor, interface units and an attenuator/driver circuit. Live or digitised (stored or synthetic) speech is presented to the skin spectrally through a belt housing eight or 16 vibrators. Speech is processed in real time using a fast Fourier transform. The system is also capable of presenting any arbitrary spatiotemporal pattern on the skin for artificial hearing experiments. A preliminary experiment with a deaf subject indicates that the system is potentially an effective device for artificial hearing.

Crossmodal integration in the identification of consonant segments

Although speechreading can be facilitated by auditory or tactile supplements, the process that integrates cues across modalities is not well understood. This paper describes two "optimal processing" models for the types of integration that can be used in speechreading consonant segments and compares their predictions with those of the Fuzzy Logical Model of Perception (FLMP, Massaro, 1987). In "pre-labelling" integration, continuous sensory data is combined across modalities before response labels are assigned. In "post-labelling" integration, the responses that would be made under unimodal conditions are combined, and a joint response is derived from the pair. To describe pre-labelling integration, confusion matrices are characterized by a multidimensional decision model that allows performance to be described by a subject's sensitivity and bias in using continuous-valued cues. The cue space is characterized by the locations of stimulus and response centres. The distance between a pair of stimulus centres determines how well two stimuli can be distinguished in a given experiment. In the multimodal case, the cue space is assumed to be the product space of the cue spaces corresponding to the stimulation modes. Measurements of multimodal accuracy in five modern studies of consonant identification are more consistent with the predictions of the pre-labelling integration model than the FLMP or the post-labelling model.

Electrotactile and vibrotactile displays for sensory substitution systems

Sensory substitution systems provide their users with environmental information through a human sensory channel (eye, ear, or skin) different from that normally used, or with the information processed in some useful way. We review the methods used to present visual, auditory, and modified tactile information to the skin. First, we discuss present and potential future applications of sensory substitution, including tactile vision substitution (TVS), tactile auditory substitution, and remote tactile sensing or feedback (teletouch). Next, we review the relevant sensory physiology of the skin, including both the mechanisms of normal touch and the mechanisms and sensations associated with electrical stimulation of the skin using surface electrodes (electrotactile (also called electrocutaneous) stimulation). We briefly summarize the information-processing ability of the tactile sense and its relevance to sensory substitution. Finally, we discuss the limitations of current tactile display technologies and suggest areas requiring further research for sensory substitution systems to become more practical.

Discrimination of synthetic vowels by using tactile vocoder and a comparison to that of an eight-channel cochlear implant

A vowel discrimination test using a tactual vocoder was administered and the results were compared to that of an eight-channel cochlear implant. Both the tactile vocoder and the cochlear implant divided the speech signals into 16 frequency components using band-pass filters and lateral inhibition circuits. In the tactile vocoder, these 16 components were converted into a vibration with 200 Hz frequency and applied to a 3 x 16 element vibrator array using bimorph piezoelectric elements. The vibratory patterns were sensed on the fingertip. In the cochlear implant, the 16 components were reduced to eight current stimulation signals, consisting of biphasic pulses with 200 Hz frequency, which were applied to an eight-channel electrode array implanted in the scala tympani. The electrode array passed through the round window into the scala tympani to a depth of 23 mm. These psychophysical experiments investigate the ability of human subjects to discriminate synthetic vowels as a function of the number of channels employed. The results suggested that an eight-channel and a 16-channel tactile vocoder provided essentially the same discrimination scores. However, the ability to discriminate synthetic vowels decreased rapidly when less than eight channels were employed. The ability of an eight-channel tactile vocoder is expected to be better than that of the eight-channel cochlear implant because it is supposed that vowel discrimination is degraded by a phenomenon known as "current spreading" in the case of cochlear stimulation. However, the comparison between the two devices was not done on the cochlear implant subject.

Tactual perception of speech-like stimuli with an electrocutaneous vocoder

A deficiency in current tactual vocoder designs is the relatively poor transmission of rapid spectral changes (formant transitions). To understand better the application of electrocutaneous tactual stimulation as a basis for an artificial hearing system, the tactual discrimination of rapidly changing patterns was studied. Stimuli were presented on a belt containing 32 electrodes that was worn in a linear array 5 cm above the navel. Bipolar pulse trains (height 10 mA; width 13 microseconds) of specified frequencies (200, 400, or 1,000 Hz) were presented sequentially to adjacent electrodes to simulate a tactual vocoder equivalent of simple frequency transitions. Discrimination of the direction of stimulation movement (i.e., stimulation on the belt to either a subject's left or right) was assessed with one experienced subject as a function of both electrode number (four or eight electrodes) and stimulation frequency. Identification of the direction of stimulation movement was influenced both by the number of stimulus channels and by the frequency of electrocutaneous stimulation. These preliminary results indicate that the discrimination of formant trajectory transmitted in an electrocutaneous vocoder scheme is within the temporal limits typically imposed by natural speech.

The effects of an acquired profound hearing loss on speech production. A case study

Three parallel studies conducted with an early adolescent male subject deafened by meningitis at age 11 are reported. Study 1 plotted phonetic errors in spontaneous and read speech, long-term average spectra and vowel formant frequencies. Study 2 measured the subject's speech intelligibility using monosyllabic word lists. Study 3 involved normal hearer 's ratings of the subject's speech 2 months and 30 months after the onset of deafness. The results obtained revealed a deterioration in both speech intelligibility and quality.

Information transmission across the skin: high-resolution tactile sensory aids for the deaf and the blind

New techniques are described for implementing high-resolution communication interfaces to the skin, by direct electrical stimulation of the sense of touch. The psychophysical properties of these displays are summarized, and a practical application is presented: an electrotactile sensory aid for profoundly deaf children, which displays acoustic information on the skin of the abdomen, via a belt of electrotactile stimulators. The resulting tactile patterns provide cues for recognizing environmental sounds, enhance the accuracy of lipreading, and enable the child to monitor his/her own voice. The tactile patterns corresponding to sounds must be learned, much like acquiring a second language; substantial training and experience is required. Other applications are discussed, both for the rehabilitation of sensory handicaps and for tactile communication in general.

The bimodal perception of speech in infancy

Infants 18 to 20 weeks old recognize the correspondence between auditorially and visually presented speech sounds, and the spectral information contained in the sounds is critical to the detection of these correspondences. Some infants imitated the sounds presented during the experiment. Both the ability to detect auditory-visual correspondences and the tendency to imitate may reflect the infant's knowledge of the relationship between audition and articulation.

A video-recorded test of lipreading for British English

Tests of lipreading are valuable in many applications, particularly with hearing impaired people. Desirable properties of such a test are outlined, and the deficiencies of present tests noted. All the stated criteria are met by a new video recording of the BKB sentence lists. A calibration study with 22 normally-hearing subjects shows that 12 of the 21 lists may be considered equally difficult. Different scoring methods are investigated, including a subjective technique. Correlations among all the scoring methods are high. Practice effects can be large, and vary with lipreading ability. While low scorers show little increase in performance over the set of lists, high scorers show big improvements. Two examples of the test's usefulness are detailed--one from the laboratory and one from the clinic. Both demonstrate that hearing which would not lead to significant speech perception on its own can still greatly improve lipreading ability.

Electrocutaneous code pairs for artificial sensory communication systems

Pairs of electrocutaneous codes suitable for dual-channel sensory communication systems were compared using a dual-channel electrocutaneous tracking task. The tracking task required the test subject to dynamically respond to changes in the tactile sensation being modulated by two independent pseudorandom signals, one for each channel. The rule (or method) by which the signals changed the tactile sensations was called an electrocutaneous code. Four frequency variation codes and two intensity variation codes were paired in different combinations and then checked as to their effectiveness for sensory communications. The experimental protocol used a balanced incomplete block design which involved 24 subjects testing 3 of 8 code pairs each. Although the variance in the tracking performances between subjects was larger than the differences between the code pairs, learning rates for the various pairs were significantly different. The easiest one to learn was the Low Pulse Rate Modulation Code paired with itself. other findings included the general superiority of monophasic stimulation code pairs over biphasic stimulation code pairs, the need for placement of the two electrodes on different dermatomes in order to achieve satisfactory dual-channel communications, and the greater sensitivity to electrocutaneous stimulation of the ventral side of the forearm versus its dorsal side.