Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?

Full freedom-of-motion actuators as advanced haptic interfaces

The sense of touch conveys critical environmental information, facilitating object recognition, manipulation, and social interaction, and can be engineered through haptic actuators that stimulate cutaneous receptors. An unfulfilled challenge lies in haptic interface technologies that can engage all the various mechanoreceptors in a programmable, spatiotemporal fashion across large areas of the body. Here, we introduce a small-scale actuator technology that can impart omnidirectional, superimposable, dynamic forces to the surface of skin, as the basis for stimulating individual classes of mechanoreceptors or selected combinations of them. High-bit haptic information transfer and realistic virtual tactile sensations are possible, as illustrated through human subject perception studies in extended reality applications that include advanced hand navigation, realistic texture reproduction, and sensory substitution for music perception.

Real-Time Musical Haptics With Ultra-Wideband: A Study on Latency, Reliability, and Perception

Ultra wideband (UWB) radios are popular for accurate distance estimation between devices. However, UWB also offers low-power, fast, reliable wireless communication. We exploit it here in a real-time musical haptics system for live performances: a wearable, wirelessly activated via UWB by the performer's instrument, augments the audience musical experience with a tactile sensory layer. Two challenges are crucial to the experience quality: i) communication must be reliable, to prevent corruption of tactile signals, and ii) these must reach the audience synchronously with the instrument sounds. We perform micro-benchmarks of UWB links alone in a controlled setup, showing that the haptic signal can be delivered reliably over UWB before the instrument sound, thus enabling proper compensation delays to perfectly realign sound and tactile vibration. We confirm this holds on the end-to-end system including haptic components by characterizing four proof-of-concept prototypes combining different UWB-enabled instruments and wearables. Finally, we reconcile these objective measures with subjective ones via a user study focusing on perception, yielding very positive outcomes. Together, these results confirm the potential of UWB-based musical haptics for enhancing the audience experience at live performances in ways hitherto unexplored.

A Narrative Review of Haptic Technologies and Their Value for Training, Rehabilitation, and the Education of Persons with Special Needs

Haptic technologies are increasingly valuable for human-computer interaction in its many flavors, including, of course, virtual reality systems, which are becoming very useful tools for education, training, and rehabilitation in many areas of medicine, engineering, and daily life. There is a broad spectrum of technologies and approaches that provide haptic stimuli, ranging from the well-known force feedback to subtile pseudo-haptics and visual haptics. Correspondingly, there is a broad spectrum of applications and system designs that include haptic technologies as a relevant component and interaction feature. Paramount is their use in training of medical procedures, but they appear in a plethora of systems deploying virtual reality applications. This narrative review covers the panorama of haptic devices and approaches and the most salient areas of application. Special emphasis is given to education of persons with special needs, aiming to foster the development of innovative systems and methods addressing the enhancement of the quality of life of this segment of the population.

Improved tactile speech perception and noise robustness using audio-to-tactile sensory substitution with amplitude envelope expansion

Recent advances in haptic technology could allow haptic hearing aids, which convert audio to tactile stimulation, to become viable for supporting people with hearing loss. A tactile vocoder strategy for audio-to-tactile conversion, which exploits these advances, has recently shown significant promise. In this strategy, the amplitude envelope is extracted from several audio frequency bands and used to modulate the amplitude of a set of vibro-tactile tones. The vocoder strategy allows good consonant discrimination, but vowel discrimination is poor and the strategy is susceptible to background noise. In the current study, we assessed whether multi-band amplitude envelope expansion can effectively enhance critical vowel features, such as formants, and improve speech extraction from noise. In 32 participants with normal touch perception, tactile-only phoneme discrimination with and without envelope expansion was assessed both in quiet and in background noise. Envelope expansion improved performance in quiet by 10.3% for vowels and by 5.9% for consonants. In noise, envelope expansion improved overall phoneme discrimination by 9.6%, with no difference in benefit between consonants and vowels. The tactile vocoder with envelope expansion can be deployed in real-time on a compact device and could substantially improve clinical outcomes for a new generation of haptic hearing aids.

Vibrotactile enhancement of musical engagement

Sound is sensed by the ear but can also be felt on the skin, by means of vibrotactile stimulation. Only little research has addressed perceptual implications of vibrotactile stimulation in the realm of music. Here, we studied which perceptual dimensions of music listening are affected by vibrotactile stimulation and whether the spatial segregation of vibrations improves vibrotactile stimulation. Forty-one listeners were presented with vibrotactile stimuli via a chair's surfaces (left and right arm rests, back rest, seat) in addition to music presented over headphones. Vibrations for each surface were derived from individual tracks of the music (multi condition) or conjointly by a mono-rendering, in addition to incongruent and headphones-only conditions. Listeners evaluated unknown music from popular genres according to valence, arousal, groove, the feeling of being part of a live performance, the feeling of being part of the music, and liking. Results indicated that the multi- and mono vibration conditions robustly enhanced the nature of the musical experience compared to listening via headphones alone. Vibrotactile enhancement was strong in the latent dimension of 'musical engagement', encompassing the sense of being a part of the music, arousal, and groove. These findings highlight the potential of vibrotactile cues for creating intensive musical experiences.

Improved tactile speech robustness to background noise with a dual-path recurrent neural network noise-reduction method

Many people with hearing loss struggle to understand speech in noisy environments, making noise robustness critical for hearing-assistive devices. Recently developed haptic hearing aids, which convert audio to vibration, can improve speech-in-noise performance for cochlear implant (CI) users and assist those unable to access hearing-assistive devices. They are typically body-worn rather than head-mounted, allowing additional space for batteries and microprocessors, and so can deploy more sophisticated noise-reduction techniques. The current study assessed whether a real-time-feasible dual-path recurrent neural network (DPRNN) can improve tactile speech-in-noise performance. Audio was converted to vibration on the wrist using a vocoder method, either with or without noise reduction. Performance was tested for speech in a multi-talker noise (recorded at a party) with a 2.5-dB signal-to-noise ratio. An objective assessment showed the DPRNN improved the scale-invariant signal-to-distortion ratio by 8.6 dB and substantially outperformed traditional noise-reduction (log-MMSE). A behavioural assessment in 16 participants showed the DPRNN improved tactile-only sentence identification in noise by 8.2%. This suggests that advanced techniques like the DPRNN could substantially improve outcomes with haptic hearing aids. Low-cost haptic devices could soon be an important supplement to hearing-assistive devices such as CIs or offer an alternative for people who cannot access CI technology.

Improving speech perception for hearing-impaired listeners using audio-to-tactile sensory substitution with multiple frequency channels

Cochlear implants (CIs) have revolutionised treatment of hearing loss, but large populations globally cannot access them either because of disorders that prevent implantation or because they are expensive and require specialist surgery. Recent technology developments mean that haptic aids, which transmit speech through vibration, could offer a viable low-cost, non-invasive alternative. One important development is that compact haptic actuators can now deliver intense stimulation across multiple frequencies. We explored whether these multiple frequency channels can transfer spectral information to improve tactile phoneme discrimination. To convert audio to vibration, the speech amplitude envelope was extracted from one or more audio frequency bands and used to amplitude modulate one or more vibro-tactile tones delivered to a single-site on the wrist. In 26 participants with normal touch sensitivity, tactile-only phoneme discrimination was assessed with one, four, or eight frequency bands. Compared to one frequency band, performance improved by 5.9% with four frequency bands and by 8.4% with eight frequency bands. The multi-band signal-processing approach can be implemented in real-time on a compact device, and the vibro-tactile tones can be reproduced by the latest compact, low-powered actuators. This approach could therefore readily be implemented in a low-cost haptic hearing aid to deliver real-world benefits.

Recent Developments in Haptic Devices Designed for Hearing-Impaired People: A Literature Review

Haptic devices transmit information to the user, using tactile stimuli to augment or replace sensory input. People with limited sensory abilities, such as vision or hearing can receive supplementary information by relying on them. This review analyses recent developments in haptic devices for deaf and hard-of-hearing individuals by extracting the most relevant information from each of the selected papers. The process of finding relevant literature is detailed using the PRISMA guidelines for literature reviews. In this review, the devices are categorized to better understand the review topic. The categorization results have highlighted several areas of future research into haptic devices for hearing-impaired users. We believe this review may be useful to researchers interested in haptic devices, assistive technologies, and human-computer interaction.

Effect of audio-tactile congruence on vibrotactile music enhancement

Music listening experiences can be enhanced with tactile vibrations. However, it is not known which parameters of the tactile vibration must be congruent with the music to enhance it. Devices that aim to enhance music with tactile vibrations often require coding an acoustic signal into a congruent vibrotactile signal. Therefore, understanding which of these audio-tactile congruences are important is crucial. Participants were presented with a simple sine wave melody through supra-aural headphones and a haptic actuator held between the thumb and forefinger. Incongruent versions of the stimuli were made by randomizing physical parameters of the tactile stimulus independently of the auditory stimulus. Participants were instructed to rate the stimuli against the incongruent stimuli based on preference. It was found making the intensity of the tactile stimulus incongruent with the intensity of the auditory stimulus, as well as misaligning the two modalities in time, had the biggest negative effect on ratings for the melody used. Future vibrotactile music enhancement devices can use time alignment and intensity congruence as a baseline coding strategy, which improved strategies can be tested against.