Optimum Information Transfer Rates for Communication through Haptic and Other Sensory Modalities

Exploring Bimanual Haptic Feedback for Spatial Search in Virtual Reality

Spatial search tasks are common and crucial in many Virtual Reality (VR) applications. Traditional methods to enhance the performance of spatial search often employ sensory cues such as visual, auditory, or haptic feedback. However, the design and use of bimanual haptic feedback with two VR controllers for spatial search in VR remains largely unexplored. In this work, we explored bimanual haptic feedback with various combinations of haptic properties, where four types of bimanual haptic feedback were designed, for spatial search tasks in VR. Two experiments were designed to evaluate the effectiveness of bimanual haptic feedback on spatial direction guidance and search in VR. The results from the first experiment reveal that our proposed bimanual haptic schemes significantly enhanced the recognition of spatial directions in terms of accuracy and speed compared to spatial audio feedback. The second experiment's findings suggest that the performance of bimanual haptic feedback was comparable to or even better than the visual arrow, especially in reducing the angle of head movement and enhancing searching targets behind the participants, which was supported by subjective feedback as well. Based on these findings, we have derived a set of design recommendations for spatial search using bimanual haptic feedback in VR.

Understanding the Effect of Speed on Human Emotion Perception in Mediated Social Touch Using Voice Coil Actuators

Touch as a modality in social communication has been getting more attention with recent developments in wearable technology and an increase in awareness of how limited physical contact can lead to touch starvation and feelings of depression. Although several mediated touch methods have been developed for conveying emotional support, the transfer of emotion through mediated touch has not been widely studied. This work addresses this need by exploring emotional communication through a novel wearable haptic system. The system records physical touch patterns through an array of force sensors, processes the recordings using novel gesture-based algorithms to create actuator control signals, and generates mediated social touch through an array of voice coil actuators. We conducted a human subject study (N = 20) to understand the perception and emotional components of this mediated social touch for common social touch gestures, including poking, patting, massaging, squeezing, and stroking. Our results show that the speed of the virtual gesture significantly alters the participants' ratings of valence, arousal, realism, and comfort of these gestures with increased speed producing negative emotions and decreased realism. The findings from the study will allow us to better recognize generic patterns from human mediated touch perception and determine how mediated social touch can be used to convey emotion. Our system design, signal processing methods, and results can provide guidance in future mediated social touch design.

Immersive reading in virtual and augmented reality environment

Purpose

This study aims to investigate “immersive reading,” which occurs when individuals read text while in a virtual reality (VR) or augmented reality (AR) environment.

Design/methodology/approach

In Experiment 1, 64 participants read text passages and answered multiple-choice questions in VR and AR head-mounted displays (HMDs) compared with doing the same task on liquid crystal display (LCD). In Experiment 2, 31 participants performed the same reading tasks but with two VR HMDs of different display quality.

Findings

Compared with reading on LCD as the baseline, participants reading in VR and AR HMDs got 82% (VR) and 88% (AR) of the information accurately. Participants tended to respond more accurately and faster, though not statistically significant, with the VR HMD of higher pixel density in the speed-reading task.

Originality/value

The authors observed the speed and accuracy of reading in VR and AR environments, compared with the reading speed and accuracy on an LCD monitor. The authors also compared the reading performance on two VR HMDs that differed in display quality but were otherwise similar in every way.

Body-Mounted Vibrotactile Stimuli: Simultaneous Display of Taps on the Fingertips and Forearm

In this article, we present a body-mounted tactile display to deliver haptic feedback to the forearm and user-initiated haptic feedback to the fingertips. The display mounts two vibrotactile actuators on the forearm, leaving the user's hands free for manipulation tasks when the hands are not interacting with the tactile display, while also exploiting the tactile sensitivity of the fingertips when needed. We test the effectiveness of the display using paired vibrotactile taps sensed through the forearm and the fingertips, either separately or simultaneously. We measure the ability of participants to identify the vibrotactile taps. The results show that mounting the device on the forearm, so that the participant touches the forearm-mounted device with their fingertips receiving feedback to both locations simultaneously, decreases performance relative to mounting on the fingertips unless large amplitudes are used. We also test the accuracy with which participants identified different numbers of vibration taps (4, 8, 16, and 25 signals). The results show that as the number of signals changes, participant accuracy is not different when stimulating the fingertips alone compared to stimulating the fingertips and forearm together. We conclude with an example of a portable and wearable vibration display, and discuss future use cases of such a display.

A Multisensory Approach to Present Phonemes as Language Through a Wearable Haptic Device

Communication is an important part of our daily interactions; however, communication can be hindered, either through visual or auditory impairment, or because usual communication channels are overloaded. When standard communication channels are not available, our sense of touch offers an alternative sensory modality for transmitting messages. Multi-sensory haptic cues that combine multiple types of haptic sensations have shown promise for applications, such as haptic communication, that require large discrete cue sets while maintaining a small, wearable form factor. This article presents language transmission using a multi-sensory haptic device that occupies a small footprint on the upper arm. In our approach, phonemes are encoded as multisensory haptic cues consisting of vibration, radial squeeze, and lateral skin stretch components. Participants learned to identify haptically transmitted phonemes and words after training across a four day training period. A subset of our participants continued their training to extend word recognition free response. Participants were able to identify words after four days using multiple choice with an accuracy of 89% and after eight days using free response with an accuracy of 70%. These results show promise for the use of multisensory haptics for haptic communication, demonstrating high word recognition performance with a small, wearable device.

Multi-Sensory Stimuli Improve Distinguishability of Cutaneous Haptic Cues

Wearable haptic systems offer portable, private tactile communication to a human user. To date, advances in wearable haptic devices have typically focused on the optimization of haptic cue transmission using a single modality, or have combined two types of cutaneous feedbacks, each mapped to a particular parameter of the task. Alternatively, researchers have employed arrays of haptic tactile actuators to maximize information throughput to a user. However, when large cue sets are to be transmitted, such as those required to communicate language, perceptual interference between transmitted cues can decrease the efficacy of single-sensory systems, or require large footprints to ensure salient spatiotemporal cues are rendered to the user. In this paper, we present a wearable, multi-sensory haptic feedback system, MISSIVE (Multi-sensory Interface of Stretch, Squeeze, and Integrated Vibration Elements), that conveys multi-sensory haptic cues to the user's upper arm. We present experimental results that demonstrate that rendering haptic cues with multi-sensory components-specifically, lateral skin stretch, radial squeeze, and vibrotactile stimuli-improved perceptual distinguishability in comparison to similar cues with all-vibrotactile components. These results support the incorporation of diverse stimuli, both vibrotactile and nonvibrotactile, for applications requiring large haptic cue sets.

Evaluation of the cognitive-motor performance of adults with Duchenne Muscular Dystrophy in a hand-related task

Duchenne muscular Dystrophy (DMD) is a progressive degenerative muscle disease, affecting, among others, the upper extremities. Effective hand rehabilitation can improve the hand function of people with DMD. To reach this goal, we first need to gain more insight into the hand cognitive-motor performance of people with DMD. This is the first study employing a systematic analysis on multi-finger, cognitive-motor performance of people with DMD. For this purpose, we propose an active dynamic visuo-motor task. The task employed six visual stimuli, a subset of which was activated at each trial. The stimuli were activated with a frequency of 1, 2, 3 and 4 Hz. Eight healthy participants and three participants with DMD performed the task. Additionally, the healthy participants performed seven sessions, and we assessed the training effects. Task-related cognitive-motor performance was evaluated using information transfer rate (ITR) and perceived workload. Regarding ITR, healthy participants performed significantly better than DMD participants; however, this was more evident for trials involving more than three fingers. Workload showed no difference between the healthy and the DMD groups. Healthy participants significantly improved their performance during training. Our results suggest that hand rehabilitation of people with DMD should consider multi-finger dynamic training. However, additional research with more people with DMD is needed for further generalization of our conclusions.

Electrically-Evoked Proximity Sensation Can Enhance Fine Finger Control in Telerobotic Pinch

For teleoperation tasks requiring high control accuracy, it is essential to provide teleoperators with information on the interaction between the end effector and the remote environment. Real-time imaging devices have been widely adopted, but it delivers limited information, especially when the end effectors approach the target following the line-of-sight. In such situations, teleoperators rely on the perspective at the screen and can apply high force unintentionally at the initial contact. This research proposes to deliver the distance information at teleoperation to the fingertips of teleoperators, i.e., proximity sensation. Transcutaneous electrical stimulation was applied onto the fingertips of teleoperators, with the pulsing frequency inversely proportional to the distance. The efficacy of the proximity sensation was evaluated by the initial contact force during telerobotic pinch in three sensory conditions: vision only, vision + visual assistance (distance on the screen), and vision + proximity sensation. The experiments were repeated at two viewing angles: 30–60° and line-of-sight, for eleven healthy human subjects. For both cases, the initial contact force could be significantly reduced by either visual assistance (20–30%) or the proximity sensation (60–70%), without additional processing time. The proximity sensation is two-to-three times more effective than visual assistance regarding the amount of force reduction.

ActivEarring: Spatiotemporal Haptic Cues on the Ears

The symmetric configuration and the sensitivity of ears in addition to the long tradition of earrings as adornment open up the possibility for smart ear-worn devices. Taking advantage of these attributes, past research mostly focused on creating novel unobtrusive sensing input devices and auditory displays placed on the ear. Meanwhile, the tactile sensitivity of the ear has long been overshadowed by its auditory capacity, presenting the opportunity to investigate how ears can be exploited for unobtrusive tactile information transfer. With three studies and a total of 38 participants, we suggest the design of ActivEarring, a ear-worn device capable of imparting information by stimulating six different locations on both ears. We evaluated the performance of ActivEarring in a semi-realistic mobile condition and its practical use for information transfer with spatiotemporal patterns. Finally, we demonstrate that ActivEarring can be incorporated in common jewelry design, and present three applications that illustrate promising usage scenarios.

Haptic Enchanters: Attachable and Detachable Vibrotactile Modules and Their Advantages

This paper showcases attachable/detachable haptic modules, nicknamed haptic enchanters, by presenting their conceptual prototypes and quantifying their information transmission capacity. Haptic enchanters can be attached to ordinary devices and wearables to endow them with the ability of creating programmable haptic stimuli, thereby transforming them to effective haptic communication devices. In particular, our prototype enchanters are designed to localize vibrotactile stimulation to the neighborhood of attachment site, and this spatial isolation allows for effective spatiotemporal information delivery using multiple enchanters. We demonstrate that haptic enchanters enable very high capacity of information transfer (4.55-7.06 bits) for spatiotemporal vibration sequences rendered with two, three, and four enchanters. We also assert that haptic enchanters provide better user experiences by using location-enabled feedback in a game application than the whole-device feedback. These results instantiate the performance of haptic enchanters as effective and convenient communication accessories and their potential for applications.

Sensory integration of apparent motion speed and vibration magnitude

Tactile apparent motion can display directional information in an intuitive way. It can for example be used to give directions to visually impaired individuals, or for waypoint navigation while cycling on busy streets, when vision or audition should not be loaded further. However, although humans can detect very short tactile patterns, discriminating between similar motion speeds has been shown to be difficult. Here we develop and investigate a method where the speed of tactile apparent motion around the user's wrist is coupled with vibration magnitude. This redundant coupling is used to produce tactile patterns from slow&weak to fast&strong. We compared the just noticeable difference (JND) of the coupled and the individual variables. The results show that the perception of the coupled variable can be characterised by JND smaller than JNDs of the individual variables. This allowed us to create short tactile pattens (tactons) for display of direction and speed, which can be distinguished significantly better than tactons based on motion alone. Additionally, most subjects were also able to identify the coupled-variable tactons better than the magnitude-based tactons.

Shape Localization and Recognition Using a Magnetorheological-Fluid Haptic Display

Smart materials such as magnetorheological fluids (MRF) offer an interesting technology for use in haptic displays as changes in the magnetic field are rapid, reversible, and controllable. These interfaces have been evaluated in a number of medical and surgical simulators where they can provide cues regarding the viscoelastic properties of tissues. The objective of the present set of experiments was first to determine whether a shape embedded in the MRF could be precisely localized and second whether 10 shapes rendered in a MRF haptic display could be accurately identified. It was also of interest to determine how the information transfer associated with this type of haptic display compares to that achieved using other haptic channels of communication. The overall performance of participants at identifying the shapes rendered in the MRF was good with a mean score of 73 percent correct and an Information Transfer (IT) of 2.2 bits. Participants could also localize a rigid object in the display accurately. These findings indicate that this technology has potential for use in training manual palpation skills and in exploring haptic shape perception in dynamic environments.

Quantification of information transfer rate of the human hand during a mouse clicking task with healthy adults and one adult with Duchenne muscular Dystrophy

Duchenne muscular Dystrophy (DMD) Is a progressive muscle degenerative disease. Active hand assistive devices, can improve the quality of life of people with DMD. Such devices show a rejection rate due to complexity. Our hypothesis is, that a simple orthosis might prove more functional and realistic in assisting people with DMD. To investigate, we developed a portable setup that provides various visual stimuli and records the response of the subjects' fingers through a mouse clicking task. Six LEDs served as visual stimuli. The subjects' responses were obtained through mechanical interaction with two vertical mice. Different combinations of frequencies and numbers of stimuli were tested with 8 healthy subjects and one with DMD. Performance was evaluated in terms of information transfer rate (ITR), pattern accuracy and perceived workload. The outcome shows that lower complexity results in lower ITR and lower workload for all subjects. While for healthy subjects, maximum ITR was 4.3 bits/s, for DMD maximum ITR was 2.5 bits/s. Both maxima were achieved at the same trial (3 fingers at 2 Hz). This trial agrees with a pareto optimization analysis of ITR with respect to workload. The results support our hypothesis for a simple yet functional solution. Furthermore healthy subjects and the individual with DMD, in principal show similar finger control, albeit with lower absolute performance.

Displaying shape haptically using MRF-based device

Smart materials such as magnetorheological fluids (MRF) offer an interesting medium to present viscoelastic cues in haptic displays as changes in the magnetic field are rapid, reversible and controllable. These interfaces have been evaluated in a number of medical and surgical simulators where they can provide cues regarding the viscoelastic properties of tissues. The present experiment determined whether eight different shapes could be identified reliably with a MRF haptic display and compared the information transfer (IT) associated with this type of display with that achieved by other forms of haptic communication. The overall performance of participants at identifying the shapes was good with a mean score of 70% correct and an IT of 2.13 bits. This type of display shows promise as a training tool for simulating tissue properties.

Evaluating vibrotactile dimensions for the design of tactons

Vibrotactile stimuli are defined in terms of their amplitude, frequency, waveform and temporal profile all of which have been varied to create tactons. A number of approaches have been adopted to design tactons including multidimensional scaling, iterative empirical methods and using perceptual processing models. The objective of the present set of experiments was to create sets of tactons based on the properties of the dimensions of vibrotactile stimuli. An absolute identification paradigm was used in which each of nine tactons was presented eight times using a tactor mounted on either the index finger or forearm. It was found that tactons created by varying the frequency, amplitude and temporal profile of the vibrotactile stimuli were correctly identified on 73-83 percent of the trials, with a mean information transfer of 2.41 bits. The latter metric indicates that for these sets of nine tactons between five and six could be reliably identified. The vibrotactile stimuli delivered in the experiments were identified as consistently on the forearm as the hand and the IT values were similar at the two locations. This suggests that sites other than the hand can be used effectively in tactile communication systems and that it is channel capacity that ultimately determines performance on this type of task.

Application of psychophysical techniques to haptic research

Various psychophysical methods have been used to study human haptic perception, although the selection of a particular method is often based on convention, rather than an analysis of which technique is optimal for the question being addressed. In this review, classical psychophysical techniques used to measure sensory thresholds are described as well as more modern methods such as adaptive procedures and those associated with signal detection theory. Details are provided as to how these techniques should be implemented to measure absolute and difference thresholds and factors that influence subjects' responses are noted. In addition to the methods used to measure sensory thresholds, the techniques available for measuring the perception of suprathreshold stimuli are presented. These scaling methods are reviewed in the context of the various stimulus and response biases that influence how subjects respond to stimuli. The importance of understanding the factors that influence perceptual processing is highlighted throughout the review with reference to experimental studies of haptic perception.

Mechanical and psychophysical studies of surface wave propagation during vibrotactile stimulation

Tactile displays are often used to present spatial cues about the environment, although the optimal configuration of a display used for spatial cuing is not known. The objective of the present set of experiments was to characterize the properties of surface waves induced by vibrotactile stimulation and to determine if the propagation of surface waves was a factor influencing the ability to localize a point of stimulation in a tactile display. Three sites on the body were tested: the palm of the hand, the forearm, and the thigh. An accelerometer array was fabricated and used to measure the surface waves. The results indicated that there were significant differences between glabrous and hairy skin in terms of the frequency and amplitude of oscillation of the motor. Analyses of the motion of the surface waves across the skin indicated that they were markedly attenuated at 8 mm from the motor, but even at 24 mm the amplitude was still above perceptual threshold. The localization experiment indicated that subjects were much better at identifying the site of stimulation on the palm as compared to the forearm and thigh, and that the latter two sites were not significantly different.

Force feedback facilitates multisensory integration during robotic tool use

The present study investigated the effects of force feedback in relation to tool use on the multisensory integration of visuo-tactile information. Participants learned to control a robotic tool through a surgical robotic interface. Following tool-use training, participants performed a crossmodal congruency task, by responding to tactile vibrations applied to their hands, while ignoring visual distractors superimposed on the robotic tools. In the first experiment it was found that tool-use training with force feedback facilitates multisensory integration of signals from the tool, as reflected in a stronger crossmodal congruency effect with the force feedback training compared to training without force feedback and to no training. The second experiment extends these findings by showing that training with realistic online force feedback resulted in a stronger crossmodal congruency effect compared to training in which force feedback was delayed. The present study highlights the importance of haptic information for multisensory integration and extends findings from classical tool-use studies to the domain of robotic tools. We argue that such crossmodal congruency effects are an objective measure of robotic tool integration and propose some potential applications in surgical robotics, robotic tools, and human-tool interaction.

Design and Evaluation of Identifiable Key-Click Signals for Mobile Devices

As touch based input becomes more popular in mobile devices, there is an increasing need for haptic feedback on key-less input surface. Four experiments were conducted to design and evaluate identifiable emulated key-click signals using a piezoelectric actuator. Experiments I and II assessed the information transmission capacity for the amplitude, frequency, and number of cycles of raised cosine waveforms used to drive the piezo actuators under fixed- and roving-background conditions, respectively. Experiment III estimated the total information transfer for all three parameters. The results were used to reduce the number of stimulus alternatives in the key-click signal set with the goal to achieve perfect identification performance. Experiment IV verified that up to 5 to 6 identifiable key-click signals could be achieved with the experimental setup. The present study outlines an information theoretic approach to conducting identification experiments to guide the design of and to evaluate a perfectly identifiable stimulus set. The methodology can be applied to other applications in need of perceptually identifiable stimulation patterns.