Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures

Tactile, Audio, and Visual Dataset During Bare Finger Interaction with Textured Surfaces

This paper presents a comprehensive multi-modal dataset capturing concurrent haptic, audio, and visual signals recorded from ten participants as they interacted with ten different textured surfaces using their bare fingers. The dataset includes stereoscopic images of the textures, and fingertip position, speed, applied load, emitted sound, and friction-induced vibrations, providing an unprecedented insight into the complex dynamics underlying human tactile perception. Our approach utilizes a human finger (while most previous studies relied on rigid sensorized probes), enabling the naturalistic acquisition of haptic data and addressing a significant gap in resources for studies of human tactile exploration, perceptual mechanisms, and artificial tactile perception. Additionally, fifteen participants completed a questionnaire to evaluate their subjective perception of the surfaces. Through carefully designed data collection protocols, encompassing both controlled and free exploration scenarios, this dataset offers a rich resource for studying human multi-sensory integration and supports the development of algorithms for texture recognition based on multi-modal inputs. A preliminary analysis demonstrates the dataset's potential, as classifiers trained on different combinations of data modalities show promising accuracy in surface identification, highlighting its value for advancing research in multi-sensory perception and the development of human-machine interfaces.

Visually Impaired People Learning Virtual Textures Through Multimodal Feedback Combining Vibrotactile and Voice

In recent years, various haptic rendering methods have been proposed to help people obtain interactive experiences with virtual textures through vibration feedback. However, due to impaired vision, the blind or visually impaired (BVI) is still unable to effectively perceive and learn virtual textures through these methods. To help BVIs have the opportunity to improve their object cognition by learning virtual textures, we built a virtual texture learning system based on multimodal feedback. We first propose an Informer based haptic texture rendering model that can fuse texture images with real-time action information to generate vibration acceleration (VA) signals. We further propose a texture classification method using the generated VA signals, and broadcast the classified texture description information to BVI through a speaker. We described the construction process of rendering model and classification method in detail, and compared the perceptual effects of subjects on textures under four rendering models through user experiments, as well as the accuracy of texture matching under two learning modes. The experimental results show that the proposed rendering model can accurately and efficiently generate VA signals, providing subjects with realistic vibration feedback. The constructed learning system enables BVI to know the type, material and other attribute information of virtual texture in the process of obtaining vibrotactile sensation. By establishing the correspondence between haptic stimuli and texture attributes, the system enables BVIs to enhance their ability to recognize objects through learning a large number of virtual textures.

Robust Surface Recognition With the Maximum Mean Discrepancy: Degrading Haptic-Auditory Signals Through Bandwidth and Noise

Sliding a tool across a surface generates rich sensations that can be analyzed to recognize what is being touched. However, the optimal configuration for capturing these signals is yet unclear. To bridge this gap, we consider haptic-auditory data as a human explores surfaces with different steel tools, including accelerations of the tool and finger, force and torque applied to the surface, and contact sounds. Our classification pipeline uses the maximum mean discrepancy (MMD) to quantify differences in data distributions in a high-dimensional space for inference. With recordings from three hemispherical tool diameters and ten diverse surfaces, we conducted two degradation studies by decreasing sensing bandwidth and increasing added noise. We evaluate the haptic-auditory recognition performance achieved with the MMD to compare newly gathered data to each surface in our known library. The results indicate that acceleration signals alone have great potential for high-accuracy surface recognition and are robust against noise contamination. The optimal accelerometer bandwidth exceeds 1000 Hz, suggesting that useful vibrotactile information extends beyond human perception range. Finally, smaller tool tips generate contact vibrations with better noise robustness. The provided sensing guidelines may enable superhuman performance in portable surface recognition, which could benefit quality control, material documentation, and robotics.

Establishing haptic texture attribute space and predicting haptic attributes from image features using 1D-CNN

The current study strives to provide a haptic attribute space where texture surfaces are located based on their haptic attributes. The main aim of the haptic attribute space is to come up with a standardized model for representing and identifying haptic textures analogous to the RGB model for colors. To this end, a four dimensional haptic attribute space is established by conducting a psychophysical experiment where human participants rate 100 real-life texture surfaces according to their haptic attributes. The four dimensions of the haptic attribute space are rough-smooth, flat-bumpy, sticky-slippery, and hard-soft. The generalization and scalability of the haptic attribute space is achieved by training a 1D-CNN model for predicting attributes of haptic textures. The 1D-CNN is trained using the attribute data from psychophysical experiments and image features collected from the images of real textures. The prediction power granted by the 1D-CNN renders scalability to the haptic attribute space. The prediction accuracy of the proposed 1D-CNN model is compared against other machine learning and deep learning algorithms. The results show that the proposed method outperforms the other models on MAE and RMSE metrics.

Learning to Feel Textures: Predicting Perceptual Similarities From Unconstrained Finger-Surface Interactions

Whenever we touch a surface with our fingers, we perceive distinct tactile properties that are based on the underlying dynamics of the interaction. However, little is known about how the brain aggregates the sensory information from these dynamics to form abstract representations of textures. Earlier studies in surface perception all used general surface descriptors measured in controlled conditions instead of considering the unique dynamics of specific interactions, reducing the comprehensiveness and interpretability of the results. Here, we present an interpretable modeling method that predicts the perceptual similarity of surfaces by comparing probability distributions of features calculated from short time windows of specific physical signals (finger motion, contact force, fingernail acceleration) elicited during unconstrained finger-surface interactions. The results show that our method can predict the similarity judgments of individual participants with a maximum Spearman's correlation of 0.7. Furthermore, we found evidence that different participants weight interaction features differently when judging surface similarity. Our findings provide new perspectives on human texture perception during active touch, and our approach could benefit haptic surface assessment, robotic tactile perception, and haptic rendering.

Perceptual Quality Assessment of Compressed Vibrotactile Signals Through Comparative Judgment

In this article, we present a comprehensive scheme for the quality assessment of compressed vibrotactile signals with human assessors. Inspired by the multiple stimulus test with hidden reference and anchors (MUSHRA) from the audio domain, we designed a method in which each compressed signal is compared to its original signal and rated on a numerical scale. For each signal tested, the hidden reference and two anchor signals are used to validate the results and provide assessor screening criteria. Differing from previous approaches, our method is hierarchically structured and strictly timed in a sequential manner to avoid experimental confounds and provide precise psychophysical assessments. We validated our method in an experiment with 20 human participants in which we compared two state-of-the-art lossy codecs. The results show that, with our approach, the performance of different codecs can be compared effectively. Furthermore, the method also provides a measure of subjective quality at different data compression rates. The proposed procedure can be easily adapted to evaluate other vibrotactile codecs.