AI-Driven Sensing Technology: Review

Machine learning and deep learning technologies are rapidly advancing the capabilities of sensing technologies, bringing about significant improvements in accuracy, sensitivity, and adaptability. These advancements are making a notable impact across a broad spectrum of fields, including industrial automation, robotics, biomedical engineering, and civil infrastructure monitoring. The core of this transformative shift lies in the integration of artificial intelligence (AI) with sensor technology, focusing on the development of efficient algorithms that drive both device performance enhancements and novel applications in various biomedical and engineering fields. This review delves into the fusion of ML/DL algorithms with sensor technologies, shedding light on their profound impact on sensor design, calibration and compensation, object recognition, and behavior prediction. Through a series of exemplary applications, the review showcases the potential of AI algorithms to significantly upgrade sensor functionalities and widen their application range. Moreover, it addresses the challenges encountered in exploiting these technologies for sensing applications and offers insights into future trends and potential advancements.

Handloomed fabrics recognition with deep learning

Every nation treasures its handloom heritage, and in India, the handloom industry safeguards cultural traditions, sustains millions of artisans, and preserves ancient weaving techniques. To protect this legacy, a critical need arises to distinguish genuine handloom products, exemplified by the renowned "gamucha" from India's northeast, from counterfeit powerloom imitations. Our study's objective is to create an AI tool for effortless detection of authentic handloom items amidst a sea of fakes. Six deep learning architectures-VGG16, VGG19, ResNet50, InceptionV3, InceptionResNetV2, and DenseNet201-were trained on annotated image repositories of handloom and powerloom towels (17,484 images in total, with 14,020 for training and 3464 for validation). A novel deep learning model was also proposed. Despite respectable training accuracies, the pre-trained models exhibited lower performance on the validation dataset compared to our novel model. The proposed model outperformed pre-trained models, demonstrating superior validation accuracy, lower validation loss, computational efficiency, and adaptability to the specific classification problem. Notably, the existing models showed challenges in generalizing to unseen data and raised concerns about practical deployment due to computational expenses. This study pioneers a computer-assisted approach for automated differentiation between authentic handwoven "gamucha"s and counterfeit powerloom imitations-a groundbreaking recognition method. The methodology presented not only holds scalability potential and opportunities for accuracy improvement but also suggests broader applications across diverse fabric products.

Classification and Identification of Contaminants in Recyclable Containers Based on a Recursive Feature Elimination-Light Gradient Boosting Machine Algorithm Using an Electronic Nose

Establishing an excellent recycling mechanism for containers is of great importance for environmental protection, so many technical approaches applied during the whole recycling stage have become popular research issues. Among them, classification is considered a key step, but this work is mostly achieved manually in practical applications. Due to the influence of human subjectivity, the classification accuracy often varies significantly. In order to overcome this shortcoming, this paper proposes an identification method based on a Recursive Feature Elimination-Light Gradient Boosting Machine (RFE-LightGBM) algorithm using electronic nose. Firstly, odor features were extracted, and feature datasets were then constructed based on the response data of the electronic nose to the detected gases. Afterwards, a principal component analysis (PCA) and the RFE-LightGBM algorithm were applied to reduce the dimensionality of the feature datasets, and the differences between these two methods were analyzed, respectively. Finally, the differences in the classification accuracies on the three datasets (the original feature dataset, PCA dimensionality reduction dataset, and RFE-LightGBM dimensionality reduction dataset) were discussed. The results showed that the highest classification accuracy of 95% could be obtained by using the RFE-LightGBM algorithm in the classification stage of recyclable containers, compared to the original feature dataset (88.38%) and PCA dimensionality reduction dataset (92.02%).

Machine Learning-Enhanced Flexible Mechanical Sensing

To realize a hyperconnected smart society with high productivity, advances in flexible sensing technology are highly needed. Nowadays, flexible sensing technology has witnessed improvements in both the hardware performances of sensor devices and the data processing capabilities of the device's software. Significant research efforts have been devoted to improving materials, sensing mechanism, and configurations of flexible sensing systems in a quest to fulfill the requirements of future technology. Meanwhile, advanced data analysis methods are being developed to extract useful information from increasingly complicated data collected by a single sensor or network of sensors. Machine learning (ML) as an important branch of artificial intelligence can efficiently handle such complex data, which can be multi-dimensional and multi-faceted, thus providing a powerful tool for easy interpretation of sensing data. In this review, the fundamental working mechanisms and common types of flexible mechanical sensors are firstly presented. Then how ML-assisted data interpretation improves the applications of flexible mechanical sensors and other closely-related sensors in various areas is elaborated, which includes health monitoring, human-machine interfaces, object/surface recognition, pressure prediction, and human posture/motion identification. Finally, the advantages, challenges, and future perspectives associated with the fusion of flexible mechanical sensing technology and ML algorithms are discussed. These will give significant insights to enable the advancement of next-generation artificial flexible mechanical sensing.

Object stiffness recognition and vibratory feedback without ad-hoc sensing on the Hannes prosthesis: A machine learning approach

Introduction

In recent years, hand prostheses achieved relevant improvements in term of both motor and functional recovery. However, the rate of devices abandonment, also due to their poor embodiment, is still high. The embodiment defines the integration of an external object - in this case a prosthetic device - into the body scheme of an individual. One of the limiting factors causing lack of embodiment is the absence of a direct interaction between user and environment. Many studies focused on the extraction of tactile information via custom electronic skin technologies coupled with dedicated haptic feedback, though increasing the complexity of the prosthetic system. Contrary wise, this paper stems from the authors' preliminary works on multi-body prosthetic hand modeling and the identification of possible intrinsic information to assess object stiffness during interaction.

Methods

Based on these initial findings, this work presents the design, implementation and clinical validation of a novel real-time stiffness detection strategy, without ad-hoc sensing, based on a Non-linear Logistic Regression (NLR) classifier. This exploits the minimum grasp information available from an under-sensorized and under-actuated myoelectric prosthetic hand, Hannes. The NLR algorithm takes as input motor-side current, encoder position, and reference position of the hand and provides as output a classification of the grasped object (no-object, rigid object, and soft object). This information is then transmitted to the user via vibratory feedback to close the loop between user control and prosthesis interaction. This implementation was validated through a user study conducted both on able bodied subjects and amputees.

Results

The classifier achieved excellent performance in terms of F1Score (94.93%). Further, the able-bodied subjects and amputees were able to successfully detect the objects' stiffness with a F1Score of 94.08% and 86.41%, respectively, by using our proposed feedback strategy. This strategy allowed amputees to quickly recognize the objects' stiffness (response time of 2.82 s), indicating high intuitiveness, and it was overall appreciated as demonstrated by the questionnaire. Furthermore, an embodiment improvement was also obtained as highlighted by the proprioceptive drift toward the prosthesis (0.7 cm).

Bibliometric Analysis of Artificial Intelligence in Textiles

Generally, comprehensive documents are needed to provide the research community with relevant details of any research direction. This study conducted the first descriptive bibliometric analysis to examine the most influential journals, institutions, and countries in the field of artificial intelligence in textiles. Furthermore, bibliometric mapping analysis was also used to examine diverse research topics of artificial intelligence in textiles. VOSviewer was used to process 996 articles retrieved from Web of Science—Core Collection from 2007 to 2020. The results show that China and the United States have the largest number of publications, while Donghua University and Jiangnan University have the highest output. These three themes have also appeared in textile artificial intelligence publications and played a significant role in the textile structure, textile inspection, and textile clothing production. The authors believe that this research will unfold new research domains for researchers in computer science, electronics, material science, imaging science, and optics and will benefit academic and industrial circles.

Texture Identification and Object Recognition Using a Soft Robotic Hand Innervated Bio-Inspired Proprioception

In this study, we innervated bio-inspired proprioception into a soft hand, facilitating a robust perception of textures and object shapes. The tendon-driven soft finger with three joints, inspired by the human finger, was detailed. With tension sensors embedded in the tendon that simulate the Golgi tendon organ of the human body, 17 types of textures can be identified under uncertain rotation angles and actuator displacements. Four classifiers were used and the highest identification accuracy was 98.3%. A three-fingered soft hand based on the bionic finger was developed. Its basic grasp capability was tested experimentally. The soft hand can distinguish 10 types of objects that vary in shape with top grasp and side grasp, with the highest accuracies of 96.33% and 96.00%, respectively. Additionally, for six objects with close shapes, the soft hand obtained an identification accuracy of 97.69% with a scan-grasp method. This study offers a novel bionic solution for the texture identification and object recognition of soft manipulators.

A Review of Sensors Used on Fabric-Handling Robots

While in most industries, most processes are automated and human workers have either been replaced by robots or work alongside them, fewer changes have occurred in industries that use limp materials, like fabrics, clothes, and garments, than might be expected with today’s technological evolution. Integration of robots in these industries is a relatively demanding and challenging task, mostly because of the natural and mechanical properties of limp materials. In this review, information on sensors that have been used in fabric-handling applications is gathered, analyzed, and organized based on criteria such as their working principle and the task they are designed to support. Categorization and related works are presented in tables and figures so someone who is interested in developing automated fabric-handling applications can easily get useful information and ideas, at least regarding the necessary sensors for the most common handling tasks. Finally, we hope this work will inspire researchers to design new sensor concepts that could promote automation in the industry and boost the robotization of domestic chores involving with flexible materials.

Texture Recognition Based on Perception Data from a Bionic Tactile Sensor

Texture recognition is important for robots to discern the characteristics of the object surface and adjust grasping and manipulation strategies accordingly. It is still challenging to develop texture classification approaches that are accurate and do not require high computational costs. In this work, we adopt a bionic tactile sensor to collect vibration data while sliding against materials of interest. Under a fixed contact pressure and speed, a total of 1000 sets of vibration data from ten different materials were collected. With the tactile perception data, four types of texture recognition algorithms are proposed. Three machine learning algorithms, including support vector machine, random forest, and K-nearest neighbor, are established for texture recognition. The test accuracy of those three methods are 95%, 94%, 94%, respectively. In the detection process of machine learning algorithms, the asamoto and polyester are easy to be confused with each other. A convolutional neural network is established to further increase the test accuracy to 98.5%. The three machine learning models and convolutional neural network demonstrate high accuracy and excellent robustness.

Electrotactile Feedback for the Discrimination of Different Surface Textures Using a Microphone

Most commercial prosthetic hands lack closed-loop feedback, thus, a lot of research has been focusing on implementing sensory feedback systems to provide the user with sensory information during activities of daily living. This study evaluates the possibilities of using a microphone and electrotactile feedback to identify different textures. A condenser microphone was used as a sensor to detect the friction sound generated from the contact between different textures and the microphone. The generated signal was processed to provide a characteristic electrical stimulation presented to the participants. The main goal of the processing was to derive a continuous and intuitive transfer function between the microphone signal and stimulation frequency. Twelve able-bodied volunteers participated in the study, in which they were asked to identify the stroked texture (among four used in this study: Felt, sponge, silicone rubber, and string mesh) using only electrotactile feedback. The experiments were done in three phases: (1) Training, (2) with-feedback, (3) without-feedback. Each texture was stroked 20 times each during all three phases. The results show that the participants were able to differentiate between different textures, with a median accuracy of 85%, by using only electrotactile feedback with the stimulation frequency being the only variable parameter.

Bioluminescent Proteins Prediction with Voting Strategy

Background:

Bioluminescence is a unique and significant phenomenon in nature. Bioluminescence is important for the lifecycle of some organisms and is valuable in biomedical research, including for gene expression analysis and bioluminescence imaging technology. In recent years, researchers have identified a number of methods for predicting bioluminescent proteins (BLPs), which have increased in accuracy, but could be further improved.

Method:

In this study, a new bioluminescent proteins prediction method, based on a voting algorithm, is proposed. Four methods of feature extraction based on the amino acid sequence were used. 314 dimensional features in total were extracted from amino acid composition, physicochemical properties and k-spacer amino acid pair composition. In order to obtain the highest MCC value to establish the optimal prediction model, a voting algorithm was then used to build the model. To create the best performing model, the selection of base classifiers and vote counting rules are discussed.

Results:

The proposed model achieved 93.4% accuracy, 93.4% sensitivity and 91.7% specificity in the test set, which was better than any other method. A previous prediction of bioluminescent proteins in three lineages was also improved using the model building method, resulting in greatly improved accuracy.

Recent Developments in Prosthesis Sensors, Texture Recognition, and Sensory Stimulation for Upper Limb Prostheses

Current developments being made in upper limb prostheses are focused on replacing lost sensory information to the amputees. Providing sensory stimulation from the prosthesis can directly improve control over the prosthetic and provide a sense of body ownership. The focus of this review article is on recent developments while including foundational knowledge for some of the critical concepts in neural prostheses. Reported concepts follow the flow of information from sensors to signal processing, with emphasis on texture recognition, and then to sensory stimulation strategies that reestablish the lost sensory feedback loop. Prosthetic sensors are used to detect the physical environment, converting pressure, force, and position into electrical signals. The electrical signals can then be processed in an effort to identify the surrounding environment using distinctive characteristics such as stiffness and texture. In order for the amputee to use this information in a natural manner, there must be real-time sensory stimulation, perception, and motor control of the prosthesis. Although truly complete sensory replacement has not yet been realized, some basic percepts can be partially restored, allowing progress towards a more realistic prosthesis with natural sensations.

Texture Discrimination with a Soft Biomimetic Finger Using a Flexible Neuromorphic Tactile Sensor Array That Provides Sensory Feedback

The compliant nature of soft fingers allows for safe and dexterous manipulation of objects by humans in an unstructured environment. A soft prosthetic finger design with tactile sensing capabilities for texture discrimination and subsequent sensory stimulation has the potential to create a more natural experience for an amputee. In this work, a pneumatically actuated soft biomimetic finger is integrated with a textile neuromorphic tactile sensor array for a texture discrimination task. The tactile sensor outputs were converted into neuromorphic spike trains, which emulate the firing pattern of biological mechanoreceptors. Spike-based features from each taxel compressed the information and were then used as inputs for the support vector machine classifier to differentiate the textures. Our soft biomimetic finger with neuromorphic encoding was able to achieve an average overall classification accuracy of 99.57% over 16 independent parameters when tested on 13 standardized textured surfaces. The 16 parameters were the combination of 4 angles of flexion of the soft finger and 4 speeds of palpation. To aid in the perception of more natural objects and their manipulation, subjects were provided with transcutaneous electrical nerve stimulation to convey a subset of four textures with varied textural information. Three able-bodied subjects successfully distinguished two or three textures with the applied stimuli. This work paves the way for a more human-like prosthesis through a soft biomimetic finger with texture discrimination capabilities using neuromorphic techniques that provide sensory feedback; furthermore, texture feedback has the potential to enhance user experience when interacting with their surroundings.

Embodied tactile perception and learning

Various living creatures exhibit embodiment intelligence, which is reflected by a collaborative interaction of the brain, body, and environment. The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation. The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning. Tactile information plays a critical role in this physical interaction process. It can be used to ensure safety, stability, and compliance, and can provide unique information that is difficult to capture using other perception modalities. However, due to the limitations of existing sensors and perception and learning methods, the development of robotic tactile research lags significantly behind other sensing modalities, such as vision and hearing, thereby seriously restricting the development of robotic embodiment intelligence. This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence. Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large‐scale tactile array sensing devices, with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.

Electronic skins and machine learning for intelligent soft robots

Soft robots have garnered interest for real-world applications because of their intrinsic safety embedded at the material level. These robots use deformable materials capable of shape and behavioral changes and allow conformable physical contact for manipulation. Yet, with the introduction of soft and stretchable materials to robotic systems comes a myriad of challenges for sensor integration, including multimodal sensing capable of stretching, embedment of high-resolution but large-area sensor arrays, and sensor fusion with an increasing volume of data. This Review explores the emerging confluence of e-skins and machine learning, with a focus on how roboticists can combine recent developments from the two fields to build autonomous, deployable soft robots, integrated with capabilities for informative touch and proprioception to stand up to the challenges of real-world environments.

Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures

We present a framework for the acquisition and parametrization of object material properties. The introduced acquisition device, denoted as Texplorer2, is able to extract surface material properties while a human operator is performing exploratory procedures. Using the Texplorer2, we scanned 184 material classes which we labeled according to biological, chemical, and geological naming conventions. Based on these real material recordings, we introduce a novel set of mathematical features which align with corresponding material properties defined in perceptual studies from related work and classify the materials using common machine learning techniques. Validation results of the proposed multi-modal features lead to an overall classification accuracy of 90.2% ± 1.2% and an F[Formula: see text] score of 0.90 ± 0.01 using the random forest classifier. For the sake of comparison, a deep neural network is trained and tested on images of the material surfaces; it outperforms (90.7% ± 1.0%) the hand-crafted feature-based approach yet leads to more critical misclassifications in terms of the proposed taxonomy.

Using 3D Convolutional Neural Networks for Tactile Object Recognition with Robotic Palpation

In this paper, a novel method of active tactile perception based on 3D neural networks and a high-resolution tactile sensor installed on a robot gripper is presented. A haptic exploratory procedure based on robotic palpation is performed to get pressure images at different grasping forces that provide information not only about the external shape of the object, but also about its internal features. The gripper consists of two underactuated fingers with a tactile sensor array in the thumb. A new representation of tactile information as 3D tactile tensors is described. During a squeeze-and-release process, the pressure images read from the tactile sensor are concatenated forming a tensor that contains information about the variation of pressure matrices along with the grasping forces. These tensors are used to feed a 3D Convolutional Neural Network (3D CNN) called 3D TactNet, which is able to classify the grasped object through active interaction. Results show that 3D CNN performs better, and provide better recognition rates with a lower number of training data.

A new robotic tactile sensor with bio-mimetic structural colour inspired by Morpho butterflies

Since tactile perception and robotic manipulation play important roles in human survival, we propose a new method for developing robotic tactile sensors based on the structural colours of Morpho menelaus (a kind of Morpho butterfly). The first task is to fabricate a flexible bioinspired grating with a similar microstructure to the wings of Morpho menelaus using the transfer technique, onto the surfaces of polydimethylsiloxane (PDMS) films. The second task, depending on the angle of diffracted light, is to integrate the flexible diffraction grating with a polychromatic light source and a CCD camera, and then predict the position and magnitude of the contact force caused by a change in the diffraction pattern. The final task is to set up an experimental calibration platform and a marker point array with an interval of 1 mm for an image processing algorithm and a deep learning method to establish the relationship between the contact point position, and the magnitude of the force and diffraction pattern. The results showed that this tactile sensor has high sensitivity and resolution, with the position of the contact force of 1 mm. This practical application of the UR-5 manipulator verifies the feasibility of the prototype as a tactile sensor. This tactile sensing method may be widely used in robotics by miniaturising the design.

A 3-D Surface Reconstruction with Shadow Processing for Optical Tactile Sensors

An optical tactile sensor technique with 3-dimension (3-D) surface reconstruction is proposed for robotic fingers. The hardware of the tactile sensor consists of a surface deformation sensing layer, an image sensor and four individually controlled flashing light emitting diodes (LEDs). The image sensor records the deformation images when the robotic finger touches an object. For each object, four deformation images are taken with the LEDs providing different illumination directions. Before the 3-D reconstruction, the look-up tables are built to map the intensity distribution to the image gradient data. The possible image shadow will be detected and amended. Then the 3-D depth distribution of the object surface can be reconstructed from the 2-D gradient obtained using the look-up tables. The architecture of the tactile sensor and the proposed signal processing flow have been presented in details. A prototype tactile sensor has been built. Both the simulation and experimental results have validated the effectiveness of the proposed 3-D surface reconstruction method for the optical tactile sensors. The proposed 3-D surface reconstruction method has the unique feature of image shadow detection and compensation, which differentiates itself from those in the literature.

Active Tactile Transfer Learning for Object Discrimination in an Unstructured Environment Using Multimodal Robotic Skin

In this paper, we propose a probabilistic active tactile transfer learning (ATTL) method to enable robotic systems to exploit their prior tactile knowledge while discriminating among objects via their physical properties (surface texture, stiffness, and thermal conductivity). Using the proposed method, the robot autonomously selects and exploits its most relevant prior tactile knowledge to efficiently learn about new unknown objects with a few training samples or even one. The experimental results show that using our proposed method, the robot successfully discriminated among new objects with [Formula: see text] discrimination accuracy using only one training sample (on-shot-tactile-learning). Furthermore, the results demonstrate that our method is robust against transferring irrelevant prior tactile knowledge (negative tactile knowledge transfer).

Enhancing Perception with Tactile Object Recognition in Adaptive Grippers for Human-Robot Interaction

The use of tactile perception can help first response robotic teams in disaster scenarios, where visibility conditions are often reduced due to the presence of dust, mud, or smoke, distinguishing human limbs from other objects with similar shapes. Here, the integration of the tactile sensor in adaptive grippers is evaluated, measuring the performance of an object recognition task based on deep convolutional neural networks (DCNNs) using a flexible sensor mounted in adaptive grippers. A total of 15 classes with 50 tactile images each were trained, including human body parts and common environment objects, in semi-rigid and flexible adaptive grippers based on the fin ray effect. The classifier was compared against the rigid configuration and a support vector machine classifier (SVM). Finally, a two-level output network has been proposed to provide both object-type recognition and human/non-human classification. Sensors in adaptive grippers have a higher number of non-null tactels (up to 37% more), with a lower mean of pressure values (up to 72% less) than when using a rigid sensor, with a softer grip, which is needed in physical human–robot interaction (pHRI). A semi-rigid implementation with 95.13% object recognition rate was chosen, even though the human/non-human classification had better results (98.78%) with a rigid sensor.

A Novel Inverse Solution of Contact Force Based on a Sparse Tactile Sensor Array

High-density tactile sensing has been pursued for humanoid robotic hands to obtain contact force information while the elastomer skin cover is traditionally considered to impair the force discrimination. In this work, we try to utilize the diffusion effect of the elastomer cover to identify an arbitrary contact force load just based on a sparse tactile sensor array. By numerical analysis, we proved the monotonous relation between the Pearson's correlation coefficient and the relative distance of two single-force loads. Then, we meshed the elastomer surface and conducted the calibration load process to establish the calibration database of the sensing outputs. Afterwards, we applied the correlation method to the database and the sensing output of the unknown load to determine its location and intensity. For validation tests of the proposed method, we designed and fabricated a 3 × 3 sparse tactile sensor array with flat elastomer cover and established an automatic three-axis loading system. The validation tests were implemented including 100 random points with force intensity ranging from 0.1 to 1 N. The test results show that the method has good accuracy of detecting force load with the mean location error of 0.46 mm and the mean intensity error of 0.043 N, which meets the basic requirements of tactile sensing. Therefore, it is feasible for the sparse tactile sensor array to realize high-density load detection.

Recent progress on tactile object recognition

Conventional visual perception technology is subject to many restrictions, such as illumination, background clutter, and occlusion. Many intrinsic properties of objects, like stiffness, hardness, and internal state, cannot be effectively perceived by visual sensors. For robots, tactile perception is a key approach to obtain environmental and object information. Different from vision sensors, tactile sensors can directly measure some physical properties of objects and environment. At the same time, humans also utilize touch sensory receptors as an important means to perceive and interact with the environment. In this article, we present a detailed discussion on tactile object recognition problem. We divide the current studies on the tactile object recognition into three subcategories and detailed analysis has been put forward on them. In addition, we also discuss some advanced topics such as visual–tactile fusion, exploratory procedure, and data sets.

Object recognition combining vision and touch

This paper explores ways of combining vision and touch for the purpose of object recognition. In particular, it focuses on scenarios when there are few tactile training samples (as these are usually costly to obtain) and when vision is artificially impaired. Whilst machine vision is a widely studied field, and machine touch has received some attention recently, the fusion of both modalities remains a relatively unexplored area. It has been suggested that, in the human brain, there exist shared multi-sensorial representations of objects. This provides robustness when one or more senses are absent or unreliable. Modern robotics systems can benefit from multi-sensorial input, in particular in contexts where one or more of the sensors perform poorly. In this paper, a recently proposed tactile recognition model was extended by integrating a simple vision system in three different ways: vector concatenation (vision feature vector and tactile feature vector), object label posterior averaging and object label posterior product. A comparison is drawn in terms of overall accuracy of recognition and in terms of how quickly (number of training samples) learning occurs. The conclusions reached are: (1) the most accurate system is “posterior product”, (2) multi-modal recognition has higher accuracy to either modality alone if all visual and tactile training data are pooled together, and (3) in the case of visual impairment, multi-modal recognition “learns faster”, i.e. requires fewer training samples to achieve the same accuracy as either other modality.

Multimodal Feature-Based Surface Material Classification

When a tool is tapped on or dragged over an object surface, vibrations are induced in the tool, which can be captured using acceleration sensors. The tool-surface interaction additionally creates audible sound waves, which can be recorded using microphones. Features extracted from camera images provide additional information about the surfaces. We present an approach for tool-mediated surface classification that combines these signals and demonstrate that the proposed method is robust against variable scan-time parameters. We examine freehand recordings of 69 textured surfaces recorded by different users and propose a classification system that uses perception-related features, such as hardness, roughness, and friction; selected features adapted from speech recognition, such as modified cepstral coefficients applied to our acceleration signals; and surface texture-related image features. We focus on mitigating the effect of variable contact force and exploration velocity conditions on these features as a prerequisite for a robust machine-learning-based approach for surface classification. The proposed system works without explicit scan force and velocity measurements. Experimental results show that our proposed approach allows for successful classification of textured surfaces under variable freehand movement conditions, exerted by different human operators. The proposed subset of six features, selected from the described sound, image, friction force, and acceleration features, leads to a classification accuracy of 74 percent in our experiments when combined with a Naive Bayes classifier.

Texture- and deformability-based surface recognition by tactile image analysis

Deformability and texture are two unique object characteristics which are essential for appropriate surface recognition by tactile exploration. Tactile sensation is required to be incorporated in artificial arms for rehabilitative and other human-computer interface applications to achieve efficient and human-like manoeuvring. To accomplish the same, surface recognition by tactile data analysis is one of the prerequisites. The aim of this work is to develop effective technique for identification of various surfaces based on deformability and texture by analysing tactile images which are obtained during dynamic exploration of the item by artificial arms whose gripper is fitted with tactile sensors. Tactile data have been acquired, while human beings as well as a robot hand fitted with tactile sensors explored the objects. The tactile images are pre-processed, and relevant features are extracted from the tactile images. These features are provided as input to the variants of support vector machine (SVM), linear discriminant analysis and k-nearest neighbour (kNN) for classification. Based on deformability, six household surfaces are recognized from their corresponding tactile images. Moreover, based on texture five surfaces of daily use are classified. The method adopted in the former two cases has also been applied for deformability- and texture-based recognition of four biomembranes, i.e. membranes prepared from biomaterials which can be used for various applications such as drug delivery and implants. Linear SVM performed best for recognizing surface deformability with an accuracy of 83 % in 82.60 ms, whereas kNN classifier recognizes surfaces of daily use having different textures with an accuracy of 89 % in 54.25 ms and SVM with radial basis function kernel recognizes biomembranes with an accuracy of 78 % in 53.35 ms. The classifiers are observed to generalize well on the unseen test datasets with very high performance to achieve efficient material recognition based on its deformability and texture.

A biomimetic tactile sensing system based on polyvinylidene fluoride film

Polyvinylidene fluoride (PVDF) film has been widely investigated as a sensing material due to its outstanding properties such as biocompatibility, high thermal stability, good chemical resistance, high piezo-, pyro- and ferro-electric properties. This paper reports on the design, test, and analysis of a biomimetic tactile sensor based on PVDF film. This sensor consists of a PVDF film with aluminum electrodes, a pair of insulating layers, and a "handprint" friction layer with a copper foil. It is designed for easy fabrication and high reliability in outputting signals. In bionics, the fingerprint of the glabrous skin plays an important role during object handling. Therefore, in order to enhance friction and to provide better manipulation, the ridges of the fingertips were introduced into the design of the proposed tactile sensor. And, a basic experimental study on the selection of the high sensitivity fingerprint type for the biomimetic sensor was performed. In addition, we proposed a texture distinguish experiment to verify the sensor sensitivity. The experiment's results show that the novel biomimetic sensor is effective in discriminating object surface characteristics. Furthermore, an efficient visual application program (LabVIEW) and a quantitative evaluation method were proposed for the verification of the biomimetic sensor. The proposed tactile sensor shows great potential for contact force and slip measurements.

Slip classification for dynamic tactile array sensors

The manipulation of objects held in a robotic hand or gripper is accompanied by events such as making and breaking contact and slippage, between the fingertips and the grasped object and between the grasped object and external surfaces. Humans can distinguish among such events, in part, because they excite the various mechanoreceptors in the hands differently. As part of an effort to provide robots with a similar capability, we propose two features that can be extracted from dynamic tactile array data and used to discriminate between hand/object and object/world slips. Both features rely on examining how slippage affects an array of dynamic tactile sensors compared with the way it affects individual elements of the array. In comparison with approaches that require extensive training with particular combinations of objects and skin, the features work for a wide range of frequencies and grasp conditions. The performance and generalizability of the features are verified with testing on three different kinds of sensors and for a range of object textures, grasp forces and slip conditions. Both features demonstrate greater than 85% accuracy in identifying the location of slip.

Computational intelligence techniques for tactile sensing systems

Tactile sensing helps robots interact with humans and objects effectively in real environments. Piezoelectric polymer sensors provide the functional building blocks of the robotic electronic skin, mainly thanks to their flexibility and suitability for detecting dynamic contact events and for recognizing the touch modality. The paper focuses on the ability of tactile sensing systems to support the challenging recognition of certain qualities/modalities of touch. The research applies novel computational intelligence techniques and a tensor-based approach for the classification of touch modalities; its main results consist in providing a procedure to enhance system generalization ability and architecture for multi-class recognition applications. An experimental campaign involving 70 participants using three different modalities in touching the upper surface of the sensor array was conducted, and confirmed the validity of the approach.

A finger-shaped tactile sensor for fabric surfaces evaluation by 2-dimensional active sliding touch

Sliding tactile perception is a basic function for human beings to determine the mechanical properties of object surfaces and recognize materials. Imitating this process, this paper proposes a novel finger-shaped tactile sensor based on a thin piezoelectric polyvinylidene fluoride (PVDF) film for surface texture measurement. A parallelogram mechanism is designed to ensure that the sensor applies a constant contact force perpendicular to the object surface, and a 2-dimensional movable mechanical structure is utilized to generate the relative motion at a certain speed between the sensor and the object surface. By controlling the 2-dimensional motion of the finger-shaped sensor along the object surface, small height/depth variation of surface texture changes the output charge of PVDF film then surface texture can be measured. In this paper, the finger-shaped tactile sensor is used to evaluate and classify five different kinds of linen. Fast Fourier Transformation (FFT) is utilized to get original attribute data of surface in the frequency domain, and principal component analysis (PCA) is used to compress the attribute data and extract feature information. Finally, low dimensional features are classified by Support Vector Machine (SVM). The experimental results show that this finger-shaped tactile sensor is effective and high accurate for discriminating the five textures.

Bayesian action&perception: representing the world in the brain

Theories of perception seek to explain how sensory data are processed to identify previously experienced objects, but they usually do not consider the decisions and effort that goes into acquiring the sensory data. Identification of objects according to their tactile properties requires active exploratory movements. The sensory data thereby obtained depend on the details of those movements, which human subjects change rapidly and seemingly capriciously. Bayesian Exploration is an algorithm that uses prior experience to decide which next exploratory movement should provide the most useful data to disambiguate the most likely possibilities. In previous studies, a simple robot equipped with a biomimetic tactile sensor and operated according to Bayesian Exploration performed in a manner similar to and actually better than humans on a texture identification task. Expanding on this, "Bayesian Action&Perception" refers to the construction and querying of an associative memory of previously experienced entities containing both sensory data and the motor programs that elicited them. We hypothesize that this memory can be queried (i) to identify useful next exploratory movements during identification of an unknown entity ("action for perception") or (ii) to characterize whether an unknown entity is fit for purpose ("perception for action") or (iii) to recall what actions might be feasible for a known entity (Gibsonian affordance). The biomimetic design of this mechatronic system may provide insights into the neuronal basis of biological action and perception.