Ion trap and release dynamics enables nonintrusive tactile augmentation in monolithic sensory neuron

An iontronic-based artificial tactile nerve is a promising technology for emulating the tactile recognition and learning of human skin with low power consumption. However, its weak tactile memory and complex integration structure remain challenging. We present an ion trap and release dynamics (iTRD)-driven, neuro-inspired monolithic artificial tactile neuron (NeuroMAT) that can achieve tactile perception and memory consolidation in a single device. Through the tactile-driven release of ions initially trapped within iTRD-iongel, NeuroMAT only generates nonintrusive synaptic memory signals when mechanical stress is applied under voltage stimulation. The induced tactile memory is augmented by auxiliary voltage pulses independent of tactile sensing signals. We integrate NeuroMAT with an anthropomorphic robotic hand system to imitate memory-based human motion; the robust tactile memory of NeuroMAT enables the hand to consistently perform reliable gripping motion.

Design and Evaluation of Capacitive Smart Transducer for a Forestry Crane Gripper

Stable grasps are essential for robots handling objects. This is especially true for "robotized" large industrial machines as heavy and bulky objects that are unintentionally dropped by the machine can lead to substantial damages and pose a significant safety risk. Consequently, adding a proximity and tactile sensing to such large industrial machinery can help to mitigate this problem. In this paper, we present a sensing system for proximity/tactile sensing in gripper claws of a forestry crane. In order to avoid difficulties with respect to the installation of cables (in particular in retrofitting of existing machinery), the sensors are truly wireless and can be powered using energy harvesting, leading to autarkic, i.e., self-contained, sensors. The sensing elements are connected to a measurement system which transmits the measurement data to the crane automation computer via Bluetooth low energy (BLE) compliant to IEEE 1451.0 (TEDs) specification for eased logical system integration. We demonstrate that the sensor system can be fully integrated in the grasper and that it can withstand the challenging environmental conditions. We present experimental evaluation of detection in various grasping scenarios such as grasping at an angle, corner grasping, improper closure of the gripper and proper grasp for logs of three different sizes. Results indicate the ability to detect and differentiate between good and poor grasping configurations.

Flexible Electronic Skin for Monitoring of Grasping State During Robotic Manipulation

Electronic skin for robotic tactile sensing has been studied extensively over the past years, yet practical applications of electronic skin for the grasping state monitoring during robotic manipulation are still limited. In this study, we present the fabrication and implementation of electronic skin sensor arrays for the detection of unstable grasping. The piezoresistive sensor arrays have the advantages of facile fabrication, fast response, and high reliability. With the tactile data from the sensor array, we propose two quantitative indicators, correlation coefficient and wavelet coefficient, to identify grasping with variable forces and slippage. Those two indicators reflect both time and frequency domain characteristics in the contact forces from the sensor array and can be obtained without large amount of calculation. We demonstrate the utility of this method under various conditions, the results indicate grasping with variable forces, and slippage can be distinguished by this method. The flexible sensor arrays are adopted for tactile sensing on a bionic hand, and the effectiveness of this method in detecting various grasping states has been verified. The electronic skin sensor array and the grasping state monitoring method are promising for applications in robotic dexterous manipulation.

Neuro-inspired electronic skin for robots

Touch is a complex sensing modality owing to large number of receptors (mechano, thermal, pain) nonuniformly embedded in the soft skin all over the body. These receptors can gather and encode the large tactile data, allowing us to feel and perceive the real world. This efficient somatosensation far outperforms the touch-sensing capability of most of the state-of-the-art robots today and suggests the need for neural-like hardware for electronic skin (e-skin). This could be attained through either innovative schemes for developing distributed electronics or repurposing the neuromorphic circuits developed for other sensory modalities such as vision and audio. This Review highlights the hardware implementations of various computational building blocks for e-skin and the ways they can be integrated to potentially realize human skin-like or peripheral nervous system-like functionalities. The neural-like sensing and data processing are discussed along with various algorithms and hardware architectures. The integration of ultrathin neuromorphic chips for local computation and the printed electronics on soft substrate used for the development of e-skin over large areas are expected to advance robotic interaction as well as open new avenues for research in medical instrumentation, wearables, electronics, and neuroprosthetics.

Identifying the Strength Level of Objects' Tactile Attributes Using a Multi-Scale Convolutional Neural Network

In order to solve the problem in which most currently existing research focuses on the binary tactile attributes of objects and ignores identifying the strength level of tactile attributes, this paper establishes a tactile data set of the strength level of objects' elasticity and hardness attributes to make up for the lack of relevant data, and proposes a multi-scale convolutional neural network to identify the strength level of object attributes. The network recognizes the different attributes and identifies differences in the strength level of the same object attributes by fusing the original features, i.e., the single-channel features and multi-channel features of the data. A variety of evaluation methods were used for comparison with multiple models in terms of strength levels of elasticity and hardness. The results show that our network has a more significant effect in accuracy. In the prediction results of the positive examples in the predicted value, the true value has a higher proportion of positive examples, that is, the precision is better. The prediction effect for the positive examples in the true value is better, that is, the recall is better. Finally, the recognition rate for all classes is higher in terms of f1_score. For the overall sample, the prediction of the multi-scale convolutional neural network has a higher recognition rate and the network's ability to recognize each strength level is more stable.

Fabric Classification Using a Finger-Shaped Tactile Sensor via Robotic Sliding

Tactile sensing endows the robots to perceive certain physical properties of the object in contact. Robots with tactile perception can classify textures by touching. Interestingly, textures of fine micro-geometry beyond the nominal resolution of the tactile sensors can also be identified through exploratory robotic movements like sliding. To study the problem of fine texture classification, we design a robotic sliding experiment using a finger-shaped multi-channel capacitive tactile sensor. A feature extraction process is presented to encode the acquired tactile signals (in the form of time series) into a low dimensional (≤7D) feature vector. The feature vector captures the frequency signature of a fabric texture such that fabrics can be classified directly. The experiment includes multiple combinations of sliding parameters, i.e., speed and pressure, to investigate the correlation between sliding parameters and the generated feature space. Results show that changing the contact pressure can greatly affect the significance of the extracted feature vectors. Instead, variation of sliding speed shows no apparent effects. In summary, this paper presents a study of texture classification on fabrics by training a simple k-NN classifier, using only one modality and one type of exploratory motion (sliding). The classification accuracy can reach up to 96%. The analysis of the feature space also implies a potential parametric representation of textures for tactile perception, which could be used for the adaption of motion to reach better classification performance.

Hierarchical Tactile Sensation Integration from Prosthetic Fingertips Enables Multi-Texture Surface Recognition

Multifunctional flexible tactile sensors could be useful to improve the control of prosthetic hands. To that end, highly stretchable liquid metal tactile sensors (LMS) were designed, manufactured via photolithography, and incorporated into the fingertips of a prosthetic hand. Three novel contributions were made with the LMS. First, individual fingertips were used to distinguish between different speeds of sliding contact with different surfaces. Second, differences in surface textures were reliably detected during sliding contact. Third, the capacity for hierarchical tactile sensor integration was demonstrated by using four LMS signals simultaneously to distinguish between ten complex multi-textured surfaces. Four different machine learning algorithms were compared for their successful classification capabilities: K-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), and neural network (NN). The time-frequency features of the LMSs were extracted to train and test the machine learning algorithms. The NN generally performed the best at the speed and texture detection with a single finger and had a 99.2 ± 0.8% accuracy to distinguish between ten different multi-textured surfaces using four LMSs from four fingers simultaneously. The capability for hierarchical multi-finger tactile sensation integration could be useful to provide a higher level of intelligence for artificial hands.

Trends of Human-Robot Collaboration in Industry Contexts: Handover, Learning, and Metrics

Repetitive industrial tasks can be easily performed by traditional robotic systems. However, many other works require cognitive knowledge that only humans can provide. Human-Robot Collaboration (HRC) emerges as an ideal concept of co-working between a human operator and a robot, representing one of the most significant subjects for human-life improvement.The ultimate goal is to achieve physical interaction, where handing over an object plays a crucial role for an effective task accomplishment. Considerable research work had been developed in this particular field in recent years, where several solutions were already proposed. Nonetheless, some particular issues regarding Human-Robot Collaboration still hold an open path to truly important research improvements. This paper provides a literature overview, defining the HRC concept, enumerating the distinct human-robot communication channels, and discussing the physical interaction that this collaboration entails. Moreover, future challenges for a natural and intuitive collaboration are exposed: the machine must behave like a human especially in the pre-grasping/grasping phases and the handover procedure should be fluent and bidirectional, for an articulated function development. These are the focus of the near future investigation aiming to shed light on the complex combination of predictive and reactive control mechanisms promoting coordination and understanding. Following recent progress in artificial intelligence, learning exploration stand as the key element to allow the generation of coordinated actions and their shaping by experience.

Fingerpad-Inspired Multimodal Electronic Skin for Material Discrimination and Texture Recognition

Human skin plays a critical role in a person communicating with his or her environment through diverse activities such as touching or deforming an object. Various electronic skin (E-skin) devices have been developed that show functional or geometrical superiority to human skin. However, research into stretchable E-skin that can simultaneously distinguish materials and textures has not been established yet. Here, the first approach to achieving a stretchable multimodal device is reported, that operates on the basis of various electrical properties of piezoelectricity, triboelectricity, and piezoresistivity and that exceeds the capabilities of human tactile perception. The prepared E-skin is composed of a wrinkle-patterned silicon elastomer, hybrid nanomaterials of silver nanowires and zinc oxide nanowires, and a thin elastomeric dielectric layer covering the hybrid nanomaterials, where the dielectric layer exhibits high surface roughness mimicking human fingerprints. This versatile device can identify and distinguish not only mechanical stress from a single stimulus such as pressure, tensile strain, or vibration but also that from a combination of multiple stimuli. With simultaneous sensing and analysis of the integrated stimuli, the approach enables material discrimination and texture recognition for a biomimetic prosthesis when the multifunctional E-skin is applied to a robotic hand.

Multi-IMF Sample Entropy Features with Machine Learning for Surface Texture Recognition Based on Robot Tactile Perception

Discrimination of surface textures using tactile sensors has attracted increasing attention. Intelligent robotics with the ability to recognize and discriminate the surface textures of grasped objects are crucial. In this paper, a novel method for surface texture classification based on tactile signals is proposed. For the proposed method, first, the tactile signals of each channel (X, Y, Z, and S) are decomposed based on empirical mode decomposition (EMD). Then, the intrinsic mode functions (IMFs) are obtained. Second, based on the multiple IMFs, the sample entropy is calculated for each IMF. Therefore, the multi-IMF sample entropy (MISE) features are obtained. Last but not least, based on the two public datasets, a variety of machine learning algorithms are used to recognize different textures. The results show that the SVM classification method, with the proposed MISE features, achieves the highest classification accuracy. Undeniably, the MISE features with the SVM method, proposed in this paper, provide a novel idea for the recognition of surface texture based on tactile perception.

Physical Human-Robot Collaboration: Robotic Systems, Learning Methods, Collaborative Strategies, Sensors, and Actuators

This article presents a state-of-the-art survey on the robotic systems, sensors, actuators, and collaborative strategies for physical human-robot collaboration (pHRC). This article starts with an overview of some robotic systems with cutting-edge technologies (sensors and actuators) suitable for pHRC operations and the intelligent assist devices employed in pHRC. Sensors being among the essential components to establish communication between a human and a robotic system are surveyed. The sensor supplies the signal needed to drive the robotic actuators. The survey reveals that the design of new generation collaborative robots and other intelligent robotic systems has paved the way for sophisticated learning techniques and control algorithms to be deployed in pHRC. Furthermore, it revealed the relevant components needed to be considered for effective pHRC to be accomplished. Finally, a discussion of the major advances is made, some research directions, and future challenges are presented.

Automatic Fracture Characterization Using Tactile and Proximity Optical Sensing

This paper demonstrates how tactile and proximity sensing can be used to perform automatic mechanical fractures detection (surface cracks). For this purpose, a custom-designed integrated tactile and proximity sensor has been implemented. With the help of fiber optics, the sensor measures the deformation of its body, when interacting with the physical environment, and the distance to the environment's objects. This sensor slides across different surfaces and records data which are then analyzed to detect and classify fractures and other mechanical features. The proposed method implements machine learning techniques (handcrafted features, and state of the art classification algorithms). An average crack detection accuracy of ~94% and width classification accuracy of ~80% is achieved. Kruskal-Wallis results (p < 0.001) indicate statistically significant differences among results obtained when analysing only integrated deformation measurements, only proximity measurements and both deformation and proximity data. A real-time classification method has been implemented for online classification of explored surfaces. In contrast to previous techniques, which mainly rely on visual modality, the proposed approach based on optical fibers might be more suitable for operation in extreme environments (such as nuclear facilities) where radiation may damage electronic components of commonly employed sensing devices, such as standard force sensors based on strain gauges and video cameras.

Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures

We present a framework for the acquisition and parametrization of object material properties. The introduced acquisition device, denoted as Texplorer2, is able to extract surface material properties while a human operator is performing exploratory procedures. Using the Texplorer2, we scanned 184 material classes which we labeled according to biological, chemical, and geological naming conventions. Based on these real material recordings, we introduce a novel set of mathematical features which align with corresponding material properties defined in perceptual studies from related work and classify the materials using common machine learning techniques. Validation results of the proposed multi-modal features lead to an overall classification accuracy of 90.2% ± 1.2% and an F[Formula: see text] score of 0.90 ± 0.01 using the random forest classifier. For the sake of comparison, a deep neural network is trained and tested on images of the material surfaces; it outperforms (90.7% ± 1.0%) the hand-crafted feature-based approach yet leads to more critical misclassifications in terms of the proposed taxonomy.

Using 3D Convolutional Neural Networks for Tactile Object Recognition with Robotic Palpation

In this paper, a novel method of active tactile perception based on 3D neural networks and a high-resolution tactile sensor installed on a robot gripper is presented. A haptic exploratory procedure based on robotic palpation is performed to get pressure images at different grasping forces that provide information not only about the external shape of the object, but also about its internal features. The gripper consists of two underactuated fingers with a tactile sensor array in the thumb. A new representation of tactile information as 3D tactile tensors is described. During a squeeze-and-release process, the pressure images read from the tactile sensor are concatenated forming a tensor that contains information about the variation of pressure matrices along with the grasping forces. These tensors are used to feed a 3D Convolutional Neural Network (3D CNN) called 3D TactNet, which is able to classify the grasped object through active interaction. Results show that 3D CNN performs better, and provide better recognition rates with a lower number of training data.

Proposal of the Tactile Glove Device

This project aims to develop a tactile glove device and a virtual environment inserted in the context of tactile internet. The tactile glove allows a human operator to interact remotely with objects from a 3D environment through tactile feedback or tactile sensation. In other words, the human operator is able to feel the contour and texture from virtual objects. Applications such as remote diagnostics, games, remote analysis of materials, and others in which objects could be virtualized can be significantly improved using this kind of device. These gloves have been an essential device in all research on the internet next generation called "Tactile Internet", in which this project is inserted. Unlike the works presented in the literature, the novelty of this work is related to architecture, and tactile devices developed. They are within the 10 ms round trip latency limits required in a tactile internet environment. Details of hardware and software designs of a tactile glove, as well as the virtual environment, are described. Results and comparative analysis about round trip latency time in the tactile internet environment is developed.

Biomimetic Tactile Sensors with Bilayer Fingerprint Ridges Demonstrating Texture Recognition

In this article, we report on a biomimetic tactile sensor that has a surface kinetic interface (SKIN) that imitates human epidermal fingerprint ridges and the epidermis. The SKIN is composed of a bilayer polymer structure with different elastic moduli. We improved the tactile sensitivity of the SKIN by using a hard epidermal fingerprint ridge and a soft epidermal board. We also evaluated the effectiveness of the SKIN layer in shear transfer characteristics while varying the elasticity and geometrical factors of the epidermal fingerprint ridges and the epidermal board. The biomimetic tactile sensor with the SKIN layer showed a detection capability for surface structures under 100 μm with only 20-μm height differences. Our sensor could distinguish various textures that can be easily accessed in everyday life, demonstrating that the sensor may be used for texture recognition in future artificial and robotic fingers.

Memristive Circuit Design of Emotional Generation and Evolution Based on Skin-Like Sensory Processor

Sensory processor in human skin is used for processing and transmitting sensations to the brain, which leads to body actions and emotional responses. In this paper, a memristive circuit of emotional generation and evolution based on skin-like sensory processor is proposed. The circuit includes: first, memristive skin-like sensory processor module; second, emotional generation and evolution module; and third, emotional expression module. The first module consists of four single-memristor skin-like sensory processors, which correspond to process sensations of pain, cold, warm, and tactile. It will automatically return to its initial state if sensory signals disappear. But if sensory signals are much strong, it will not automatically return to initial state unless applied "restoring signal" just like a surgical operation. The second module realizes a conversion mechanism from sensations to emotions using memristor as emotional synapse. Given signals from skin-like sensory processor, the memristance will decrease, which means the extent of emotion will increase, such as more happy. This is the emotional generation. The extent of emotion will be changed if the same sensation is applied to skin-like sensory processor repeatedly, which is the emotional evolution. The third module can show the generated emotions visually. The simulation results in PSPICE show that the proposed circuit can generate and evolve emotions like human beings after processing sensory signals from skin. The proposed circuit can be applied in a perceptual robot platform to realize the conversion from sensations to emotions, enabling the robot to have the ability to sense and process information.

Highly Sensitive Tactile Shear Sensor Using Spatially Digitized Contact Electrodes

In this article, we report on a highly sensitive tactile shear sensor that was able to detect minute levels of shear and surface slip. The sensor consists of a suspended elastomer diaphragm with a top ridge structure, a graphene layer underneath, and a bottom substrate with multiple spatially digitized contact electrodes. When shear is applied to the top ridge structure, it creates torque and deflects the elastomer downwards. Then, the graphene electrode makes contact with the bottom spatially digitized electrodes completing a circuit producing output currents depending on the number of electrodes making contact. The tactile shear sensor was able to detect shear forces as small as 6 μN, detect shear direction, and also distinguish surface friction and roughness differences of shearing objects. We also succeeded in detecting the contact slip motion of a single thread demonstrating possible applications in future robotic fingers and remote surgical tools.

Haptic Glove and Platform with Gestural Control For Neuromorphic Tactile Sensory Feedback In Medical Telepresence †

Advancements in the study of the human sense of touch are fueling the field of haptics. This is paving the way for augmenting sensory perception during object palpation in tele-surgery and reproducing the sensed information through tactile feedback. Here, we present a novel tele-palpation apparatus that enables the user to detect nodules with various distinct stiffness buried in an ad-hoc polymeric phantom. The contact force measured by the platform was encoded using a neuromorphic model and reproduced on the index fingertip of a remote user through a haptic glove embedding a piezoelectric disk. We assessed the effectiveness of this feedback in allowing nodule identification under two experimental conditions of real-time telepresence: In Line of Sight (ILS), where the platform was placed in the visible range of a user; and the more demanding Not In Line of Sight (NILS), with the platform and the user being 50 km apart. We found that the entailed percentage of identification was higher for stiffer inclusions with respect to the softer ones (average of 74% within the duration of the task), in both telepresence conditions evaluated. These promising results call for further exploration of tactile augmentation technology for telepresence in medical interventions.

A 3-D Surface Reconstruction with Shadow Processing for Optical Tactile Sensors

An optical tactile sensor technique with 3-dimension (3-D) surface reconstruction is proposed for robotic fingers. The hardware of the tactile sensor consists of a surface deformation sensing layer, an image sensor and four individually controlled flashing light emitting diodes (LEDs). The image sensor records the deformation images when the robotic finger touches an object. For each object, four deformation images are taken with the LEDs providing different illumination directions. Before the 3-D reconstruction, the look-up tables are built to map the intensity distribution to the image gradient data. The possible image shadow will be detected and amended. Then the 3-D depth distribution of the object surface can be reconstructed from the 2-D gradient obtained using the look-up tables. The architecture of the tactile sensor and the proposed signal processing flow have been presented in details. A prototype tactile sensor has been built. Both the simulation and experimental results have validated the effectiveness of the proposed 3-D surface reconstruction method for the optical tactile sensors. The proposed 3-D surface reconstruction method has the unique feature of image shadow detection and compensation, which differentiates itself from those in the literature.