Wearable Sensor-Based Sign Language Recognition: A Comprehensive Review

Triboelectric Bending Sensors for AI-Enabled Sign Language Recognition

Human-machine interfaces and wearable electronics, as fundamentals to achieve human-machine interactions, are becoming increasingly essential in the era of the Internet of Things. However, contemporary wearable sensors based on resistive and capacitive mechanisms demand an external power, impeding them from extensive and diverse deployment. Herein, a smart wearable system is developed encompassing five arch-structured self-powered triboelectric sensors, a five-channel data acquisition unit to collect finger bending signals, and an artificial intelligence (AI) methodology, specifically a long short-term memory (LSTM) network, to recognize signal patterns. A slider-crank mechanism that precisely controls the bending angle is designed to quantitively assess the sensor's performance. Thirty signal patterns of sign language of each letter are collected and analyzed after the environment noise and cross-talks among different channels are reduced and removed, respectively, by leveraging low pass filters. Two LSTM models are trained using different training sets, and four indexes are introduced to evaluate their performance, achieving a recognition accuracy of 96.15%. This work demonstrates a novel integration of triboelectric sensors with AI for sign language recognition, paving a new application avenue of triboelectric sensors in wearable electronics.

Novel Wearable System to Recognize Sign Language in Real Time

The aim of this study is to develop a practical software solution for real-time recognition of sign language words using two arms. This will facilitate communication between hearing-impaired individuals and those who can hear. We are aware of several sign language recognition systems developed using different technologies, including cameras, armbands, and gloves. However, the system we propose in this study stands out for its practicality, utilizing surface electromyography (muscle activity) and inertial measurement unit (motion dynamics) data from both arms. We address the drawbacks of other methods, such as high costs, low accuracy due to ambient light and obstacles, and complex hardware requirements, which have limited their practical application. Our software can run on different operating systems using digital signal processing and machine learning methods specific to this study. For the test, we created a dataset of 80 words based on their frequency of use in daily life and performed a thorough feature extraction process. We tested the recognition performance using various classifiers and parameters and compared the results. The random forest algorithm emerged as the most successful, achieving a remarkable 99.875% accuracy, while the naïve Bayes algorithm had the lowest success rate with 87.625% accuracy. The new system promises to significantly improve communication for people with hearing disabilities and ensures seamless integration into daily life without compromising user comfort or lifestyle quality.

EchoGest: Soft Ultrasonic Waveguides Based Sensing Skin for Subject-Independent Hand Gesture Recognition

Gesture recognition is crucial for enhancing human-computer interaction and is particularly pivotal in rehabilitation contexts, aiding individuals recovering from physical impairments and significantly improving their mobility and interactive capabilities. However, current wearable hand gesture recognition approaches are often limited in detection performance, wearability, and generalization. We thus introduce EchoGest, a novel hand gesture recognition system based on soft, stretchable, transparent artificial skin with integrated ultrasonic waveguides. Our presented system is the first to use soft ultrasonic waveguides for hand gesture recognition. EcoflexTM 00-31 and EcoflexTM 00-45 Near ClearTM silicone elastomers were employed to fabricate the artificial skin and ultrasonic waveguides, while 0.1 mm diameter silver-plated copper wires connected the transducers in the waveguides to the electrical system. The wires are enclosed within an additional elastomer layer, achieving a sensing skin with a total thickness of around 500 μ m. Ten participants wore the EchoGest system and performed static hand gestures from two gesture sets: 8 daily life gestures and 10 American Sign Language (ASL) digits 0-9. Leave-One-Subject-Out Cross-Validation analysis demonstrated accuracies of 91.13% for daily life gestures and 88.5% for ASL gestures. The EchoGest system has significant potential in rehabilitation, particularly for tracking and evaluating hand mobility, which could substantially reduce the workload of therapists in both clinical and home-based settings. Integrating this technology could revolutionize hand gesture recognition applications, from real-time sign language translation to innovative rehabilitation techniques.

Design of a Wearable Real-Time Hand Motion Tracking System Using an Array of Soft Polymer Acoustic Waveguides

Robust hand motion tracking holds promise for improved human-machine interaction in diverse fields, including virtual reality, and automated sign language translation. However, current wearable hand motion tracking approaches are typically limited in detection performance, wearability, and durability. This article presents a hand motion tracking system using multiple soft polymer acoustic waveguides (SPAWs). The innovative use of SPAWs as strain sensors offers several advantages that address the limitations. SPAWs are easily manufactured by casting a soft polymer shaped as a soft acoustic waveguide and containing a commercially available small ceramic piezoelectric transducer. When used as strain sensors, SPAWs demonstrate high stretchability (up to 100%), high linearity (R2 > 0.996 in all quasi-static, dynamic, and durability tensile tests), negligible hysteresis (<0.7410% under strain of up to 100%), excellent repeatability, and outstanding durability (up to 100,000 cycles). SPAWs also show high accuracy for continuous finger angle estimation (average root-mean-square errors [RMSE] <2.00°) at various flexion-extension speeds. Finally, a hand-tracking system is designed based on a SPAW array. An example application is developed to demonstrate the performance of SPAWs in real-time hand motion tracking in a three-dimensional (3D) virtual environment. To our knowledge, the system detailed in this article is the first to use soft acoustic waveguides to capture human motion. This work is part of an ongoing effort to develop soft sensors using both time and frequency domains, with the goal of extracting decoupled signals from simple sensing structures. As such, it represents a novel and promising path toward soft, simple, and wearable multimodal sensors.

A Low-Complexity Hand Gesture Recognition Framework via Dual mmWave FMCW Radar System

In this paper, we propose a novel low-complexity hand gesture recognition framework via a multiple Frequency Modulation Continuous Wave (FMCW) radar-based sensing system. In this considered system, two radars are deployed distributively to acquire motion vectors from different observation perspectives. We first independently extract reflection points of the interested target from different radars by applying the proposed neighboring reflection points detection method after processing the traditional 2-dimensional Fast Fourier Transform (2D-FFT). The obtained sufficient corresponding information of detected reflection points, e.g., distances, velocities, and angle information, can be exploited to synthesize motion velocity vectors to achieve a high signal-to-noise ratio (SNR) performance, which does not require knowledge of the relative position of the two radars. Furthermore, we utilize a long short-term memory (LSTM) network as well as the synthesized motion velocity vectors to classify different gestures, which can achieve a significantly high accuracy of gesture recognition with a 1600-sample data set, e.g., 98.0%. The experimental results also illustrate the robustness of the proposed gesture recognition systems, e.g., changing the environment background and adding new gesture performers.

[Deaf people and sign language: an enigma in dentistry]

Sign language is the main means of communication for deaf people; consists of the combination of manual, body and facial movements with specific meanings. Among the main barriers faced by this community are the lack of communication, shortage of health services, added to the inexperience and lack of knowledge of sign language by the health professional. This article aims to raise awareness of the importance and need of sign language in dental clinical care and professional training, as well as strategies, tools and recommendations for better care for deaf people. Databases such as SciELO, Google Scholar, PubMed and ScienceDirect were included during the years 2007 and 2021 It was concluded that the main barriers during communication are the little knowledge of sign language in dentists, the little access to interpreters and difficulties in obtaining appointments; affecting the susceptibility to develop oral diseases. The inclusion of sign language courses within the university curriculum provides academic and professional ethics benefits; however, follow-up is necessary to confirm compliance with the requested parameters. Among the main tools and strategies, the use of educational videos, the accompaniment of interpreters and, in view of technological development, the use of mobile applications, can facilitate communication. Likewise, a set of recommendations is proposed for the approach of adult and child deaf patients.

Dynamic Gesture Recognition Based on FMCW Millimeter Wave Radar: Review of Methodologies and Results

As a convenient and natural way of human-computer interaction, gesture recognition technology has broad research and application prospects in many fields, such as intelligent perception and virtual reality. This paper summarized the relevant literature on gesture recognition using Frequency Modulated Continuous Wave (FMCW) millimeter-wave radar from January 2015 to June 2023. In the manuscript, the widely used methods involved in data acquisition, data processing, and classification in gesture recognition were systematically investigated. This paper counts the information related to FMCW millimeter wave radar, gestures, data sets, and the methods and results in feature extraction and classification. Based on the statistical data, we provided analysis and recommendations for other researchers. Key issues in the studies of current gesture recognition, including feature fusion, classification algorithms, and generalization, were summarized and discussed. Finally, this paper discussed the incapability of the current gesture recognition technologies in complex practical scenes and their real-time performance for future development.

BenSignNet: Bengali Sign Language Alphabet Recognition Using Concatenated Segmentation and Convolutional Neural Network

Sign language recognition is one of the most challenging applications in machine learning and human-computer interaction. Many researchers have developed classification models for different sign languages such as English, Arabic, Japanese, and Bengali; however, no significant research has been done on the general-shape performance for different datasets. Most research work has achieved satisfactory performance with a small dataset. These models may fail to replicate the same performance for evaluating different and larger datasets. In this context, this paper proposes a novel method for recognizing Bengali sign language (BSL) alphabets to overcome the issue of generalization. The proposed method has been evaluated with three benchmark datasets such as ‘38 BdSL’, ‘KU-BdSL’, and ‘Ishara-Lipi’. Here, three steps are followed to achieve the goal: segmentation, augmentation, and Convolutional neural network (CNN) based classification. Firstly, a concatenated segmentation approach with YCbCr, HSV and watershed algorithm was designed to accurately identify gesture signs. Secondly, seven image augmentation techniques are selected to increase the training data size without changing the semantic meaning. Finally, the CNN-based model called BenSignNet was applied to extract the features and classify purposes. The performance accuracy of the model achieved 94.00%, 99.60%, and 99.60% for the BdSL Alphabet, KU-BdSL, and Ishara-Lipi datasets, respectively. Experimental findings confirmed that our proposed method achieved a higher recognition rate than the conventional ones and accomplished a generalization property in all datasets for the BSL domain.

A Comparative Review on Applications of Different Sensors for Sign Language Recognition

Sign language recognition is challenging due to the lack of communication between normal and affected people. Many social and physiological impacts are created due to speaking or hearing disability. A lot of different dimensional techniques have been proposed previously to overcome this gap. A sensor-based smart glove for sign language recognition (SLR) proved helpful to generate data based on various hand movements related to specific signs. A detailed comparative review of all types of available techniques and sensors used for sign language recognition was presented in this article. The focus of this paper was to explore emerging trends and strategies for sign language recognition and to point out deficiencies in existing systems. This paper will act as a guide for other researchers to understand all materials and techniques like flex resistive sensor-based, vision sensor-based, or hybrid system-based technologies used for sign language until now.

Decoding lip language using triboelectric sensors with deep learning

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.

Artificial Intelligence Technologies for Sign Language

AI technologies can play an important role in breaking down the communication barriers of deaf or hearing-impaired people with other communities, contributing significantly to their social inclusion. Recent advances in both sensing technologies and AI algorithms have paved the way for the development of various applications aiming at fulfilling the needs of deaf and hearing-impaired communities. To this end, this survey aims to provide a comprehensive review of state-of-the-art methods in sign language capturing, recognition, translation and representation, pinpointing their advantages and limitations. In addition, the survey presents a number of applications, while it discusses the main challenges in the field of sign language technologies. Future research direction are also proposed in order to assist prospective researchers towards further advancing the field.

Emerging Wearable Interfaces and Algorithms for Hand Gesture Recognition: A Survey

Hands are vital in a wide range of fundamental daily activities, and neurological diseases that impede hand function can significantly affect quality of life. Wearable hand gesture interfaces hold promise to restore and assist hand function and to enhance human-human and human-computer communication. The purpose of this review is to synthesize current novel sensing interfaces and algorithms for hand gesture recognition, and the scope of applications covers rehabilitation, prosthesis control, sign language recognition, and human-computer interaction. Results showed that electrical, dynamic, acoustical/vibratory, and optical sensing were the primary input modalities in gesture recognition interfaces. Two categories of algorithms were identified: 1) classification algorithms for predefined, fixed hand poses and 2) regression algorithms for continuous finger and wrist joint angles. Conventional machine learning algorithms, including linear discriminant analysis, support vector machines, random forests, and non-negative matrix factorization, have been widely used for a variety of gesture recognition applications, and deep learning algorithms have more recently been applied to further facilitate the complex relationship between sensor signals and multi-articulated hand postures. Future research should focus on increasing recognition accuracy with larger hand gesture datasets, improving reliability and robustness for daily use outside of the laboratory, and developing softer, less obtrusive interfaces.