Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

Fuzzy Synchronization Likelihood Graph in Deep Neural Networks for Human Motion Time Series Analysis

Variable interactivity is crucial in biological multivariate time series analysis. This research suggests using graph structures to represent such interactions for more explainable decision-making processes. However, measuring the variable interaction in a graph is an open problem with no unique solution. Existing graph construction methods are either computationally costly, require extensive training, or disregard the inherent data nonlinearities and nonstationarity. We propose using the Fuzzy Synchronization Likelihood (FSL) criterion to address these challenges in constructing a graph and examining the qualitative similarity and dependency among variables. We propose applying this strategy to automated rehabilitation exercise evaluations based on human joint motion data. This multivariate time series application benefits from FSL-constructed graphs by offering further insight into the kinematics of joint interactions. Finally, we extend the convolutional layer in the Deep Mixture Density Neural Network (DMDN) to process the FSL-constructed graph, extracting practical information regarding task-based variable dependencies. An ablation study shows that the proposed FSL Graph-based Deep Neural Network (FSLGDN) outperforms its competitive approaches that use linear correlations and human anatomy for graph construction. Results also indicate that task-based consideration of joint motion data interactions is more beneficial than anatomy. Furthermore, the inherent nonstationarity of motion data leads to the extraction of more information than its linear correlation counterpart. Finally, while the proposed approach ranks competitively with a DMDN, the proposed approach's graph construction and representation of feature dependencies are more intuitive, leading to more explainable decision-making processes.

Fusing CNNs and attention-mechanisms to improve real-time indoor Human Activity Recognition for classifying home-based physical rehabilitation exercises

Physical rehabilitation plays a critical role in enhancing health outcomes globally. However, the shortage of physiotherapists, particularly in developing countries where the ratio is approximately ten physiotherapists per million people, poses a significant challenge to effective rehabilitation services. The existing literature on rehabilitation often falls short in data representation and the employment of diverse modalities, limiting the potential for advanced therapeutic interventions. To address this gap, This study integrates Computer Vision and Human Activity Recognition (HAR) technologies to support home-based rehabilitation. The study mitigates this gap by exploring various modalities and proposing a framework for data representation. We introduce a novel framework that leverages both Continuous Wavelet Transform (CWT) and Mel-Frequency Cepstral Coefficients (MFCC) for skeletal data representation. CWT is particularly valuable for capturing the time-frequency characteristics of dynamic movements involved in rehabilitation exercises, enabling a comprehensive depiction of both temporal and spectral features. This dual capability is crucial for accurately modelling the complex and variable nature of rehabilitation exercises. In our analysis, we evaluate 20 CNN-based models and one Vision Transformer (ViT) model. Additionally, we propose 12 hybrid architectures that combine CNN-based models with ViT in bi-model and tri-model configurations. These models are rigorously tested on the UI-PRMD and KIMORE benchmark datasets using key evaluation metrics, including accuracy, precision, recall, and F1-score, with 5-fold cross-validation. Our evaluation also considers real-time performance, model size, and efficiency on low-power devices, emphasising practical applicability. The proposed fused tri-model architectures outperform both single-architectures and bi-model configurations, demonstrating robust performance across both datasets and making the fused models the preferred choice for rehabilitation tasks. Our proposed hybrid model, DenMobVit, consistently surpasses state-of-the-art methods, achieving accuracy improvements of 2.9% and 1.97% on the UI-PRMD and KIMORE datasets, respectively. These findings highlight the effectiveness of our approach in advancing rehabilitation technologies and bridging the gap in physiotherapy services.

Mapping Method of Human Arm Motion Based on Surface Electromyography Signals

This paper investigates a method for precise mapping of human arm movements using sEMG signals. A multi-channel approach captures the sEMG signals, which, combined with the accurately calculated joint angles from an Inertial Measurement Unit, allows for action recognition and mapping through deep learning algorithms. Firstly, signal acquisition and processing were carried out, which involved acquiring data from various movements (hand gestures, single-degree-of-freedom joint movements, and continuous joint actions) and sensor placement. Then, interference signals were filtered out through filters, and the signals were preprocessed using normalization and moving averages to obtain sEMG signals with obvious features. Additionally, this paper constructs a hybrid network model, combining Convolutional Neural Networks and Artificial Neural Networks, and employs a multi-feature fusion algorithm to enhance the accuracy of gesture recognition. Furthermore, a nonlinear fitting between sEMG signals and joint angles was established based on a backpropagation neural network, incorporating momentum term and adaptive learning rate adjustments. Finally, based on the gesture recognition and joint angle prediction model, prosthetic arm control experiments were conducted, achieving highly accurate arm movement prediction and execution. This paper not only validates the potential application of sEMG signals in the precise control of robotic arms but also lays a solid foundation for the development of more intuitive and responsive prostheses and assistive devices.

Action Recognition of Taekwondo Unit Actions Using Action Images Constructed with Time-Warped Motion Profiles

Taekwondo has evolved from a traditional martial art into an official Olympic sport. This study introduces a novel action recognition model tailored for Taekwondo unit actions, utilizing joint-motion data acquired via wearable inertial measurement unit (IMU) sensors. The utilization of IMU sensor-measured motion data facilitates the capture of the intricate and rapid movements characteristic of Taekwondo techniques. The model, underpinned by a conventional convolutional neural network (CNN)-based image classification framework, synthesizes action images to represent individual Taekwondo unit actions. These action images are generated by mapping joint-motion profiles onto the RGB color space, thus encapsulating the motion dynamics of a single unit action within a solitary image. To further refine the representation of rapid movements within these images, a time-warping technique was applied, adjusting motion profiles in relation to the velocity of the action. The effectiveness of the proposed model was assessed using a dataset compiled from 40 Taekwondo experts, yielding remarkable outcomes: an accuracy of 0.998, a precision of 0.983, a recall of 0.982, and an F1 score of 0.982. These results underscore this time-warping technique's contribution to enhancing feature representation, as well as the proposed method's scalability and effectiveness in recognizing Taekwondo unit actions.

Skeleton-Based Human Pose Recognition Using Channel State Information: A Survey

With the increasing demand for human-computer interaction and health monitoring, human behavior recognition with device-free patterns has attracted extensive attention. The fluctuations of the Wi-Fi signal caused by human actions in a Wi-Fi coverage area can be used to precisely identify the human skeleton and pose, which effectively overcomes the problems of the traditional solution. Although many promising results have been achieved, no survey summarizes the research progress. This paper aims to comprehensively investigate and analyze the latest applications of human behavior recognition based on channel state information (CSI) and the human skeleton. First, we review the human profile perception and skeleton recognition progress based on wireless perception technologies. Second, we summarize the general framework of precise pose recognition, including signal preprocessing methods, neural network models, and performance results. Then, we classify skeleton model generation methods into three categories and emphasize the crucial difference among these typical applications. Furthermore, we discuss two aspects, such as experimental scenarios and recognition targets. Finally, we conclude the paper by summarizing the issues in typical systems and the main research directions for the future.

A Spatio-Temporal Graph Convolutional Network Model for Internet of Medical Things (IoMT)

In order to provide intelligent and efficient healthcare services in the Internet of Medical Things (IoMT), human action recognition (HAR) can play a crucial role. As a result of their stringent requirements, such as high computational complexity and memory efficiency, classical HAR techniques are not applicable to modern and intelligent healthcare services, e.g., IoMT. To address these issues, we present in this paper a novel HAR technique for healthcare services in IoMT. This model, referred to as the spatio-temporal graph convolutional network (STGCN), primarily aims at skeleton-based human-machine interfaces. By independently extracting spatial and temporal features, STGCN significantly reduces information loss. Spatio-temporal information is extracted independently of the exact spatial and temporal point, ensuring the extraction of useful features for HAR. Using only joint data and fewer parameters, we demonstrate that our proposed STGCN achieved 92.2% accuracy on the skeleton dataset. Unlike multi-channel methods, which use a combination of joint and bone data and have a large number of parameters, multi-channel methods use both joint and bone data. As a result, STGCN offers a good balance between accuracy, memory consumption, and processing time, making it suitable for detecting medical conditions.

A Survey on Wireless Wearable Body Area Networks: A Perspective of Technology and Economy

The deployment of wearable or body-worn devices is increasing rapidly, and thus researchers' interests mainly include technical and economical issues, such as networking, interoperability, security, power optimization, business growth and regulation. To address these issues properly, previous survey papers usually focused on describing the wireless body area network architecture and network protocols. This implies that deployment issues and awareness issues of wearable and BAN devices are not emphasized in previous work. To defeat this problem, in this study, we have focused on feasibility, limitations, and security concerns in wireless body area networks. In the aspect of the economy, we have focused on the compound annual growth rate of these devices in the global market, different regulations of wearable/wireless body area network devices in different regions and countries of the world and feasible research projects for wireless body area networks. In addition, this study focuses on the domain of devices that are equally important to physicians, sportsmen, trainers and coaches, computer scientists, engineers, and investors. The outcomes of this study relating to physicians, fitness trainers and coaches indicate that the use of these devices means they would be able to treat their clients in a more effective way. The study also converges the focus of businessmen on the Annual Growth Rate (CAGR) and provides manufacturers and vendors with information about different regulatory bodies that are monitoring and regulating WBAN devices. Therefore, by providing deployment issues in the aspects of technology and economy at the same time, we believe that this survey can serve as a preliminary material that will lead to more advancements and improvements in deployment in the area of wearable wireless body area networks. Finally, we present open issues and further research direction in the area of wireless body area networks.

A Graph Skeleton Transformer Network for Action Recognition

Skeleton-based action recognition is a research hotspot in the field of computer vision. Currently, the mainstream method is based on Graph Convolutional Networks (GCNs). Although there are many advantages of GCNs, GCNs mainly rely on graph topologies to draw dependencies between the joints, which are limited in capturing long-distance dependencies. Meanwhile, Transformer-based methods have been applied to skeleton-based action recognition because they effectively capture long-distance dependencies. However, existing Transformer-based methods lose the inherent connection information of human skeleton joints because they do not yet focus on initial graph structure information. This paper aims to improve the accuracy of skeleton-based action recognition. Therefore, a Graph Skeleton Transformer network (GSTN) for action recognition is proposed, which is based on Transformer architecture to extract global features, while using undirected graph information represented by the symmetric matrix to extract local features. Two encodings are utilized in feature processing to improve joints’ semantic and centrality features. In the process of multi-stream fusion strategies, a grid-search-based method is used to assign weights to each input stream to optimize the fusion results. We tested our method using three action recognition datasets: NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA. The experimental results show that our model’s accuracy is comparable to state-of-the-art approaches.