Skeleton Driven Action Recognition Using an Image-Based Spatial-Temporal Representation and Convolution Neural Network

A Multimodal Fusion Approach for Human Activity Recognition

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

Multiview child motor development dataset for AI-driven assessment of child development

BACKGROUND

Children's motor development is a crucial tool for assessing developmental levels, identifying developmental disorders early, and taking appropriate action. Although the Korean Developmental Screening Test for Infants and Children (K-DST) can accurately assess childhood development, its dependence on parental surveys rather than reliable, professional observation limits it. This study constructed a dataset based on a skeleton of recordings of K-DST behaviors in children aged between 20 and 71 months, with and without developmental disorders. The dataset was validated using a child behavior artificial intelligence (AI) learning model to highlight its possibilities.

RESULTS

The 339 participating children were divided into 3 groups by age. We collected videos of 4 behaviors by age group from 3 different angles and extracted skeletons from them. The raw data were used to annotate labels for each image, denoting whether each child performed the behavior properly. Behaviors were selected from the K-DST's gross motor section. The number of images collected differed by age group. The original dataset underwent additional processing to improve its quality. Finally, we confirmed that our dataset can be used in the AI model with 93.94%, 87.50%, and 96.31% test accuracy for the 3 age groups in an action recognition model. Additionally, the models trained with data including multiple views showed the best performance.

CONCLUSION

Ours is the first publicly available dataset that constitutes skeleton-based action recognition in young children according to the standardized criteria (K-DST). This dataset will enable the development of various models for developmental tests and screenings.

Multiple Sensor Synchronization with theRealSense RGB-D Camera

When reconstructing a 3D object, it is difficult to obtain accurate 3D geometric information using a single camera. In order to capture detailed geometric information of a 3D object, it is inevitable to increase the number of cameras to capture the object. However, cameras need to be synchronized in order to simultaneously capture frames. If cameras are incorrectly synchronized, many artifacts are produced in the reconstructed 3D object. The RealSense RGB-D camera, which is commonly used for obtaining geometric information of a 3D object, provides synchronization modes to mitigate synchronization errors. However, the synchronization modes provided by theRealSense cameras can only sync depth cameras and have limitations in the number of cameras that can be synchronized using a single host due to the hardware issue of stable data transmission. Therefore, in this paper, we propose a novel synchronization method that synchronizes an arbitrary number of RealSense cameras by adjusting the number of hosts to support stable data transmission. Our method establishes a master–slave architecture in order to synchronize the system clocks of the hosts. While synchronizing the system clocks, delays that resulted from the process of synchronization were estimated so that the difference between the system clocks could be minimized. Through synchronization of the system clocks, cameras connected to the different hosts can be synchronized based on the timestamp of the data received by the hosts. Thus, our method synchronizes theRealSense cameras to simultaneously capture accurate 3D information of an object at a constant frame rate without dropping it.