RGB-D ergonomic assessment system of adopted working postures

Soft computing applications in the field of human factors and ergonomics: A review of the past decade of research

The main objectives of this study were to 1) review the literature on the applications of soft computing concepts to the field of human factors and ergonomics (HFE) between 2013 and 2022 and 2) highlight future developments and trends. Multiple soft computing methods and techniques have been investigated for their ability to address various applications in HFE effectively. These techniques include fuzzy logic, artificial neural networks, genetic algorithms, and their combinations. Applications of these methods in HFE have been highlighted in one hundred and four articles selected from 406 papers. The results of this study help address the challenges of complexity, vagueness, and imprecision in human factors and ergonomics research through the application of soft computing methodologies.

Assessing Non-Specific Neck Pain through Pose Estimation from Images Based on Ensemble Learning

BACKGROUND

Mobile phones, laptops, and computers have become an indispensable part of our lives in recent years. Workers may have an incorrect posture when using a computer for a prolonged period of time. Using these products with an incorrect posture can lead to neck pain. However, there are limited data on postures in real-life situations.

METHODS

In this study, we used a common camera to record images of subjects carrying out three different tasks (a typing task, a gaming task, and a video-watching task) on a computer. Different artificial intelligence (AI)-based pose estimation approaches were applied to analyze the head's yaw, pitch, and roll and coordinate information of the eyes, nose, neck, and shoulders in the images. We used machine learning models such as random forest, XGBoost, logistic regression, and ensemble learning to build a model to predict whether a subject had neck pain by analyzing their posture when using the computer.

RESULTS

After feature selection and adjustment of the predictive models, nested cross-validation was applied to evaluate the models and fine-tune the hyperparameters. Finally, the ensemble learning approach was utilized to construct a model via bagging, which achieved a performance with 87% accuracy, 92% precision, 80.3% recall, 95.5% specificity, and an AUROC of 0.878.

CONCLUSIONS

We developed a predictive model for the identification of non-specific neck pain using 2D video images without the need for costly devices, advanced environment settings, or extra sensors. This method could provide an effective way for clinically evaluating poor posture during real-world computer usage scenarios.

Study on the Interaction Behaviors Identification of Construction Workers Based on ST-GCN and YOLO

The construction industry is accident-prone, and unsafe behaviors of construction workers have been identified as a leading cause of accidents. One important countermeasure to prevent accidents is monitoring and managing those unsafe behaviors. The most popular way of detecting and identifying workers' unsafe behaviors is the computer vision-based intelligent monitoring system. However, most of the existing research or products focused only on the workers' behaviors (i.e., motions) recognition, limited studies considered the interaction between man-machine, man-material or man-environments. Those interactions are very important for judging whether the workers' behaviors are safe or not, from the standpoint of safety management. This study aims to develop a new method of identifying construction workers' unsafe behaviors, i.e., unsafe interaction between man-machine/material, based on ST-GCN (Spatial Temporal Graph Convolutional Networks) and YOLO (You Only Look Once), which could provide more direct and valuable information for safety management. In this study, two trained YOLO-based models were, respectively, used to detect safety signs in the workplace, and objects that interacted with workers. Then, an ST-GCN model was trained to detect and identify workers' behaviors. Lastly, a decision algorithm was developed considering interactions between man-machine/material, based on YOLO and ST-GCN results. Results show good performance of the developed method, compared to only using ST-GCN, the accuracy was significantly improved from 51.79% to 85.71%, 61.61% to 99.11%, and 58.04% to 100.00%, respectively, in the identification of the following three kinds of behaviors, throwing (throwing hammer, throwing bottle), operating (turning on switch, putting bottle), and crossing (crossing railing and crossing obstacle). The findings of the study have some practical implications for safety management, especially workers' behavior monitoring and management.

Effect of Face Blurring on Human Pose Estimation: Ensuring Subject Privacy for Medical and Occupational Health Applications

The face blurring of images plays a key role in protecting privacy. However, in computer vision, especially for the human pose estimation task, machine-learning models are currently trained, validated, and tested on original datasets without face blurring. Additionally, the accuracy of human pose estimation is of great importance for kinematic analysis. This analysis is relevant in areas such as occupational safety and clinical gait analysis where privacy is crucial. Therefore, in this study, we explore the impact of face blurring on human pose estimation and the subsequent kinematic analysis. Firstly, we blurred the subjects’ heads in the image dataset. Then we trained our neural networks using the face-blurred and the original unblurred dataset. Subsequently, the performances of the different models, in terms of landmark localization and joint angles, were estimated on blurred and unblurred testing data. Finally, we examined the statistical significance of the effect of face blurring on the kinematic analysis along with the strength of the effect. Our results reveal that the strength of the effect of face blurring was low and within acceptable limits (<1°). We have thus shown that for human pose estimation, face blurring guarantees subject privacy while not degrading the prediction performance of a deep learning model.

Sensor-Based Motion Tracking System Evaluation for RULA in Assembly Task

Industries need a mechanism to monitor the workers' safety and to prevent Work-related Musculoskeletal Disorders (WMSDs). The development of ergonomics assessment tools helps the industry evaluate workplace design and worker posture. Many studies proposed the automated ergonomics assessment method to replace the manual; however, it only focused on calculating body angle and assessing the wrist section manually. This study aims to (a) propose a wrist kinematics measurement based on unobtrusive sensors, (b) detect potential WMSDs related to wrist posture, and (c) compare the wrist posture of subjects while performing assembly tasks to achieve a comprehensive and personalized ergonomic assessment. The wrist posture measurement is combined with the body posture measurement to provide a comprehensive ergonomics assessment based on RULA. Data were collected from subjects who performed the assembly process to evaluate our method. We compared the risk score assessed by the ergonomist and the risk score generated by our method. All body segments achieved more than an 80% similarity score, enhancing the scores for wrist position and wrist twist by 6.8% and 0.3%, respectively. A hypothesis analysis was conducted to evaluate the difference across the subjects. The results indicate that every subject performs tasks differently and has different potential risks regarding wrist posture.

Reliability Analysis of Observation-Based Exposure Assessment Tools for the Upper Extremities: A Systematic Review

(1) Background: The objectives of this systematic review were to (i) summarize the results of studies evaluating the reliability of observational ergonomics exposure assessment tools addressing exposure to physical risk factors associated with upper extremity musculoskeletal disorders (MSDs), and (ii) identify best practices for assessing the reliability of new observational exposure assessment tools. (2) Methods: A broad search was conducted in March 2020 of four academic databases: PubMed, Science Direct, Ergonomic Abstracts, and Web of Science. Articles were systematically excluded by removing redundant articles, examining titles and abstracts, assessing relevance to physical ergonomics and the upper extremities, and article type. (3) Results: Eleven articles were included in the review. The results indicated no singular best practice; instead, there were multiple methodological approaches researchers chose to use. Some of the significant variations in methodologies include the selection of reliability coefficients, rater and participant selection, and direct vs. digital observation. (4) Conclusion: The findings serve as a resource summarizing the reliability of existing observational risk assessment tools and identify common methods for assessing the reliability of new observational risk assessment tools. Limitations of this review include the number of databases searched, the removal of truncation symbols, and the selection of keywords used for the initial search.

Deep learning applications in manufacturing operations: a review of trends and ways forward

Purpose

Deep learning (DL) technologies assist manufacturers to manage their business operations. This research aims to present state-of-the-art insights on the trends and ways forward for DL applications in manufacturing operations.

Design/methodology/approach

Using bibliometric analysis and the SPAR-4-SLR protocol, this research conducts a systematic literature review to present a scientific mapping of top-tier research on DL applications in manufacturing operations.

Findings

This research discovers and delivers key insights on six knowledge clusters pertaining to DL applications in manufacturing operations: automated system modelling, intelligent fault diagnosis, forecasting, sustainable manufacturing, environmental management, and intelligent scheduling.

Research limitations/implications

This research establishes the important roles of DL in manufacturing operations. However, these insights were derived from top-tier journals only. Therefore, this research does not discount the possibility of the availability of additional insights in alternative outlets, such as conference proceedings, where teasers into emerging and developing concepts may be published.

Originality/value

This research contributes seminal insights into DL applications in manufacturing operations. In this regard, this research is valuable to readers (academic scholars and industry practitioners) interested to gain an understanding of the important roles of DL in manufacturing operations as well as the future of its applications for Industry 4.0, such as Maintenance 4.0, Quality 4.0, Logistics 4.0, Manufacturing 4.0, Sustainability 4.0, and Supply Chain 4.0.

A Work-Related Musculoskeletal Disorders (WMSDs) Risk-Assessment System Using a Single-View Pose Estimation Model

Musculoskeletal disorders are an unavoidable occupational health problem. In particular, workers who perform repetitive tasks onsite in the manufacturing industry suffer from musculoskeletal problems. In this paper, we propose a system that evaluates the posture of workers in the manufacturing industry with single-view 3D human pose-estimation that can estimate the posture in 3D using an RGB camera that can easily acquire the posture of a worker in a complex workplace. The proposed system builds a Duckyang-Auto Worker Health Safety Environment (DyWHSE), a manufacturing-industry-specific dataset, to estimate the wrist pose evaluated by the Rapid Limb Upper Assessment (RULA). Additionally, we evaluate the quality of the built DyWHSE dataset using the Human3.6M dataset, and the applicability of the proposed system is verified by comparing it with the evaluation results of the experts. The proposed system provides quantitative assessment guidance for working posture risk assessment, assisting the continuous posture assessment of workers.

Accuracy Assessment of Joint Angles Estimated from 2D and 3D Camera Measurements

To automatically evaluate the ergonomics of workers, 3D skeletons are needed. Most ergonomic assessment methods, like REBA, are based on the different 3D joint angles. Thanks to the huge amount of training data, 2D skeleton detectors have become very accurate. In this work, we test three methods to calculate 3D skeletons from 2D detections: using the depth from a single RealSense range camera, triangulating the joints using multiple cameras, and combining the triangulation of multiple camera pairs. We tested the methods using recordings of a person doing different assembly tasks. We compared the resulting joint angles to the ground truth of a VICON marker-based tracking system. The resulting RMS angle error for the triangulation methods is between 12° and 16°, showing that they are accurate enough to calculate a useful ergonomic score from.

The role of machine learning in the primary prevention of work-related musculoskeletal disorders: A scoping review

To determine the applications of machine learning (ML) techniques used for the primary prevention of work-related musculoskeletal disorders (WMSDs), a scoping review was conducted using seven literature databases. Of the 4,639 initial results, 130 primary research studies were deemed relevant for inclusion. Studies were reviewed and classified as a contribution to one of six steps within the primary WMSD prevention research framework by van der Beek et al. (2017). ML techniques provided the greatest contributions to the development of interventions (48 studies), followed by risk factor identification (33 studies), underlying mechanisms (29 studies), incidence of WMSDs (14 studies), evaluation of interventions (6 studies), and implementation of effective interventions (0 studies). Nearly a quarter (23.8%) of all included studies were published in 2020. These findings provide insight into the breadth of ML techniques used for primary WMSD prevention and can help identify areas for future research and development.

Collaborative Workplace Design: A Knowledge-Based Approach to Promote Human–Robot Collaboration and Multi-Objective Layout Optimization

The innovation-driven Industry 5.0 leads us to consider humanity in a prominent position as the center of the manufacturing field even more than Industry 4.0. This pushes us towards the hybridization of manufacturing plants promoting a full collaboration between humans and robots. However, there are currently very few workplaces where effective Human–Robot Collaboration takes place. Layout designing plays a key role in assuring safe and efficient Human–Robot Collaboration. The layout design, especially in the context of collaborative robotics, is a complex problem to face, since it is related to safety, ergonomics, and productivity aspects. In the current work, a Knowledge-Based Approach (KBA) is adopted to face the complexity of the layout design problem. The framework resulting from the KBA allows for developing a modeling paradigm that enables us to define a streamlined approach for the layout design. The proposed approach allows for placing resource within the workplace according to a defined optimization criterion, and also ensures compliance with various standards. This approach is applied to an industrial case study in order to prove its feasibility. A what-if analysis is performed by applying the proposed approach. Changing three control factors (i.e., minimum distance, robot speed, logistic space configuration) on three levels, in a Design of Experiments, 27 layout configurations of the same workplace are generated. Consequently, the inputs that most affect the layout design are identified by means of an Analysis of Variance (ANOVA). The results show that only one layout is eligible to be the best configuration, and only two out of three control factors are very significant for the designing of the HRC workplace layout. Hence, the proposed approach enables the designing of standard compliant and optimized HRC workplace layouts. Therefore, several alternatives of the layout for the same workplace can be easily generated and investigated in a systematic manner.

An Ergonomic Assessment of Different Postures and Children Risk during Evacuations

Crawling is recommended for avoiding high heat and toxic fumes and for obtaining more breathable air during evacuations. Few studies have evaluated the effects of crawling on physical joints and velocity, especially in children. Based on motion capture technology, this study proposes a novel method of using wearable sensors to collect exposure (e.g., mean duration, frequency) on children’s joints to objectively quantify the impacts of different locomotion methods on physical characteristics. An on-site experiment was conducted in a kindergarten with 28 children (13 boys and 15 girls) of different ages (4–6 years old) who traveled up to 22 m in three different postures: upright walking (UW), stoop walking (SW), and knee and hand crawling (KHC). The results showed that: (1) The level of joint fatigue for KHC was heavier than bipedal walking (p < 0.05), which was evidenced by higher mean duration and frequency. There was no significant difference between UW and SW (p > 0.05). (2) The physical characteristics of the children in the different postures observed in this study were different (p < 0.05). The ankle was more fatigued than other joints during bipedal walking. Unlike infants, the wrists and hips of the children became fatigued while crawling. The key actions flexion/extension are more likely to induce joint fatigue vs. other actions. (3) Crawling velocity was significantly slower than the bipedal velocities, and UW was 10.6% faster than SW (p < 0.05). The bipedal walking velocity started to decrease after the children had travelled up to 13 m, while the KHC velocity started to decrease after traveling up to 11.6 m. (4) In a severe fire, the adoption of SW is suggested, as the evacuees can both evacuate quickly and avoid overworking their joints. (5) There were no significant differences in the age (p > 0.05) and gender (p > 0.05) of the children on the joints in any of the three postures. To conclude, KHC causes more damage to body joints compared to bipedal walking, as evidenced by higher exposure (mean duration, frequency), whereas UW and SW are similar in terms of the level of joint fatigue. The above findings are expected to provide a useful reference for future applications in the children’s risk assessment and in the prevention design of buildings.

Using Kinect body joint detection system to predict energy expenditures during physical activities

The purpose of this study was to explore the potential of Kinect body joint detection to facilitate the calculation of energy expenditure during exergame exercises. Two Kinect-based biomechanical models - mechanical energy (KineticE) and work (WorkE) were employed to estimate the energy expenditure during four Wii™ exergame session. Consequently, two stepwise regression models were developed from nineteen participants' data and then validated by five holdout participants. The data collected using an accelerometer (r = 0.835, p < 0.001) had the highest correlation as compared to that of the WorkE (r = 0.805, p < 0.001) and KineticE (r = 0.466, p < 0.001) correlations with the reference indirect calorimetry using Quark activity energy expenditure (QuarkAEE). The regression results show that KineticE and the weight of the participant were significant factors for mechanical energy prediction (AEEKinetic). However, according to the work prediction equation (AEEWork), only WorkE was significant. The new energy prediction models showed significant agreement with the standard QuarkAEE (AEEKinect, r = 0.641, p = 0.02; AEEWork, r = 0.793, p < 0.001), and they were comparable to accelerometer predictions (r = 0.682, p = 0.001). The findings indicate that Kinect can be a potentially viable alternative to measure energy expenditures. The models can be applied with higher accuracy, especially when the activity demands high body movements.

Digitalization of Musculoskeletal Risk Assessment in a Robotic-Assisted Assembly Workstation

The ergonomic assessment of adopted working postures is essential for avoiding musculoskeletal risk factors in manufacturing contexts. Several observational methods based on external analyst observations are available; however, they are relatively subjective and suffer low repeatability. Over the past decade, the digitalization of this assessment has received high research interest. Robotic applications have the potential to lighten workers’ workload and improve working conditions. Therefore, this work presents a musculoskeletal risk assessment before and after robotic implementation in an assembly workstation. We also emphasize the importance of using novel and non-intrusive technologies for musculoskeletal risk assessment. A kinematic study was conducted using inertial motion units (IMU) in a convenience sample of two workers during their normal performance of assembly work cycles. The musculoskeletal risk was estimated according to a semi-automated solution, called the Rapid Upper Limb Assessment (RULA) report. Based on previous musculoskeletal problems reported by the company, the assessment centered on the kinematic analysis of functional wrist movements (flexion/extension, ulnar/radial deviation, and pronation/supination). The results of the RULA report showed a reduction in musculoskeletal risk using robotic-assisted assembly. Regarding the kinematic analysis of the wrist during robotic-assisted tasks, a significant posture improvement of 20–45% was registered (considering the angular deviations relative to the neutral wrist position). The results obtained by direct measurements simultaneously reflect the workload and individual characteristics. The current study highlights the importance of an in-field instrumented assessment of musculoskeletal risk and the limitations of the system applied (e.g., unsuitable for tracking the motion of small joints, such as the fingers).

Simple benchmarking method for determining the accuracy of depth cameras in body landmark location estimation: Static upright posture as a measurement example

To evaluate the postures in ergonomics applications, studies have proposed the use of low-cost, marker-less, and portable depth camera-based motion tracking systems (DCMTSs) as a potential alternative to conventional marker-based motion tracking systems (MMTSs). However, a simple but systematic method for examining the estimation errors of various DCMTSs is lacking. This paper proposes a benchmarking method for assessing the estimation accuracy of depth cameras for full-body landmark location estimation. A novel alignment board was fabricated to align the coordinate systems of the DCMTSs and MMTSs. The data from an MMTS were used as a reference to quantify the error of using a DCMTS to identify target locations in a 3-D space. To demonstrate the proposed method, the full-body landmark location tracking errors were evaluated for a static upright posture using two different DCMTSs. For each landmark, we compared each DCMTS (Kinect system and RealSense system) with an MMTS by calculating the Euclidean distances between symmetrical landmarks. The evaluation trials were performed twice. The agreement between the tracking errors of the two evaluation trials was assessed using intraclass correlation coefficient (ICC). The results indicate that the proposed method can effectively assess the tracking performance of DCMTSs. The average errors (standard deviation) for the Kinect system and RealSense system were 2.80 (1.03) cm and 5.14 (1.49) cm, respectively. The highest average error values were observed in the depth orientation for both DCMTSs. The proposed method achieved high reliability with ICCs of 0.97 and 0.92 for the Kinect system and RealSense system, respectively.

Prediction and comparison of postural discomfort based on MLP and quadratic regression

OBJECTIVE

The objective of this study was to predict postural discomfort based on the deep learning-based regression (multilayer perceptron [MLP] model).

METHODS

A total of 95 participants performed 45 different static postures as a combination of 3 neck angles, 5 trunk angles, and 3 knee angles and rated the whole-body discomfort. Two different combinations of variables including model 1 (all variables: gender, height, weight, exercise, body segment angles) and model 2 (gender, body segment angles) were tested. The MLP regression and a conventional regression (quadratic regression) were both conducted, and the performance was compared.

RESULTS

In the overall regression analysis, the quadratic regression showed better performance than the MLP regression. For the postural discomfort group-specific analysis, MLP regression showed greater performance than the quadratic regression especially in the high postural discomfort group. The MLP regression also showed better performance in predicting postural discomfort among individuals who had a variability of subjective rating among different postures compared to the quadratic regression. The deep learning for postural discomfort prediction would be useful for the efficient job risk assessment for various industries that involve prolonged static postures.

CONCLUSIONS

The deep learning for postural discomfort prediction would be useful for the efficient job risk assessment for various industries that involve prolonged static postures. This information would be meaningful as basic research data to study in predicting psychophysical data in ergonomics.

Prototype System for Measuring and Analyzing Movements of the Upper Limb for the Detection of Occupational Hazards

In the work environment, there are usually different pathologies that are related to Repetitive Efforts and Movements (REM) that tend to predominantly affect the upper limbs. To determine whether a worker is at risk of suffering some type of pathology, observation techniques are usually used by qualified technical personnel. In order to define from quantitative data if there is a risk of suffering a pathology due to movements and repetitive efforts in the upper limb, a prototype of a movement measurement system has been designed and manufactured. This system interferes minimally with the activity studied, maintaining a reduced cost of manufacture and use. The system allows the study of the movements made by the subject in the work environment by determining the origin of the Musculoskeletal Disorder (MSD) from the movements of the elbow and wrist, collecting data on the position and accelerations of the arm, forearm and hand, and taking into account the risk factors established for suffering from an MSD: high repetition of movements, the use of a high force in a repetitive manner, or the adoption of forced positions. The data obtained with this system can be analyzed by qualified personnel from tables, graphs, and 3D animations at the time of execution, or stored for later analysis.

An Evaluation of Posture Recognition Based on Intelligent Rapid Entire Body Assessment System for Determining Musculoskeletal Disorders

Determining the potential risks of musculoskeletal disorders through working postures in a workplace is expensive and time-consuming. A novel intelligent rapid entire body assessment (REBA) system based on convolutional pose machines (CPM), entitled the Quick Capture system, was applied to determine the risk levels. The aim of the study was to validate the feasibility and reliability of the CPM-based REBA system through a simulation experiment. The reliability was calculated from the differences of motion angles between the CPM-based REBA and a motion capture system. Results show the data collected by the Quick Capture system were consistent with those of the motion capture system; the average of root mean squared error (RMSE) was 4.77 and the average of Spearman’s rho (ρ) correlation coefficient in the different 12 postures was 0.915. For feasibility evaluation, the linear weighted Cohen’s kappa between the REBA score obtained by the Quick Capture system and those from the three experts were used. The result shows good agreement, with an average proportion agreement index (P₀) of 0.952 and kappa of 0.738. The Quick Capture system does not only accurately analyze working posture, but also accurately determines risk level of musculoskeletal disorders. This study suggested that the Quick Capture system could be applied for a rapid and real-time on-site assessment.

Just Don’t Fall: An AI Agent’s Learning Journey Towards Posture Stabilisation

Learning to maintain postural balance while standing requires a significant, fine coordination effort between the neuromuscular system and the sensory system. It is one of the key contributing factors towards fall prevention, especially in the older population. Using artificial intelligence (AI), we can similarly teach an agent to maintain a standing posture, and thus teach the agent not to fall. In this paper, we investigate the learning progress of an AI agent and how it maintains a stable standing posture through reinforcement learning. We used the Deep Deterministic Policy Gradient method (DDPG) and the OpenSim musculoskeletal simulation environment based on OpenAI Gym. During training, the AI agent learnt three policies. First, it learnt to maintain the Centre-of-Gravity and Zero-Moment-Point in front of the body. Then, it learnt to shift the load of the entire body on one leg while using the other leg for fine tuning the balancing action. Finally, it started to learn the coordination between the two pre-trained policies. This study shows the potentials of using deep reinforcement learning in human movement studies. The learnt AI behaviour also exhibited attempts to achieve an unplanned goal because it correlated with the set goal (e.g., walking in order to prevent falling). The failed attempts to maintain a standing posture is an interesting by-product which can enrich the fall detection and prevention research efforts.