The Influence of Robots' Fairness on Humans' Reward-Punishment Behaviors and Trust in Human-Robot Cooperative Teams

OBJECTIVE

Based on social exchange theory, this study investigates the effects of robots' fairness and social status on humans' reward-punishment behaviors and trust in human-robot interactions.

BACKGROUND

In human-robot teamwork, robots show fair behaviors, dedication (altruistic unfair behaviors), and selfishness (self-interested unfair behaviors), but few studies have discussed the effects of these robots' behaviors on teamwork.

METHOD

This study adopts a 3 (the independent variable is the robot's fairness: self-interested unfair behaviors, fair behaviors, and altruistic unfair behaviors) × 3 (the moderator variable is the robot's social status: superior, peer, and subordinate) experimental design. Each participant and a robot completed the experimental task together through a computer.

RESULTS

When robots have different social statuses, the more altruistic the fairness of the robot, the more reward behaviors, the fewer punishment behaviors, and the higher human-robot trust of humans. Robots' higher social status weakens the influence of their fairness on humans' punishment behaviors. Human-robot trust will increase humans' reward behaviors and decrease humans' punishment behaviors. Humans' reward-punishment behaviors will increase repaired human-robot trust.

CONCLUSION

Robots' fairness has a significant impact on humans' reward-punishment behaviors and trust. Robots' social status moderates the effect of their fair behavior on humans' punishment behavior. There is an interaction between humans' reward-punishment behaviors and trust.

APPLICATION

The study can help to better understand the interaction mechanism of the human-robot team and can better serve the management and cooperation of the human-robot team by appropriately adjusting the robots' fairness and social status.

MARSBot: A Bristle-Bot Microrobot with Augmented Reality Steering Control for Wireless Structural Health Monitoring

Microrobots are effective for monitoring infrastructure in narrow spaces. However, they have limited computing power, and most of them are not wireless and stable enough for accessing infrastructure in difficult-to-reach areas. In this paper, we describe the fabrication of a microrobot with bristle-bot locomotion using a novel centrifugal yaw-steering control scheme. The microrobot operates in a network consisting of an augmented reality headset and an access point to monitor infrastructures using augmented reality (AR) haptic controllers for human-robot collaboration. For the development of the microrobot, the dynamics of bristle-bots in several conditions were studied, and multiple additive manufacturing processes were investigated to develop the most suitable prototype for structural health monitoring. Using the proposed network, visual data are sent in real time to a hub connected to an AR headset upon request, which can be utilized by the operator to monitor and make decisions in the field. This allows the operators wearing an AR headset to inspect the exterior of a structure with their eyes, while controlling the surveying robot to monitor the interior side of the structure.

Dynamic Virtual Fixture Generation Based on Intra-Operative 3D Image Feedback in Robot-Assisted Minimally Invasive Thoracic Surgery

This paper proposes a method for generating dynamic virtual fixtures with real-time 3D image feedback to facilitate human-robot collaboration in medical robotics. Seamless shared control in a dynamic environment, like that of a surgical field, remains challenging despite extensive research on collaborative control and planning. To address this problem, our method dynamically creates virtual fixtures to guide the manipulation of a trocar-placing robot arm using the force field generated by point cloud data from an RGB-D camera. Additionally, the "view scope" concept selectively determines the region for computational points, thereby reducing computational load. In a phantom experiment for robot-assisted port incision in minimally invasive thoracic surgery, our method demonstrates substantially improved accuracy for port placement, reducing error and completion time by 50% (p=1.06×10-2) and 35% (p=3.23×10-2), respectively. These results suggest that our proposed approach is promising in improving surgical human-robot collaboration.

Research on Teleoperated Virtual Reality Human-Robot Five-Dimensional Collaboration System

In the realm of industrial robotics, there is a growing challenge in simplifying human-robot collaboration (HRC), particularly in complex settings. The demand for more intuitive teleoperation systems is on the rise. However, optimizing robot control interfaces and streamlining teleoperation remains a formidable task due to the need for operators to possess specialized knowledge and the limitations of traditional methods regarding operational space and time constraints. This study addresses these issues by introducing a virtual reality (VR) HRC system with five-dimensional capabilities. Key advantages of our approach include: (1) real-time observation of robot work, whereby operators can seamlessly monitor the robot's real-time work environment and motion during teleoperation; (2) leveraging VR device capabilities, whereby the strengths of VR devices are harnessed to simplify robot motion control, significantly reducing the learning time for operators; and (3) adaptability across platforms and environments: our system effortlessly adapts to various platforms and working conditions, ensuring versatility across different terminals and scenarios. This system represents a significant advancement in addressing the challenges of HRC, offering improved teleoperation, simplified control, and enhanced accessibility, particularly for operators with limited prior exposure to robot operation. It elevates the overall HRC experience in complex scenarios.

Learning Compliant Box-in-Box Insertion through Haptic-Based Robotic Teleoperation

In modern logistics, the box-in-box insertion task is representative of a wide range of packaging applications, and automating compliant object insertion is difficult due to challenges in modelling the object deformation during insertion. Using Learning from Demonstration (LfD) paradigms, which are frequently used in robotics to facilitate skill transfer from humans to robots, can be one solution for complex tasks that are difficult to mathematically model. In order to automate the box-in-box insertion task for packaging applications, this study makes use of LfD techniques. The proposed framework has three phases. Firstly, a master-slave teleoperated robot system is used in the initial phase to haptically demonstrate the insertion task. Then, the learning phase involves identifying trends in the demonstrated trajectories using probabilistic methods, in this case, Gaussian Mixture Regression. In the third phase, the insertion task is generalised, and the robot adjusts to any object position using barycentric interpolation. This method is novel because it tackles tight insertion by taking advantage of the boxes' natural compliance, making it possible to complete the task even with a position-controlled robot. To determine whether the strategy is generalisable and repeatable, experimental validation was carried out.

Modeling and Analysis of Human Comfort in Human-Robot Collaboration

The emergence and recent development of collaborative robots have introduced a safer and more efficient human-robot collaboration (HRC) manufacturing environment. Since the release of COBOTs, a great amount of research efforts have been focused on improving robot working efficiency, user safety, human intention detection, etc., while one significant factor-human comfort-has been frequently ignored. The comfort factor is critical to COBOT users due to its great impact on user acceptance. In previous studies, there is a lack of a mathematical-model-based approach to quantitatively describe and predict human comfort in HRC scenarios. Also, few studies have discussed the cases when multiple comfort factors take effect simultaneously. In this study, a multi-linear-regression-based general human comfort prediction model is proposed under human-robot collaboration scenarios, which is able to accurately predict the comfort levels of humans in multi-factor situations. The proposed method in this paper tackled these two gaps at the same time and also demonstrated the effectiveness of the approach with its high prediction accuracy. The overall average accuracy among all participants is 81.33%, while the overall maximum value is 88.94%, and the overall minimum value is 72.53%. The model uses subjective comfort rating feedback from human subjects as training and testing data. Experiments have been implemented, and the final results proved the effectiveness of the proposed approach in identifying human comfort levels in HRC.

A Unified Multimodal Interface for the RELAX High-Payload Collaborative Robot

This manuscript introduces a mobile cobot equipped with a custom-designed high payload arm called RELAX combined with a novel unified multimodal interface that facilitates Human-Robot Collaboration (HRC) tasks requiring high-level interaction forces on a real-world scale. The proposed multimodal framework is capable of combining physical interaction, Ultra Wide-Band (UWB) radio sensing, a Graphical User Interface (GUI), verbal control, and gesture interfaces, combining the benefits of all these different modalities and allowing humans to accurately and efficiently command the RELAX mobile cobot and collaborate with it. The effectiveness of the multimodal interface is evaluated in scenarios where the operator guides RELAX to reach designated locations in the environment while avoiding obstacles and performing high-payload transportation tasks, again in a collaborative fashion. The results demonstrate that a human co-worker can productively complete complex missions and command the RELAX mobile cobot using the proposed multimodal interaction framework.

Industry 5 and the Human in Human-Centric Manufacturing

Industry 4 (I4) was a revolutionary new stage for technological progress in manufacturing which promised a new level of interconnectedness between a diverse range of technologies. Sensors, as a point technology, play an important role in these developments, facilitating human-machine interaction and enabling data collection for system-level technologies. Concerns for human labour working in I4 environments (e.g., health and safety, data generation and extraction) are acknowledged by Industry 5 (I5), an update of I4 which promises greater attention to human-machine relations through a values-driven approach to collaboration and co-design. This article explores how engineering experts integrate values promoted by policy-makers into both their thinking about the human in their work and in their writing. This paper demonstrates a novel interdisciplinary approach in which an awareness of different disciplinary epistemic values associated with humans and work guides a systematic literature review and interpretive coding of practice-focussed engineering papers. Findings demonstrate evidence of an I5 human-centric approach: a high value for employees as "end-users" of innovative systems in manufacturing; and an increase in output addressing human activity in modelling and the technologies available to address this concern. However, epistemic publishing practices show that efforts to increase the effectiveness of manufacturing systems often neglect worker voice.

Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool Recognition Methods

This paper presents a novel method for online tool recognition in manual assembly processes. The goal was to develop and implement a method that can be integrated with existing Human Action Recognition (HAR) methods in collaborative tasks. We examined the state-of-the-art for progress detection in manual assembly via HAR-based methods, as well as visual tool-recognition approaches. A novel online tool-recognition pipeline for handheld tools is introduced, utilizing a two-stage approach. First, a Region Of Interest (ROI) was extracted by determining the wrist position using skeletal data. Afterward, this ROI was cropped, and the tool located within this ROI was classified. This pipeline enabled several algorithms for object recognition and demonstrated the generalizability of our approach. An extensive training dataset for tool-recognition purposes is presented, which was evaluated with two image-classification approaches. An offline pipeline evaluation was performed with twelve tool classes. Additionally, various online tests were conducted covering different aspects of this vision application, such as two assembly scenarios, unknown instances of known classes, as well as challenging backgrounds. The introduced pipeline was competitive with other approaches regarding prediction accuracy, robustness, diversity, extendability/flexibility, and online capability.

Erratum: Flexible sensor concept and an integrated collision sensing for efficient human-robot collaboration using 3D local global sensors

[This corrects the article DOI: 10.3389/frobt.2023.1028411.].

Cross-Viewpoint Semantic Mapping: Integrating Human and Robot Perspectives for Improved 3D Semantic Reconstruction

Allocentric semantic 3D maps are highly useful for a variety of human-machine interaction related tasks since egocentric viewpoints can be derived by the machine for the human partner. Class labels and map interpretations, however, may differ or could be missing for the participants due to the different perspectives. Particularly, when considering the viewpoint of a small robot, which significantly differs from the viewpoint of a human. In order to overcome this issue, and to establish common ground, we extend an existing real-time 3D semantic reconstruction pipeline with semantic matching across human and robot viewpoints. We use deep recognition networks, which usually perform well from higher (i.e., human) viewpoints but are inferior from lower viewpoints, such as that of a small robot. We propose several approaches for acquiring semantic labels for images taken from unusual perspectives. We start with a partial 3D semantic reconstruction from the human perspective that we transfer and adapt to the small robot's perspective using superpixel segmentation and the geometry of the surroundings. The quality of the reconstruction is evaluated in the Habitat simulator and a real environment using a robot car with an RGBD camera. We show that the proposed approach provides high-quality semantic segmentation from the robot's perspective, with accuracy comparable to the original one. In addition, we exploit the gained information and improve the recognition performance of the deep network for the lower viewpoints and show that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real-time, so the approach enables interactive applications.

LiDAR-Based Maintenance of a Safe Distance between a Human and a Robot Arm

This paper demonstrates the capabilities of three-dimensional (3D) LiDAR scanners in supporting a safe distance maintenance functionality in human-robot collaborative applications. The use of such sensors is severely under-utilised in collaborative work with heavy-duty robots. However, even with a relatively modest proprietary 3D sensor prototype, a respectable level of safety has been achieved, which should encourage the development of such applications in the future. Its associated intelligent control system (ICS) is presented, as well as the sensor's technical characteristics. It acquires the positions of the robot and the human periodically, predicts their positions in the near future optionally, and adjusts the robot's speed to keep its distance from the human above the protective separation distance. The main novelty is the possibility to load an instance of the robot programme into the ICS, which then precomputes the future position and pose of the robot. Higher accuracy and safety are provided, in comparison to traditional predictions from known real-time and near-past positions and poses. The use of a 3D LiDAR scanner in a speed and separation monitoring application and, particularly, its specific placing, are also innovative and advantageous. The system was validated by analysing videos taken by the reference validation camera visually, which confirmed its safe operation in reasonably limited ranges of robot and human speeds.

Towards a Safe Human-Robot Collaboration Using Information on Human Worker Activity

Most industrial workplaces involving robots and other apparatus operate behind the fences to remove defects, hazards, or casualties. Recent advancements in machine learning can enable robots to co-operate with human co-workers while retaining safety, flexibility, and robustness. This article focuses on the computation model, which provides a collaborative environment through intuitive and adaptive human-robot interaction (HRI). In essence, one layer of the model can be expressed as a set of useful information utilized by an intelligent agent. Within this construction, a vision-sensing modality can be broken down into multiple layers. The authors propose a human-skeleton-based trainable model for the recognition of spatiotemporal human worker activity using LSTM networks, which can achieve a training accuracy of 91.365%, based on the InHARD dataset. Together with the training results, results related to aspects of the simulation environment and future improvements of the system are discussed. By combining human worker upper body positions with actions, the perceptual potential of the system is increased, and human-robot collaboration becomes context-aware. Based on the acquired information, the intelligent agent gains the ability to adapt its behavior according to its dynamic and stochastic surroundings.

Sensor-Driven Human-Robot Synergy: A Systems Engineering Approach

Knowledge-based synergistic automation is a potential intermediate option between the opposite extremes of manual and fully automated robotic labor in agriculture. Disruptive information and communication technologies (ICT) and sophisticated solutions for human-robot interaction (HRI) endow a skilled farmer with enhanced capabilities to perform agricultural tasks more efficiently and productively. This research aspires to apply systems engineering principles to assess the design of a conceptual human-robot synergistic platform enabled by a sensor-driven ICT sub-system. In particular, this paper firstly presents an overview of a use case, including a human-robot synergistic platform comprising a drone, a mobile platform, and wearable equipment. The technology framework constitutes a paradigm of human-centric worker-robot logistics synergy for high-value crops, which is applicable in operational environments of outdoor in-field harvesting and handling operations. Except for the physical sub-system, the ICT sub-system of the robotic framework consists of an extended sensor network for enabling data acquisition to extract the context (e.g., worker's status, environment awareness) and plan and schedule the robotic agents of the framework. Secondly, this research explicitly presents the underpinning Design Structure Matrix (DSM) that systematically captures the interrelations between the sensors in the platform and data/information signals for enabling synergistic operations. The employed Systems Engineering approach provides a comprehensible analysis of the baseline structure existing in the examined human-robot synergy platform. In particular, the applied DSM allows for understanding and synthesizing a sensor sub-system's architecture and enriching its efficacy by informing targeted interventions and reconfiguring the developed robotic solution modules depending on the required farming tasks at an orchard. Human-centric solutions for the agrarian sector demand careful study of the features that the particular agri-field possesses; thus, the insight DSM provides to system designers can turn out to be useful in the investigation of other similar data-driven applications.

A Review on Human Comfort Factors, Measurements, and Improvements in Human-Robot Collaboration

As the development of robotics technologies for collaborative robots (COBOTs), the applications of human-robot collaboration (HRC) have been growing in the past decade. Despite the tremendous efforts from both academia and industry, the overall usage and acceptance of COBOTs are still not so high as expected. One of the major affecting factors is the comfort of humans in HRC, which is usually less emphasized in COBOT development; however, it is critical to the user acceptance during HRC. Therefore, this paper gives a review of human comfort in HRC including the influential factors of human comfort, measurement of human comfort in terms of subjective and objective manners, and human comfort improvement approaches in the context of HRC. Discussions on each topic are also conducted based on the review and analysis.

Potential Roles of Teamwork and Unmet Needs on Surgical Learning Curves of Spinal Robotic Screw Placement

Background

The aim of this study was to investigate the learning curve of robotic spine surgery quantitatively with the well-described power law of practice.

Methods

Kaohsiung Medical University Hospital set up a robotic spine surgery team by the neurosurgery department in 2013 and the orthopedic department joined the well-established team in 2014. A total of consecutive 150 cases received robotic assisted spinal surgery. The 150 cases, with 841 transpedicular screws were enrolled into 3 groups: the first 50 cases performed by neurosurgeons, the first 50 cases by orthopedic surgeons, and 50 cases by neurosurgeons after the orthopedic surgeons joined the team. The time per screw and accuracy by each group and individual surgeon were analyzed.

Results

The time per screw for each group was 9.56 ± 4.19, 7.29 ± 3.64, and 8.74 ± 5.77 minutes, respectively, with p-value 0.0017. The accuracy was 99.6% (253/254), 99.5% (361/363), and 99.1% (222/224), respectively, with p-value 0.77. Though the first group took time significantly more on per screw placement but without significance on the nonlinear parallelism F-test. Analysis of 5 surgeons and their first 10 cases of short segment surgery showed the time per screw by each surgeon was 12.28 ± 5.21, 6.38 ± 1.54, 8.68 ± 3.10, 6.33 ± 1.90, and 6.73 ± 1.81 minutes. The first surgeon who initiated the robotic spine surgery took significantly more time per screw, and the nonlinear parallelism test also revealed only the first surgeon had a steeper learning curve.

Conclusion

This is the first study to demonstrate that differences of learning curves between individual surgeons and teams. The roles of teamwork and the unmet needs due to lack of active perception are discussed.

Augmented Reality for Human-Robot Collaboration and Cooperation in Industrial Applications: A Systematic Literature Review

With the continuously growing usage of collaborative robots in industry, the need for achieving a seamless human-robot interaction has also increased, considering that it is a key factor towards reaching a more flexible, effective, and efficient production line. As a prominent and prospective tool to support the human operator to understand and interact with robots, Augmented Reality (AR) has been employed in numerous human-robot collaborative and cooperative industrial applications. Therefore, this systematic literature review critically appraises 32 papers' published between 2016 and 2021 to identify the main employed AR technologies, outline the current state of the art of augmented reality for human-robot collaboration and cooperation, and point out future developments for this research field. Results suggest that this is still an expanding research field, especially with the advent of recent advancements regarding head-mounted displays (HMDs). Moreover, projector-based and HMDs developed approaches are showing promising positive influences over operator-related aspects such as performance, task awareness, and safety feeling, even though HMDs need further maturation in ergonomic aspects. Further research should focus on large-scale assessment of the proposed solutions in industrial environments, involving the solution's target audience, and on establishing standards and guidelines for developing AR assistance systems.

My Caregiver the Cobot: Comparing Visualization Techniques to Effectively Communicate Cobot Perception to People with Physical Impairments

Nowadays, robots are found in a growing number of areas where they collaborate closely with humans. Enabled by lightweight materials and safety sensors, these cobots are gaining increasing popularity in domestic care, where they support people with physical impairments in their everyday lives. However, when cobots perform actions autonomously, it remains challenging for human collaborators to understand and predict their behavior, which is crucial for achieving trust and user acceptance. One significant aspect of predicting cobot behavior is understanding their perception and comprehending how they “see” the world. To tackle this challenge, we compared three different visualization techniques for Spatial Augmented Reality. All of these communicate cobot perception by visually indicating which objects in the cobot’s surrounding have been identified by their sensors. We compared the well-established visualizations Wedge and Halo against our proposed visualization Line in a remote user experiment with participants suffering from physical impairments. In a second remote experiment, we validated these findings with a broader non-specific user base. Our findings show that Line, a lower complexity visualization, results in significantly faster reaction times compared to Halo, and lower task load compared to both Wedge and Halo. Overall, users prefer Line as a more straightforward visualization. In Spatial Augmented Reality, with its known disadvantage of limited projection area size, established off-screen visualizations are not effective in communicating cobot perception and Line presents an easy-to-understand alternative.

Digital Twin-Driven Human Robot Collaboration Using a Digital Human

Advances are being made in applying digital twin (DT) and human–robot collaboration (HRC) to industrial fields for safe, effective, and flexible manufacturing. Using a DT for human modeling and simulation enables ergonomic assessment during working. In this study, a DT-driven HRC system was developed that measures the motions of a worker and simulates the working progress and physical load based on digital human (DH) technology. The proposed system contains virtual robot, DH, and production management modules that are integrated seamlessly via wireless communication. The virtual robot module contains the robot operating system and enables real-time control of the robot based on simulations in a virtual environment. The DH module measures and simulates the worker’s motion, behavior, and physical load. The production management module performs dynamic scheduling based on the predicted working progress under ergonomic constraints. The proposed system was applied to a parts-picking scenario, and its effectiveness was evaluated in terms of work monitoring, progress prediction, dynamic scheduling, and ergonomic assessment. This study demonstrates a proof-of-concept for introducing DH technology into DT-driven HRC for human-centered production systems.

A Human-Following Motion Planning and Control Scheme for Collaborative Robots Based on Human Motion Prediction

Human–Robot Interaction (HRI) for collaborative robots has become an active research topic recently. Collaborative robots assist human workers in their tasks and improve their efficiency. However, the worker should also feel safe and comfortable while interacting with the robot. In this paper, we propose a human-following motion planning and control scheme for a collaborative robot which supplies the necessary parts and tools to a worker in an assembly process in a factory. In our proposed scheme, a 3-D sensing system is employed to measure the skeletal data of the worker. At each sampling time of the sensing system, an optimal delivery position is estimated using the real-time worker data. At the same time, the future positions of the worker are predicted as probabilistic distributions. A Model Predictive Control (MPC)-based trajectory planner is used to calculate a robot trajectory that supplies the required parts and tools to the worker and follows the predicted future positions of the worker. We have installed our proposed scheme in a collaborative robot system with a 2-DOF planar manipulator. Experimental results show that the proposed scheme enables the robot to provide anytime assistance to a worker who is moving around in the workspace while ensuring the safety and comfort of the worker.

Human Cognition in Interaction With Robots: Taking the Robot's Perspective Into Account

OBJECTIVE

The present study investigated whether and how different human-robot interactions in a physically shared workspace influenced human stimulus-response (SR) relationships.

BACKGROUND

Human work is increasingly performed in interaction with advanced robots. Since human-robot interaction often takes place in physical proximity, it is crucial to investigate the effects of the robot on human cognition.

METHOD

In two experiments, we compared conditions in which humans interacted with a robot that they either remotely controlled or monitored under otherwise comparable conditions in the same shared workspace. The cognitive extent to which the participants took the robot's perspective served as a dependent variable and was evaluated with a SR compatibility task.

RESULTS

The results showed pronounced compatibility effects from the robot's perspective when participants had to take the perspective of the robot during the task, but significantly reduced compatibility effects when human and robot did not interact. In both experiments, compatibility effects from the robot's perspective resulted in statistically significant differences in response times and in error rates between compatible and incompatible conditions.

CONCLUSION

We concluded that SR relationships from the perspective of the robot need to be considered when designing shared workspaces that require users to take the perspective of the robot.

APPLICATION

The results indicate changed compatibility relationships when users share their workplace with an interacting robot and therefore have to take its perspective from time to time. The perspective-dependent processing times are expected to be accompanied by corresponding error rates, which might affect-for instance-safety and efficiency in a production process.

Human-Machine Differentiation in Speed and Separation Monitoring for Improved Efficiency in Human-Robot Collaboration

Human-robot collaborative applications have been receiving increasing attention in industrial applications. The efficiency of the applications is often quite low compared to traditional robotic applications without human interaction. Especially for applications that use speed and separation monitoring, there is potential to increase the efficiency with a cost-effective and easy to implement method. In this paper, we proposed to add human-machine differentiation to the speed and separation monitoring in human-robot collaborative applications. The formula for the protective separation distance was extended with a variable for the kind of object that approaches the robot. Different sensors for differentiation of human and non-human objects are presented. Thermal cameras are used to take measurements in a proof of concept. Through differentiation of human and non-human objects, it is possible to decrease the protective separation distance between the robot and the object and therefore increase the overall efficiency of the collaborative application.

Co-carrying an object by robot in cooperation with humans using visual and force sensing

Human-robot collaboration poses many challenges where humans and robots work inside a shared workspace. Robots collaborating with humans indirectly bring difficulties for accomplishing co-carrying tasks. In our work, we focus on co-carrying an object by robots in cooperation with humans using visual and force sensing. A framework using visual and force sensing is proposed for human-robot co-carrying tasks, enabling robots to actively cooperate with humans and reduce human efforts. Visual sensing for perceiving human motion is involved in admittance-based force control, and a hybrid controller combining visual servoing with force feedback is proposed which generates refined robot motion. The proposed framework is validated by a co-carrying task in experiments. There exist two phases in experimental processes: in Phase 1, the human hand holds one side of the box object, and the robot gripper of the Baxter robot automatically approaches to the other side of the box object and finally holds it; in Phase 2, the human and the Baxter robot co-carry the box object over a distance to different target positions. This article is part of the theme issue 'Towards symbiotic autonomous systems'.

Image Generation for 2D-CNN Using Time-Series Signal Features from Foot Gesture Applied to Select Cobot Operating Mode

Advances in robotics are part of reducing the burden associated with manufacturing tasks in workers. For example, the cobot could be used as a "third-arm" during the assembling task. Thus, the necessity of designing new intuitive control modalities arises. This paper presents a foot gesture approach centered on robot control constraints to switch between four operating modalities. This control scheme is based on raw data acquired by an instrumented insole located at a human's foot. It is composed of an inertial measurement unit (IMU) and four force sensors. Firstly, a gesture dictionary was proposed and, from data acquired, a set of 78 features was computed with a statistical approach, and later reduced to 3 via variance analysis ANOVA. Then, the time series collected data were converted into a 2D image and provided as an input for a 2D convolutional neural network (CNN) for the recognition of foot gestures. Every gesture was assimilated to a predefined cobot operating mode. The offline recognition rate appears to be highly dependent on the features to be considered and their spatial representation in 2D image. We achieve a higher recognition rate for a specific representation of features by sets of triangular and rectangular forms. These results were encouraging in the use of CNN to recognize foot gestures, which then will be associated with a command to control an industrial robot.

Intuitive Spatial Tactile Feedback for Better Awareness about Robot Trajectory during Human-Robot Collaboration

In this work, we extend the previously proposed approach of improving mutual perception during human-robot collaboration by communicating the robot's motion intentions and status to a human worker using hand-worn haptic feedback devices. The improvement is presented by introducing spatial tactile feedback, which provides the human worker with more intuitive information about the currently planned robot's trajectory, given its spatial configuration. The enhanced feedback devices communicate directional information through activation of six tactors spatially organised to represent an orthogonal coordinate frame: the vibration activates on the side of the feedback device that is closest to the future path of the robot. To test the effectiveness of the improved human-machine interface, two user studies were prepared and conducted. The first study aimed to quantitatively evaluate the ease of differentiating activation of individual tactors of the notification devices. The second user study aimed to assess the overall usability of the enhanced notification mode for improving human awareness about the planned trajectory of a robot. The results of the first experiment allowed to identify the tactors for which vibration intensity was most often confused by users. The results of the second experiment showed that the enhanced notification system allowed the participants to complete the task faster and, in general, improved user awareness of the robot's movement plan, according to both objective and subjective data. Moreover, the majority of participants (82%) favoured the improved notification system over its previous non-directional version and vision-based inspection.

Vision and RTLS Safety Implementation in an Experimental Human-Robot Collaboration Scenario

Human–robot collaboration is becoming ever more widespread in industry because of its adaptability. Conventional safety elements are used when converting a workplace into a collaborative one, although new technologies are becoming more widespread. This work proposes a safe robotic workplace that can adapt its operation and speed depending on the surrounding stimuli. The benefit lies in its use of promising technologies that combine safety and collaboration. Using a depth camera operating on the passive stereo principle, safety zones are created around the robotic workplace, while objects moving around the workplace are identified, including their distance from the robotic system. Passive stereo employs two colour streams that enable distance computation based on pixel shift. The colour stream is also used in the human identification process. Human identification is achieved using the Histogram of Oriented Gradients, pre-learned precisely for this purpose. The workplace also features autonomous trolleys for material supply. Unequivocal trolley identification is achieved using a real-time location system through tags placed on each trolley. The robotic workplace’s speed and the halting of its work depend on the positions of objects within safety zones. The entry of a trolley with an exception to a safety zone does not affect the workplace speed. This work simulates individual scenarios that may occur at a robotic workplace with an emphasis on compliance with safety measures. The novelty lies in the integration of a real-time location system into a vision-based safety system, which are not new technologies by themselves, but their interconnection to achieve exception handling in order to reduce downtimes in the collaborative robotic system is innovative.

Variable Admittance Control Based on Human-Robot Collaboration Observer Using Frequency Analysis for Sensitive and Safe Interaction

A collaborative robot should be sensitive to the user intention while maintaining safe interaction during tasks such as hand guiding. Observers based on the discrete Fourier transform have been studied to distinguish between the low-frequency motion elicited by the operator and high-frequency behavior resulting from system instability and disturbances. However, the discrete Fourier transform requires an excessively long sampling time. We propose a human–robot collaboration observer based on an infinite impulse response filter to increase the intention recognition speed. By using this observer, we also propose a variable admittance controller to ensure safe collaboration. The recognition speed of the human–robot collaboration observer is 0.29 s, being 3.5 times faster than frequency analysis based on the discrete Fourier transform. The performance of the variable admittance controller and its improved recognition speed are experimentally verified on a two-degrees-of-freedom manipulator. We confirm that the improved recognition speed of the proposed human–robot collaboration observer allows us to timely recover from unsafe to safe collaboration.

Development of Flexible Robot Skin for Safe and Natural Human⁻Robot Collaboration

For industrial manufacturing, industrial robots are required to work together with human counterparts on certain special occasions, where human workers share their skills with robots. Intuitive human–robot interaction brings increasing safety challenges, which can be properly addressed by using sensor-based active control technology. In this article, we designed and fabricated a three-dimensional flexible robot skin made by the piezoresistive nanocomposite based on the need for enhancement of the security performance of the collaborative robot. The robot skin endowed the YuMi robot with a tactile perception like human skin. The developed sensing unit in the robot skin showed the one-to-one correspondence between force input and resistance output (percentage change in impedance) in the range of 0–6.5 N. Furthermore, the calibration result indicated that the developed sensing unit is capable of offering a maximum force sensitivity (percentage change in impedance per Newton force) of 18.83% N⁻¹ when loaded with an external force of 6.5 N. The fabricated sensing unit showed good reproducibility after loading with cyclic force (0–5.5 N) under a frequency of 0.65 Hz for 3500 cycles. In addition, to suppress the bypass crosstalk in robot skin, we designed a readout circuit for sampling tactile data. Moreover, experiments were conducted to estimate the contact/collision force between the object and the robot in a real-time manner. The experiment results showed that the implemented robot skin can provide an efficient approach for natural and secure human–robot interaction.

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human-Robot Collaboration

Human-robot collaboration could be advanced by facilitating the intuitive, gaze-based control of robots, and enabling robots to recognize human actions, infer human intent, and plan actions that support human goals. Traditionally, gaze tracking approaches to action recognition have relied upon computer vision-based analyses of two-dimensional egocentric camera videos. The objective of this study was to identify useful features that can be extracted from three-dimensional (3D) gaze behavior and used as inputs to machine learning algorithms for human action recognition. We investigated human gaze behavior and gaze-object interactions in 3D during the performance of a bimanual, instrumental activity of daily living: the preparation of a powdered drink. A marker-based motion capture system and binocular eye tracker were used to reconstruct 3D gaze vectors and their intersection with 3D point clouds of objects being manipulated. Statistical analyses of gaze fixation duration and saccade size suggested that some actions (pouring and stirring) may require more visual attention than other actions (reach, pick up, set down, and move). 3D gaze saliency maps, generated with high spatial resolution for six subtasks, appeared to encode action-relevant information. The "gaze object sequence" was used to capture information about the identity of objects in concert with the temporal sequence in which the objects were visually regarded. Dynamic time warping barycentric averaging was used to create a population-based set of characteristic gaze object sequences that accounted for intra- and inter-subject variability. The gaze object sequence was used to demonstrate the feasibility of a simple action recognition algorithm that utilized a dynamic time warping Euclidean distance metric. Averaged over the six subtasks, the action recognition algorithm yielded an accuracy of 96.4%, precision of 89.5%, and recall of 89.2%. This level of performance suggests that the gaze object sequence is a promising feature for action recognition whose impact could be enhanced through the use of sophisticated machine learning classifiers and algorithmic improvements for real-time implementation. Robots capable of robust, real-time recognition of human actions during manipulation tasks could be used to improve quality of life in the home and quality of work in industrial environments.

A Hybrid Framework for Understanding and Predicting Human Reaching Motions

Robots collaborating naturally with a human partner in a confined workspace need to understand and predict human motions. For understanding, a model-based approach is required as the human motor control system relies on the biomechanical properties to control and execute actions. The model-based control models explain human motions descriptively, which in turn enables predicting and analyzing human movement behaviors. In motor control, reaching motions are framed as an optimization problem. However, different optimality criteria predict disparate motion behavior. Therefore, the inverse problem-finding the optimality criterion from a given arm motion trajectory-is not unique. This paper implements an inverse optimal control (IOC) approach to determine the combination of cost functions that governs a motion execution. The results indicate that reaching motions depend on a trade-off between kinematics and dynamics related cost functions. However, the computational efficiency is not sufficient for online prediction to be utilized for HRI. In order to predict human reaching motions with high efficiency and accuracy, we combine the IOC approach with a probabilistic movement primitives formulation. This hybrid model allows an online-capable prediction while taking into account motor variability and the interpersonal differences. The proposed framework affords a descriptive and a generative model of human reaching motions which can be effectively utilized online for human-in-the-loop robot control and task execution.