Design and Experimental Validation of Deep Reinforcement Learning-Based Fast Trajectory Planning and Control for Mobile Robot in Unknown Environment

Rope on Rope: Reducing Residual Vibrations in Rope-Based Anchoring System and Rope-Driven Façade Operation Robot

Maintenance of the exteriors of buildings with convex façades, such as skyscrapers, is in high demand in urban centers. However, manual maintenance is inherently dangerous due to the possibility of accidental falls. Therefore, research has been conducted on cleaning robots as a replacement for human workers, e.g., the dual ascension robot (DAR), which is an underactuated rope-driven robot, and the rope-riding mobile anchor (RMA), which is a rope-riding robot. These robots are equipped with a convex-façade-cleaning system. The DAR and RMA are connected to each other by a rope that enables vibration transmission between them. It also increases the instability of the residual vibration that occurs during the operation of the DAR. This study focused on reducing the residual vibrations of a DAR to improve the stability of the overall system. Because it is a rope-on-rope (ROR) system, we assumed it to be a simplified serial spring-damper system and analyzed its kinematics and dynamics. An input-shaping technique was applied to control the residual vibrations in the DAR. We also applied a disturbance observer to mitigate factors contributing to the system uncertainty, such as rope deformation, slip, and external forces. We experimentally validated the system and assessed the effectiveness of the control method, which consisted of the input shaper and disturbance observer. Consequently, the residual vibrations were reduced.

Efficiency-Driven Adaptive Task Planning for Household Robot Based on Hierarchical Item-Environment Cognition

Task planning focused on household robots represents a conventional yet complex research domain, necessitating the development of task plans that enable robots to execute unfamiliar household services. This area has garnered significant research interest due to its extensive applications in robotics, particularly concerning household robots. Nevertheless, the majority of task planning methodologies exhibit suboptimal performance regarding the success and efficiency of completing household tasks, primarily due to a lack of cognitive capacity of household items and home environments. To address these challenges, we propose an efficiency-driven adaptive task planning approach based on hierarchical item-environment cognition. Initially, we establish a multiple semantic attribute-based priori knowledge (MSAPK) framework to facilitate the attributive representation of household items. Utilizing MSAPK, we develop a long short-term memory (LSTM) based item cognition model that assigns relevant attributes and substitutes to specified household items, thereby enhancing the cognitive capabilities of household robots at the attribute level. Subsequently, we construct an environment cognition model that delineates the relationships between household items and room types, enabling household robots to locate target items more efficiently. Through hierarchical item-environment cognition, we introduce a strategy for adaptive task planning, empowering household robots to execute household tasks with both flexibility and efficiency. The generated plans are evaluated in both virtual and real-world experiments, with promising results affirming the effectiveness of our proposed methodology.

Universal Approximation Abilities of a Modular Differentiable Neural Network

Approximation ability is one of the most important topics in the field of neural networks (NNs). Feedforward NNs, activated by rectified linear units and some of their specific smoothed versions, provide universal approximators to convex as well as continuous functions. However, most of these networks are investigated empirically, or their characteristics are analyzed based on specific operation rules. Moreover, an adequate level of interpretability of the networks is missing as well. In this work, we propose a class of new network architecture, built with reusable neural modules (functional blocks), to supply differentiable and interpretable approximators for convex and continuous target functions. Specifically, first, we introduce a concrete model construction mechanism with particular blocks based on differentiable programming and the composition essence of the max operator, extending the scope of existing activation functions. Moreover, explicit block diagrams are provided for a clear understanding of the external architecture and the internal processing mechanism. Subsequently, the approximation behavior of the proposed network to convex functions and continuous functions is rigorously proved as well, by virtue of mathematical induction. Finally, plenty of numerical experiments are conducted on a wide variety of problems, which exhibit the effectiveness and the superiority of the proposed model over some existing ones.

Pontryagin's Minimum Principle-Guided RL for Minimum-Time Exploration of Spatiotemporal Fields

This article studies the trajectory planning problem of an autonomous vehicle for exploring a spatiotemporal field subject to a constraint on cumulative information. Since the resulting problem depends on the signal strength distribution of the field, which is unknown in practice, we advocate the use of a model-free reinforcement learning (RL) method to find the solution. Given the vehicle's dynamical model, a critical (and open) question is how to judiciously merge the model-based optimality conditions into the model-free RL framework for improved efficiency and generalization, for which this work provides some positive results. Specifically, we discretize the continuous action space by leveraging analytic optimality conditions for the minimum-time optimization problem via Pontryagin's minimum principle (PMP). This allows us to develop a novel discrete PMP-based RL trajectory planning algorithm, which learns a planning policy faster than those based on a continuous action space. Simulation results: 1) validate the effectiveness of the PMP-based RL algorithm and 2) demonstrate its advantages, in terms of both learning efficiency and the vehicle's exploration time, over two baseline methods for continuous control inputs.

Deep Reinforcement Learning With Explicit Context Representation

Though reinforcement learning (RL) has shown an outstanding capability for solving complex computational problems, most RL algorithms lack an explicit method that would allow learning from contextual information. On the other hand, humans often use context to identify patterns and relations among elements in the environment, along with how to avoid making wrong actions. However, what may seem like an obviously wrong decision from a human perspective could take hundreds of steps for an RL agent to learn to avoid. This article proposes a framework for discrete environments called Iota explicit context representation (IECR). The framework involves representing each state using contextual key frames (CKFs), which can then be used to extract a function that represents the affordances of the state; in addition, two loss functions are introduced with respect to the affordances of the state. The novelty of the IECR framework lies in its capacity to extract contextual information from the environment and learn from the CKFs' representation. We validate the framework by developing four new algorithms that learn using context: Iota deep Q-network (IDQN), Iota double deep Q-network (IDDQN), Iota dueling deep Q-network (IDuDQN), and Iota dueling double deep Q-network (IDDDQN). Furthermore, we evaluate the framework and the new algorithms in five discrete environments. We show that all the algorithms, which use contextual information, converge in around 40000 training steps of the neural networks, significantly outperforming their state-of-the-art equivalents.

A Dual-Mode Robot-Assisted Plate Implantation Method for Femoral Shaft Fracture

BACKGROUND

Minimally invasive internal fixation is the preferred treatment option for femoral shaft fractures. However, there are problems such as invisibility, inaccuracy and instability in the process of plate implantation.

METHODS

In this paper, a dual-mode robot-assisted plate implantation method was proposed by combining a starting point determination algorithm, motion capture, deep learning and robotics.

RESULTS

The neural network model planned the plate implantation trajectory according to patient's condition. Then, the advantages of high stability and high precision of the robotic arm were used for plate implantation.

CONCLUSION

The trend and fluctuation of the plate implantation trajectories obtained using this method met clinical requirements. Furthermore, the robotic arm implantation process was safe.

Representation Learning and Reinforcement Learning for Dynamic Complex Motion Planning System

Indoor motion planning challenges researchers because of the high density and unpredictability of moving obstacles. Classical algorithms work well in the case of static obstacles but suffer from collisions in the case of dense and dynamic obstacles. Recent reinforcement learning (RL) algorithms provide safe solutions for multiagent robotic motion planning systems. However, these algorithms face challenges in convergence: slow convergence speed and suboptimal converged result. Inspired by RL and representation learning, we introduced the ALN-DSAC: a hybrid motion planning algorithm where attention-based long short-term memory (LSTM) and novel data replay combine with discrete soft actor-critic (SAC). First, we implemented a discrete SAC algorithm, which is the SAC in the setting of discrete action space. Second, we optimized existing distance-based LSTM encoding by attention-based encoding to improve the data quality. Third, we introduced a novel data replay method by combining the online learning and offline learning to improve the efficacy of data replay. The convergence of our ALN-DSAC outperforms that of the trainable state of the arts. Evaluations demonstrate that our algorithm achieves nearly 100% success with less time to reach the goal in motion planning tasks when compared to the state of the arts. The test code is available at https://github.com/CHUENGMINCHOU/ALN-DSAC.

A deterministic robust control with parameter optimization for uncertain two-wheel driven mobile robot

The uncertainty in mobile robot greatly affects control accuracy. This makes it difficult to apply to more rigorous high-precision engineering fields. Therefore, the fuzzy set theory is introduced to describe the uncertainty. Based on that, the fuzzy mobile robot system is established. The virtual speed controller using backstepping method is designed. Then, a robust control method is proposed to guarantee the uniform boundedness and uniform ultimate boundedness of the controlled system. Furthermore, the balance optimization problem of the performance and cost of the controlled system is explored. By minimizing the performance index containing fuzzy numbers, the optimal control parameter is obtained. Compared with the linear quadratic regulator algorithm, which is the representative optimal robust controller, the proposed control method and optimization strategy based on fuzzy set theory are verified to be effective. The control accuracy is further improved.

Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation

Reinforcement learning (RL) is a promising approach in unmanned ground vehicles (UGVs) applications, but limited computing resource makes it challenging to deploy a well-behaved RL strategy with sophisticated neural networks. Meanwhile, the training of RL on navigation tasks is difficult, which requires a carefully-designed reward function and a large number of interactions, yet RL navigation can still fail due to many corner cases. This shows the limited intelligence of current RL methods, thereby prompting us to rethink combining RL with human intelligence. In this paper, a human-guided RL framework is proposed to improve RL performance both during learning in the simulator and deployment in the real world. The framework allows humans to intervene in RL's control progress and provide demonstrations as needed, thereby improving RL's capabilities. An innovative human-guided RL algorithm is proposed that utilizes a series of mechanisms to improve the effectiveness of human guidance, including human-guided learning objective, prioritized human experience replay, and human intervention-based reward shaping. Our RL method is trained in simulation and then transferred to the real world, and we develop a denoised representation for domain adaptation to mitigate the simulation-to-real gap. Our method is validated through simulations and real-world experiments to navigate UGVs in diverse and dynamic environments based only on tiny neural networks and image inputs. Our method performs better in goal-reaching and safety than existing learning- and model-based navigation approaches and is robust to changes in input features and ego kinetics. Furthermore, our method allows small-scale human demonstrations to be used to improve the trained RL agent and learn expected behaviors online.

End-to-End Autonomous Navigation Based on Deep Reinforcement Learning with a Survival Penalty Function

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor-critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a nonholonomic wheeled mobile robot (WMR) to perform navigation in dynamic environments containing obstacles and for which no maps are available. A comprehensive reward based on the survival penalty function is introduced; this approach effectively solves the sparse reward problem and enables the WMR to move toward its target. Consecutive episodes are connected to increase the cumulative penalty for scenarios involving obstacles; this method prevents training failure and enables the WMR to plan a collision-free path. Simulations are conducted for four scenarios-movement in an obstacle-free space, in a parking lot, at an intersection without and with a central obstacle, and in a multiple obstacle space-to demonstrate the efficiency and operational safety of our method. For the same navigation environment, compared with the DDPG algorithm, the TD3 algorithm exhibits faster numerical convergence and higher stability in the training phase, as well as a higher task execution success rate in the evaluation phase.

Robust cooperative output regulation for heterogeneous nonlinear multi-agent systems with an unknown exosystem subject to jointly connected switching networks

This paper investigates the robust cooperative output regulation problem for heterogeneous lower triangular nonlinear multi-agent systems with an unknown exosystem over jointly connected switching networks. The problem has been studied for the exactly known exosystem over switching networks. However, the existing result for the unknown exosystem is still limited to the static networks. To ensure that all followers acquire the reference trajectory generated by the unknown exosystem through the jointly connected switching networks, by combining a set of auxiliary filtering variables and fixed-time stability theory, an adaptive distributed observer is designed. On the basis of the adaptive distributed observer and the distributed internal model approach, we propose a distributed controller under several standard assumptions to solve the problem. Compared with the similar work subject to the static networks, the controller in this paper is applicable to the more general communication network while weakening the assumptions of the controlled system.