Data-Driven Performance-Prescribed Reinforcement Learning Control of an Unmanned Surface Vehicle

An Information-Assisted Deep Reinforcement Learning Path Planning Scheme for Dynamic and Unknown Underwater Environment

An autonomous underwater vehicle (AUV) has shown impressive potential and promising exploitation prospects in numerous marine missions. Among its various applications, the most essential prerequisite is path planning. Although considerable endeavors have been made, there are several limitations. A complete and realistic ocean simulation environment is critically needed. As most of the existing methods are based on mathematical models, they suffer from a large gap with reality. At the same time, the dynamic and unknown environment places high demands on robustness and generalization. In order to overcome these limitations, we propose an information-assisted reinforcement learning path planning scheme. First, it performs numerical modeling based on real ocean current observations to establish a complete simulation environment with the grid method, including 3-D terrain, dynamic currents, local information, and so on. Next, we propose an information compression (IC) scheme to trim the mutual information (MI) between reinforcement learning neural network layers to improve generalization. A proof based on information theory provides solid support for this. Moreover, for the dynamic characteristics of the marine environment, we elaborately design a confidence evaluator (CE), which evaluates the correlation between two adjacent frames of ocean currents to provide confidence for the action. The performance of our method has been evaluated and proven by numerical results, which demonstrate a fair sensitivity to ocean currents and high robustness and generalization to cope with the dynamic and unknown underwater environment.

ADHDP-based robust self-learning 3D trajectory tracking control for underactuated UUVs

In this work, we propose a robust self-learning control scheme based on action-dependent heuristic dynamic programming (ADHDP) to tackle the 3D trajectory tracking control problem of underactuated uncrewed underwater vehicles (UUVs) with uncertain dynamics and time-varying ocean disturbances. Initially, the radial basis function neural network is introduced to convert the compound uncertain element, comprising uncertain dynamics and time-varying ocean disturbances, into a linear parametric form with just one unknown parameter. Then, to improve the tracking performance of the UUVs trajectory tracking closed-loop control system, an actor-critic neural network structure based on ADHDP technology is introduced to adaptively adjust the weights of the action-critic network, optimizing the performance index function. Finally, an ADHDP-based robust self-learning control scheme is constructed, which makes the UUVs closed-loop system have good robustness and control performance. The theoretical analysis demonstrates that all signals in the UUVs trajectory tracking closed-loop control system are bounded. The simulation results for the UUVs validate the effectiveness of the proposed control scheme.

Discrete-Time Reinforcement Learning Adaptive Control for Non-Gaussian Distribution of Sampling Intervals

This article proposes an optimal controller based on reinforcement learning (RL) for a class of unknown discrete-time systems with non-Gaussian distribution of sampling intervals. The critic and actor networks are implemented using the MiFRENc and MiFRENa architectures, respectively. The learning algorithm is developed with learning rates determined through convergence analysis of internal signals and tracking errors. Experimental systems with a comparative controller are conducted to validate the proposed scheme, and comparative results show superior performance for non-Gaussian distributions, with weight transfer for the critic network omitted. Additionally, the proposed learning laws, using the estimated co-state, significantly improve dead-zone compensation and nonlinear variation.

Adaptive Optimal Tracking Control of an Underactuated Surface Vessel Using Actor-Critic Reinforcement Learning

In this article, we present an adaptive reinforcement learning optimal tracking control (RLOTC) algorithm for an underactuated surface vessel subject to modeling uncertainties and time-varying external disturbances. By integrating backstepping technique with the optimized control design, we show that the desired optimal tracking performance of vessel control is guaranteed due to the fact that the virtual and actual control inputs are designed as optimized solutions of every subsystem. To enhance the robustness of vessel control systems, we employ neural network (NN) approximators to approximate uncertain vessel dynamics and present adaptive control technique to estimate the upper boundedness of external disturbances. Under the reinforcement learning framework, we construct actor-critic networks to solve the Hamilton-Jacobi-Bellman equations corresponding to subsystems of surface vessel to achieve the optimized control. The optimized control algorithm can synchronously train the adaptive parameters not only for actor-critic networks but also for NN approximators and adaptive control. By Lyapunov stability theorem, we show that the RLOTC algorithm can ensure the semiglobal uniform ultimate boundedness of the closed-loop systems. Compared with the existing reinforcement learning control results, the presented RLOTC algorithm can compensate for uncertain vessel dynamics and unknown disturbances, and obtain the optimized control performance by considering optimization in every backstepping design. Simulation studies on an underactuated surface vessel are given to illustrate the effectiveness of the RLOTC algorithm.

Real-time Trajectory Planning and Tracking Control of Bionic Underwater Robot in Dynamic Environment

In this article, we study the trajectory planning and tracking control of a bionic underwater robot under multiple dynamic obstacles. We first introduce the design of the bionic leopard cabinet underwater robot developed in our lab. Then, we model the trajectory planning problem of the bionic underwater robot by combining its dynamics and physical constraints. Furthermore, we conduct global trajectory planning for bionic underwater robots based on the temporal-spatial Bezier curves. In addition, based on the improved proximal policy optimization, local dynamic obstacle avoidance trajectory replanning is carried out. In addition, we design the fuzzy proportional-integral-derivative controller for tracking control of the planned trajectory. Finally, the effectiveness of the real-time trajectory planning and tracking control method is verified by comparative simulation in dynamic environment and semiphysical simulation of UWSim. Among them, the real-time trajectory planning method has advantages in trajectory length, trajectory smoothness, and planning time. The error of trajectory tracking control method is controlled around 0.2 m.

Reinforcement Learning-Based Fractional-Order Adaptive Fault-Tolerant Formation Control of Networked Fixed-Wing UAVs With Prescribed Performance

This article investigates the fault-tolerant formation control (FTFC) problem for networked fixed-wing unmanned aerial vehicles (UAVs) against faults. To constrain the distributed tracking errors of follower UAVs with respect to neighboring UAVs in the presence of faults, finite-time prescribed performance functions (PPFs) are developed to transform the distributed tracking errors into a new set of errors by incorporating user-specified transient and steady-state requirements. Then, the critic neural networks (NNs) are developed to learn the long-term performance indices, which are used to evaluate the distributed tracking performance. Based on the generated critic NNs, actor NNs are designed to learn the unknown nonlinear terms. Moreover, to compensate for the reinforcement learning errors of actor-critic NNs, nonlinear disturbance observers (DOs) with skillfully constructed auxiliary learning errors are developed to facilitate the FTFC design. Furthermore, by using the Lyapunov stability analysis, it is shown that all follower UAVs can track the leader UAV with predesigned offsets, and the distributed tracking errors are finite-time convergent. Finally, comparative simulation results are presented to show the effectiveness of the proposed control scheme.

Design of a reinforcement learning-based intelligent car transfer planning system for parking lots

In this study, a car transfer planning system for parking lots was designed based on reinforcement learning. The car transfer planning system for parking lots is an intelligent parking management system that is designed by using reinforcement learning techniques. The system features autonomous decision-making, intelligent path planning and efficient resource utilization. And the problem is solved by constructing a Markov decision process and using a dynamic planning-based reinforcement learning algorithm. The system has the advantage of looking to the future and using reinforcement learning to maximize its expected returns. And this is in contrast to manual transfer planning which relies on traditional thinking. In the context of this paper on parking lots, the states of the two locations form a finite set. The system ultimately seeks to find a strategy that is beneficial to the long-term development of the operation. It aims to prioritize strategies that have positive impacts in the future, rather than those that are focused solely on short-term benefits. To evaluate strategies, as its basis the system relies on the expected return of a state from now to the future. This approach allows for a more comprehensive assessment of the potential outcomes and ensures the selection of strategies that align with long-term goals. Experimental results show that the system has high performance and robustness in the area of car transfer planning for parking lots. By using reinforcement learning techniques, parking lot management systems can make autonomous decisions and plan optimal paths to achieve efficient resource utilization and reduce parking time.

Robust Actor-Critic With Relative Entropy Regulating Actor

The accurate estimation of Q-function and the enhancement of agent's exploration ability have always been challenges of off-policy actor-critic algorithms. To address the two concerns, a novel robust actor-critic (RAC) is developed in this article. We first derive a robust policy improvement mechanism (RPIM) by using the local optimal policy about the current estimated Q-function to guide policy improvement. By constraining the relative entropy between the new policy and the previous one in policy improvement, the proposed RPIM can enhance the stability of the policy update process. The theoretical analysis shows that the incentive to increase the policy entropy is endowed when the policy is updated, which is conducive to enhancing the exploration ability of agents. Then, RAC is developed by applying the proposed RPIM to regulate the actor improvement process. The developed RAC is proven to be convergent. Finally, the proposed RAC is evaluated on some continuous-action control tasks in the MuJoCo platform and the experimental results show that RAC outperforms several state-of-the-art reinforcement learning algorithms.

Adaptive optimal trajectory tracking control of AUVs based on reinforcement learning

In this paper, an adaptive model-free optimal reinforcement learning (RL) neural network (NN) control scheme based on filter error is proposed for the trajectory tracking control problem of an autonomous underwater vehicle (AUV) with input saturation. Generally, the optimal control is realized by solving the Hamilton-Jacobi-Bellman (HJB) equation. However, due to its inherent nonlinearity and complexity, the HJB equation of AUV dynamics is challenging to solve. To deal with this problem, an RL strategy based on an actor-critic framework is proposed to approximate the solution of the HJB equation, where actor and critic NNs are used to perform control behavior and evaluate control performance, respectively. In addition, for the AUV system with the second-order strict-feedback dynamic model, the optimal controller design method based on filtering errors is proposed for the first time to simplify the controller design and accelerate the response speed of the system. Then, to solve the model-dependent problem, an extended state observer (ESO) is designed to estimate the unknown nonlinear dynamics, and an adaptive law is designed to estimate the unknown model parameters. To deal with the input saturation, an auxiliary variable system is utilized in the control law. The strict Lyapunov analysis guarantees that all signals of the system are semi-global uniformly ultimately bounded (SGUUB). Finally, the superiority of the proposed method is verified by comparative experiments.

Development of a Sliding-Mode-Control-Based Path-Tracking Algorithm with Model-Free Adaptive Feedback Action for Autonomous Vehicles

This paper presents a sliding mode control (SMC)-based path-tracking algorithm for autonomous vehicles by considering model-free adaptive feedback actions. In autonomous vehicles, safe path tracking requires adaptive and robust control algorithms because driving environment and vehicle conditions vary in real time. In this study, the SMC was adopted as a robust control method to adjust the switching gain, taking into account the sliding surface and unknown uncertainty to make the control error zero. The sliding surface can be designed mathematically, but it is difficult to express the unknown uncertainty mathematically. Information of priori bounded uncertainties is needed to obtain closed-loop stability of the control system, and the unknown uncertainty can vary with changes in internal and external factors. In the literature, ongoing efforts have been made to overcome the limitation of losing control stability due to unknown uncertainty. This study proposes an integrated method of adaptive feedback control (AFC) and SMC that can adjust a bounded uncertainty. Some illustrative and representative examples, such as autonomous driving scenarios, are also provided to show the main properties of the designed integrated controller. The examples show superior control performance, and it is expected that the integrated controller could be widely used for the path-tracking algorithms of autonomous vehicles.

Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels

This article considers a problem of tracking, convergence of disturbance observer (DO) based optimal control design for uncertain surface vessels (SVs) with external disturbance. The advantage of proposed optimal control using adaptive/approximate reinforcement learning (ARL) is that consideration for whole SVs with only one dynamic equation and without conventional separation technique. Additionally, thanks to appropriate disturbance observer, the attraction region of tracking error is remarkably reduced. On the other hand, the particular case of optimal control problem is presented by directly solving for the purpose of choosing the suitable activation functions of ARL. Furthermore, the proposed ARL based optimal control also deals with non-autonomous property of closed tracking error SV model by considering the equivalent system. Based on the Lyapunov function candidate using optimal function and quadratic form of estimated error of actor/critic weight, the stability and convergence of the closed system are proven. Some examples are given to verify and demonstrate the effectiveness of the new control strategy.

Reinforcement-Learning-Based Tracking Control with Fixed-Time Prescribed Performance for Reusable Launch Vehicle under Input Constraints

This paper proposes a novel reinforcement learning (RL)-based tracking control scheme with fixed-time prescribed performance for a reusable launch vehicle subject to parametric uncertainties, external disturbances, and input constraints. First, a fixed-time prescribed performance function is employed to restrain attitude tracking errors, and an equivalent unconstrained system is derived via an error transformation technique. Then, a hyperbolic tangent function is incorporated into the optimal performance index of the unconstrained system to tackle the input constraints. Subsequently, an actor-critic RL framework with super-twisting-like sliding mode control is constructed to establish a practical solution for the optimal control problem. Benefiting from the proposed scheme, the robustness of the RL-based controller against unknown dynamics is enhanced, and the control performance can be qualitatively prearranged by users. Theoretical analysis shows that the attitude tracking errors converge to a preset region within a preassigned fixed time, and the weight estimation errors of the actor-critic networks are uniformly ultimately bounded. Finally, comparative numerical simulation results are provided to illustrate the effectiveness and improved performance of the proposed control scheme.

A Novel Fixed-Time Trajectory Tracking Strategy of Unmanned Surface Vessel Based on the Fractional Sliding Mode Control Method

A novel sliding mode control method is proposed to achieve the trajectory tracking of the Unmanned Surface Vessel (USV) and effectively deal with the unmodeled dynamics and external unknown disturbances. First, a fixed-time fractional-order sliding mode control (FTFOSMC) strategy is proposed, combined with the fixed-time control theory and fractional-order control theory based on the sliding mode control method. The FTFOSMC strategy can improve the convergence velocity of the system, and effectively track the desired path, weakening the “chattering” effect in sliding mode control systems. Second, a fixed-time fractional-order sliding mode control strategy combined with the radial basis function neural network (RBF-FTFOSMC) was designed, which can effectively estimate the lumped uncertainties, such as the disturbance of external wind, wave, and current, and the unmodeled dynamics of the USV model. Then, the stability and effectiveness of the designed control strategy are guaranteed by the Lyapunov theory and the corresponding lemmas. Finally, a rigorous simulation experiment is designed to validate the effectiveness and stability of the proposed control strategy. The simulation results show that the control strategy can effectively achieve trajectory tracking of the USV, reduce the “chattering” phenomenon of sliding mode, and effectively estimate the lumped uncertainties.