Model-Free Optimal Tracking Control of Nonlinear Input-Affine Discrete-Time Systems via an Iterative Deterministic Q-Learning Algorithm

Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis

Optimal learning output tracking control (OLOTC) in a model-free manner has received increasing attention in both the intelligent control and the reinforcement learning (RL) communities. Although the model-free tracking control has been achieved via off-policy learning and Q-learning, another popular RL idea of direct policy learning, with its easy-to-implement feature, is still rarely considered. To fill this gap, this article aims to develop a novel model-free policy optimization (PO) algorithm to achieve the OLOTC for unknown linear discrete-time (DT) systems. The iterative control policy is parameterized to directly improve the discounted value function of the augmented system via the gradient-based method. To implement this algorithm in a model-free manner, a model-free two-point policy gradient (PG) algorithm is designed to approximate the gradient of discounted value function by virtue of the sampled states and the reference trajectories. The global convergence of model-free PO algorithm to the optimal value function is demonstrated with the sufficient quantity of samples and proper conditions. Finally, numerical simulation results are provided to validate the effectiveness of the present method.

Self-triggered neural tracking control for discrete-time nonlinear systems via adaptive critic learning

In this paper, a novel self-triggered optimal tracking control method is developed based on the online action-critic technique for discrete-time nonlinear systems. First, an augmented plant is constructed by integrating the system state with the reference trajectory. This transformation redefines the optimal tracking control design as the optimal regulation issue of the reconstructed nonlinear error system. Subsequently, under the premise of ensuring the controlled system stability, a self-sampling function that depends solely on the sampling tracking error is devised, thereby determining the next triggering instant. This approach not only effectively reduces the computational burden but also eliminates the need for continuous evaluation of the triggering condition, as required in traditional event-based methods. Furthermore, the developed control method can be found to possess excellent triggering performance. The model, critic, and action neural networks are constructed to implement the online critic learning algorithm, enabling real-time adjustment of the tracking control policy to achieve optimal performance. Finally, an experimental plant with nonlinear characteristics is presented to illustrate the overall performance of the proposed online self-triggered tracking control strategy.

Data-Driven Optimal Tracking Control for Discrete-Time Nonlinear Systems With Unknown Dynamics Using Deterministic ADP

This article aims to solve the optimal tracking problem (OTP) for a class of discrete-time (DT) nonlinear systems with completely unknown dynamics. A novel data-driven deterministic approximate dynamic programming (ADP) algorithm is proposed to solve this kind of problem with only input-output (I/O) data. The proposed algorithm has two advantages compared to existing data-driven deterministic ADP algorithms for the OTP. First, our algorithm can guarantee optimality while achieving better performance in the aspects of time-saving and robustness to data. Second, the near-optimal control policy learned by our algorithm can be implemented without considering expected control and enable the system states to track the user-specified reference signals. Therefore, the tracking performance is guaranteed while simplifying the algorithm implementation. Furthermore, the convergence and stability of the proposed algorithm are strictly proved through theoretical analysis, in which the errors caused by neural networks (NNs) are considered. At the end of this article, the developed algorithm is compared with two representative deterministic ADP algorithms through a numerical example and applied to solve the tracking problem for a two-link robotic manipulator. The simulation results demonstrate the effectiveness and advantages of the developed algorithm.