Neural Network Learning and Robust Stabilization of Nonlinear Systems With Dynamic Uncertainties

Robust Approximate Dynamic Programming for Nonlinear Systems With Both Model Error and External Disturbance

Model error and external disturbance have been separately addressed by optimizing the definite performance in standard linear control problems. However, the concurrent handling of both introduces uncertainty and nonconvexity into the performance, posing a huge challenge for solving nonlinear problems. This article introduces an additional cost function in the augmented Hamilton-Jacobi-Isaacs (HJI) equation of zero-sum games to simultaneously manage the model error and external disturbance in nonlinear robust performance problems. For satisfying the Hamilton-Jacobi inequality in nonlinear robust control theory under all considered model errors, the relationship between the additional cost function and model uncertainty is revealed. A critic online learning algorithm, applying Lyapunov stabilizing terms and historical states to reinforce training stability and achieve persistent learning, is proposed to approximate the solution of the augmented HJI equation. By constructing a joint Lyapunov candidate about the critic weight and system state, both stability and convergence are proved by the second method of Lyapunov. Theoretical results also show that introducing historical data reduces the ultimate bounds of system state and critic error. Three numerical examples are conducted to demonstrate the effectiveness of the proposed method.

Evolutionary Dynamics Optimal Research-Oriented Tumor Immunity Architecture

The article is devoted to evolutionary dynamics optimal control-oriented tumor immune differential game system. First, the mathematical model covering immune cells and tumor cells considering the effects of chemotherapy drugs and immune agents. Second, the bounded optimal control problem covering is transformed into solving Hamilton-Jacobi-Bellman (HJB) equation considering the actual constraints and infinite-horizon performance index based on minimizing the amount of medication administered. Finally, approximate optimal control strategy is acquired through iterative-dual heuristic dynamic programming (I-DHP) algorithm avoiding dimensional disaster effectively and providing optimal treatment scheme for clinical applications.

Robust Admittance Control of Optimized Robot-Environment Interaction Using Reference Adaptation

In this article, a robust control scheme is proposed for robots to achieve an optimal performance in the process of interacting with external forces from environments. The environmental dynamics are defined as a linear model, and the interaction performance is evaluated by a defined cost function, which is composed of trajectory errors and force regulation. Based on admittance control, the reference adaptation method is used to minimize the cost function and achieve the optimal interaction performance. To make the trajectory tracking controller robust to the unknown disturbance of internal system dynamics, an auxiliary system is defined and the approximation optimal controller is designed. Experiments on the Baxter robot are conducted to verify the effectiveness of the proposed method.

Robust Near-Optimal Coordination in Uncertain Multiagent Networks With Motion Constraints

This article addresses the robust coordination problem for nonlinear uncertain second-order multiagent networks with motion constraints, including velocity saturation and collision avoidance. A single-critic neural network-based approximate dynamic programming approach and exact estimation of unknown dynamics are employed to learn online the optimal value function and controller. By incorporating avoidance penalties into tracking variable, constructing a novel value function, and designing of suitable learning algorithms, multiagent coordination and collision avoidance are achieved simultaneously. We show that the developed feedback-based coordination strategy guarantees uniformly ultimately bounded convergence of the closed-loop dynamical stability and all underlying motion constraints are always strictly obeyed. The effectiveness of the proposed collision-free coordination law is finally illustrated using numerical simulations.

Event-Triggered Adaptive Neural Network Control for Stochastic Nonlinear Systems With State Constraints and Time-Varying Delays

In this article, we pay attention to develop an event-triggered adaptive neural network (ANN) control strategy for stochastic nonlinear systems with state constraints and time-varying delays. The state constraints are disposed by relying on the barrier Lyapunov function. The neural networks are exploited to identify the unknown dynamics. In addition, the Lyapunov-Krasovskii functional is employed to counteract the adverse effect originating from time-varying delays. The backstepping technique is employed to design controller by combining event-triggered mechanism (ETM), which can alleviate data transmission and save communication resource. The constructed ANN control scheme can guarantee the stability of the considered systems, and the predefined constraints are not violated. Simulation results and comparison are given to validate the feasibility of the presented scheme.

Time-/Event-Triggered Adaptive Neural Asymptotic Tracking Control for Nonlinear Systems With Full-State Constraints and Application to a Single-Link Robot

This study proposes the time-/event-triggered adaptive neural control strategies for the asymptotic tracking problem of a class of uncertain nonlinear systems with full-state constraints. First, we design a time-triggered strategy. The effect caused by the residuals of the estimation via radial basis function (RBF) neural networks (NNs), and the reasonable upper bounds on the first derivative of the reference signal and the derivative of each virtual control, can be eliminated by designing appropriate adaptive laws and utilizing the basic properties of RBF NNs. Moreover, the construction of the barrier Lyapunov functions (BLFs) in this work ensures the compliance of the full-state constraints and also holds the asymptotic output tracking performance. Then, based on the time-triggered strategy, we further design a relative threshold event-triggered strategy. The proposed event-triggered adaptive neural controller can solve the main control objective of this work, that is: 1) the full-state constraint requirements of the system are not violated and 2) the output signal asymptotically tracks the reference signal. Compared with the traditional method, the event-triggered strategy can improve the utilization of communication channels and resources and has greater practical significance. Finally, an example of single-link robot under the proposed two strategies illustrates the validity of the constructed controllers.

Decentralized Event-Driven Constrained Control Using Adaptive Critic Designs

We study the decentralized event-driven control problem of nonlinear dynamical systems with mismatched interconnections and asymmetric input constraints. To begin with, by introducing a discounted cost function for each auxiliary subsystem, we transform the decentralized event-driven constrained control problem into a group of nonlinear H₂ -constrained optimal control problems. Then, we develop the event-driven Hamilton-Jacobi-Bellman equations (ED-HJBEs), which arise in the nonlinear H₂ -constrained optimal control problems. Meanwhile, we demonstrate that all the solutions of the ED-HJBEs together keep the overall system stable in the sense of uniform ultimate boundedness (UUB). To solve the ED-HJBEs, we build a critic-only architecture under the framework of adaptive critic designs. The architecture only employs critic neural networks and updates their weight vectors via the gradient descent method. After that, based on the Lyapunov approach, we prove that the UUB stability of all signals in the closed-loop auxiliary subsystems is assured. Finally, simulations of an illustrated nonlinear interconnected plant are provided to validate the present designs.

Reinforcement-Learning-Based Disturbance Rejection Control for Uncertain Nonlinear Systems

This article investigates the reinforcement-learning (RL)-based disturbance rejection control for uncertain nonlinear systems having nonsimple nominal models. An extended state observer (ESO) is first designed to estimate the system state and the total uncertainty, which represents the perturbation to the nominal system dynamics. Based on the output of the observer, the control compensates for the total uncertainty in real time, and simultaneously, online approximates the optimal policy for the compensated system using a simulation of experience-based RL technique. Rigorous theoretical analysis is given to show the practical convergence of the system state to the origin and the developed policy to the ideal optimal policy. It is worth mentioning that the widely used restrictive persistence of excitation (PE) condition is not required in the established framework. Simulation results are presented to illustrate the effectiveness of the proposed method.

Dynamic Learning From Adaptive Neural Control for Discrete-Time Strict-Feedback Systems

This article first investigates the issue on dynamic learning from adaptive neural network (NN) control of discrete-time strict-feedback nonlinear systems. To verify the exponential convergence of estimated NN weights, an extended stability result is presented for a class of discrete-time linear time-varying systems with time delays. Subsequently, by combining the n -step-ahead predictor technology and backstepping, an adaptive NN controller is constructed, which integrates the novel weight updating laws with time delays and without the σ modification. After ensuring the convergence of system output to a recurrent reference signal, the radial basis function (RBF) NN is verified to satisfy the partial persistent excitation condition. By the combination of the extended stability result, the estimated NN weights can be verified to exponentially converge to their ideal values. The convergent weight sequences are comprehensively represented and stored by constructing some elegant learning rules with some novel sequences and the mod function. The stored knowledge is used again to develop a neural learning control scheme. Compared with the traditional adaptive NN control, the proposed scheme can not only accomplish the same or similar tracking tasks but also greatly improve the transient control performance and alleviate the online computation. Finally, the validity of the presented scheme is illustrated by numerical and practical examples.

Adaptive Interleaved Reinforcement Learning: Robust Stability of Affine Nonlinear Systems With Unknown Uncertainty

This article investigates adaptive robust controller design for discrete-time (DT) affine nonlinear systems using an adaptive dynamic programming. A novel adaptive interleaved reinforcement learning algorithm is developed for finding a robust controller of DT affine nonlinear systems subject to matched or unmatched uncertainties. To this end, the robust control problem is converted into the optimal control problem for nominal systems by selecting an appropriate utility function. The performance evaluation and control policy update combined with neural networks approximation are alternately implemented at each time step for solving a simplified Hamilton-Jacobi-Bellman (HJB) equation such that the uniformly ultimately bounded (UUB) stability of DT affine nonlinear systems can be guaranteed, allowing for all realization of unknown bounded uncertainties. The rigorously theoretical proofs of convergence of the proposed interleaved RL algorithm and UUB stability of uncertain systems are provided. Simulation results are given to verify the effectiveness of the proposed method.

Adaptive Critic Learning-Based Robust Control of Systems with Uncertain Dynamics

Model uncertainties are usually unavoidable in the control systems, which are caused by imperfect system modeling, disturbances, and nonsmooth dynamics. This paper presents a novel method to address the robust control problem for uncertain systems. The original robust control problem of the uncertain system is first transformed into an optimal control of nominal system via selecting the appropriate cost function. Then, we develop an adaptive critic leaning algorithm to learn online the optimal control solution, where only the critic neural network (NN) is used, and the actor NN widely used in the existing methods is removed. Finally, the feasibility analysis of the control algorithm is given in the paper. Simulation results are given to show the availability of the presented control method.

Robust Neurooptimal Control for a Robot via Adaptive Dynamic Programming

We aim at the optimization of the tracking control of a robot to improve the robustness, under the effect of unknown nonlinear perturbations. First, an auxiliary system is introduced, and optimal control of the auxiliary system can be seen as an approximate optimal control of the robot. Then, neural networks (NNs) are employed to approximate the solution of the Hamilton-Jacobi-Isaacs equation under the frame of adaptive dynamic programming. Next, based on the standard gradient attenuation algorithm and adaptive critic design, NNs are trained depending on the designed updating law with relaxing the requirement of initial stabilizing control. In light of the Lyapunov stability theory, all the error signals can be proved to be uniformly ultimately bounded. A series of simulation studies are carried out to show the effectiveness of the proposed control.

Adaptive dynamic programming enhanced admittance control for robots with environment interaction and actuator saturation

This paper focuses on the optimal tracking control problem for robot systems with environment interaction and actuator saturation. A control scheme combined with admittance adaptation and adaptive dynamic programming (ADP) is developed. The unknown environment is modelled as a linear system and admittance controller is derived to achieve compliant behaviour of the robot. In the ADP framework, the cost function is defined with non-quadratic form and the critic network is designed with radial basis function neural network which introduces to obtain an approximate optimal control of the Hamilton–Jacobi–Bellman equation, which guarantees the optimal trajectory tracking. The system stability is analysed by Lyapunov theorem and simulations demonstrate the effectiveness of the proposed strategy.

Auxiliary-system-based composite adaptive optimal backstepping control for uncertain nonlinear guidance systems with input constraints

This paper addresses a missile-target interception guidance process considering acceleration saturation and target maneuver as a constrained nonlinear tracking issue. A dynamic auxiliary system is designed for compensating the effects of constrained input, and external disturbances are counteracted through designing a nonlinear disturbance observer (NDO). The feedforward+feedback composite architecture is built in which a feedforward backstepping control and a feedback optimal control is presented recurrently. Furthermore, the parameter adaptive updating laws are derived to estimate the unknown functions online. Subsequently, the boundedness of the closed-loop signals are guaranteed. The predefined cost function is also ensured to be minimized. Furthermore, the control input is prevented violating its boundary. The contrastive simulation results demonstrate that the robustness of the proposed method is more superior to the nonsingular terminal sliding mode (NTSM) and the proportional navigation (PN) methods.

The Cubic Dynamic Uncertain Causality Graph: A Methodology for Temporal Process Modeling and Diagnostic Logic Inference

To meet the demand for dynamic and highly reliable real-time fault diagnosis for complex systems, we extend the dynamic uncertain causality graph (DUCG) by proposing novel temporal causality modeling and reasoning methods. A new methodology, the Cubic DUCG, is therefore developed. It exploits an efficient scheme for compactly representing and accurately reasoning about the dynamic causalities in the system fault-spreading process. The Cubic DUCG is characterized by: 1) continuous generation of a causality graph that allows for causal connections penetrating among any number of time slices and discards the restrictive assumptions (about the underlying graph structure) upon which the existing research commonly relies; 2) a modeling scheme of complex causalities that includes dynamic negative feedback loops in a natural and intuitive manner; 3) a rigorous and reliable inference algorithm based on complete causalities that reflect real-time fault situations rather than on the cumulative aggregation of static time slices; and 4) some solutions to causality simplification and reduction, graphical transformation, and logical reasoning, for the sake of reducing the reasoning complexity. A series of fault diagnosis experiments on a nuclear power plant simulator verifies the accuracy, robustness, and efficiency of the proposed methodology.

Adaptive Neural Output-Feedback Controller Design of Switched Nonlower Triangular Nonlinear Systems With Time Delays

In this article, we study the issue of adaptive neural output-feedback controller design for a class of uncertain switched time-delay nonlinear systems with nonlower triangular structure. The prominent contribution of this article is that the delay-dependent stability criterion of nonswitched nonlinear systems is successfully extended to that of switched nonlower triangular nonlinear systems. The design algorithm is listed as follows. First, a switched state observer is designed such that the error dynamic system can be generated. Second, neural networks, adaptive backstepping technique, and variable separation method are, respectively, applied to construct a common controller for all subsystems, in which the Lyapunov-Krasovskii functionals are deliberately constructed such that the average dwell-time scheme can be employed to guarantee the stability and performance of the closed-loop system, despite the existence of time delays. Third, the stability analysis process confirms in detail that all the variables of the closed-loop system are semiglobally uniformly ultimately bounded. Finally, simulation study is given to show the validity of the proposed control approach.

Adaptive dynamic programming-based controller with admittance adaptation for robot–environment interaction

The problem of optimal tracking control for robot–environment interaction is studied in this article. The environment is regarded as a linear system and an admittance control with iterative linear quadratic regulator method is obtained to guarantee the compliant behaviour. Meanwhile, an adaptive dynamic programming-based controller is proposed. Under adaptive dynamic programming frame, the critic network is performed with radial basis function neural network to approximate the optimal cost, and the neural network weight updating law is incorporated with an additional stabilizing term to eliminate the requirement for the initial admissible control. The stability of the system is proved by Lyapunov theorem. The simulation results demonstrate the effectiveness of the proposed control scheme.

Composite Neural Learning-Based Nonsingular Terminal Sliding Mode Control of MEMS Gyroscopes

The efficient driving control of MEMS gyroscopes is an attractive way to improve the precision without hardware redesign. This paper investigates the sliding mode control (SMC) for the dynamics of MEMS gyroscopes using neural networks (NNs). Considering the existence of the dynamics uncertainty, the composite neural learning is constructed to obtain higher tracking precision using the serial-parallel estimation model (SPEM). Furthermore, the nonsingular terminal SMC (NTSMC) is proposed to achieve finite-time convergence. To obtain the prescribed performance, a time-varying barrier Lyapunov function (BLF) is introduced to the control scheme. Through simulation tests, it is observed that under the BLF-based NTSMC with composite learning design, the tracking precision of MEMS gyroscopes is highly improved.

Neural Network-Based Adaptive Antiswing Control of an Underactuated Ship-Mounted Crane With Roll Motions and Input Dead Zones

As a type of indispensable oceanic transportation tools, ship-mounted crane systems are widely employed to transport cargoes and containers on vessels due to their extraordinary flexibility. However, various working requirements and the oceanic environment may cause some uncertain and unfavorable factors for ship-mounted crane control. In particular, to accomplish different control tasks, some plant parameters (e.g., boom lengths, payload masses, and so on) frequently change; hence, most existing model-based controllers cannot ensure satisfactory control performance any longer. For example, inaccurate gravity compensation may result in positioning errors. Additionally, due to ship roll motions caused by sea waves, residual payload swing generally exists, which may result in safety risks in practice. To solve the above-mentioned issues, this paper designs a neural network-based adaptive control method that can provide effective control for both actuated and unactuated state variables based on the original nonlinear ship-mounted crane dynamics without any linearizing operations. In particular, the proposed update law availably compensates parameter/structure uncertainties for ship-mounted crane systems. Based on a 2-D sliding surface, the boom and rope can arrive at their preset positions in finite time, and the payload swing can be completely suppressed. Furthermore, the problem of nonlinear input dead zones is also taken into account. The stability of the equilibrium point of all state variables in ship-mounted crane systems is theoretically proven by a rigorous Lyapunov-based analysis. The hardware experimental results verify the practicability and robustness of the presented control approach.

Learning-Based Robust Tracking Control of Quadrotor With Time-Varying and Coupling Uncertainties

In this paper, a learning-based robust tracking control scheme is proposed for a quadrotor unmanned aerial vehicle system. The quadrotor dynamics are modeled including time-varying and coupling uncertainties. By designing position and attitude tracking error subsystems, the robust tracking control strategy is conducted by involving the approximately optimal control of associated nominal error subsystems. Furthermore, an improved weight updating rule is adopted, and neural networks are applied in the learning-based control scheme to get the approximately optimal control laws of the nominal error subsystems. The stability of tracking error subsystems with time-varying and coupling uncertainties is provided as the theoretical guarantee of learning-based robust tracking control scheme. Finally, considering the variable disturbances in the actual environment, three simulation cases are presented based on linear and nonlinear models of quadrotor with competitive results to demonstrate the effectiveness of the proposed control scheme.

Adaptive Tracking Control of Surface Vessel Using Optimized Backstepping Technique

In this paper, a tracking control approach for surface vessel is developed based on the new control technique named optimized backstepping (OB), which considers optimization as a backstepping design principle. Since surface vessel systems are modeled by second-order dynamic in strict feedback form, backstepping is an ideal technique for finishing the tracking task. In the backstepping control of surface vessel, the virtual and actual controls are designed to be the optimized solutions of corresponding subsystems, therefore the overall control is optimized. In general, optimization control is designed based on the solution of Hamilton-Jacobi-Bellman equation. However, solving the equation is very difficult or even impossible due to the inherent nonlinearity and complexity. In order to overcome the difficulty, the reinforcement learning (RL) strategy of actor-critic architecture is usually considered, of which the critic and actor are utilized for evaluating the control performance and executing the control behavior, respectively. By employing the actor-critic RL algorithm for both virtual and actual controls of the vessel, it is proven that the desired optimizing and tracking performances can be arrived. Simulation results further demonstrate effectiveness of the proposed surface vessel control.

Multistability of Delayed Hybrid Impulsive Neural Networks With Application to Associative Memories

The important topic of multistability of continuous-and discrete-time neural network (NN) models has been investigated rather extensively. Concerning the design of associative memories, multistability of delayed hybrid NNs is studied in this paper with an emphasis on the impulse effects. Arising from the spiking phenomenon in biological networks, impulsive NNs provide an efficient model for synaptic interconnections among neurons. Using state-space decomposition, the coexistence of multiple equilibria of hybrid impulsive NNs is analyzed. Multistability criteria are then established regrading delayed hybrid impulsive neurodynamics, for which both the impulse effects on the convergence rate and the basins of attraction of the equilibria are discussed. Illustrative examples are given to verify the theoretical results and demonstrate an application to the design of associative memories. It is shown by an experimental example that delayed hybrid impulsive NNs have the advantages of high storage capacity and high fault tolerance when used for associative memories.

Off-Policy Interleaved Q -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems

In this paper, a novel off-policy interleaved Q-learning algorithm is presented for solving optimal control problem of affine nonlinear discrete-time (DT) systems, using only the measured data along the system trajectories. Affine nonlinear feature of systems, unknown dynamics, and off-policy learning approach pose tremendous challenges on approximating optimal controllers. To this end, on-policy Q-learning method for optimal control of affine nonlinear DT systems is reviewed first, and its convergence is rigorously proven. The bias of solution to Q-function-based Bellman equation caused by adding probing noises to systems for satisfying persistent excitation is also analyzed when using on-policy Q-learning approach. Then, a behavior control policy is introduced followed by proposing an off-policy Q-learning algorithm. Meanwhile, the convergence of algorithm and no bias of solution to optimal control problem when adding probing noise to systems are investigated. Third, three neural networks run by the interleaved Q-learning approach in the actor-critic framework. Thus, a novel off-policy interleaved Q-learning algorithm is derived, and its convergence is proven. Simulation results are given to verify the effectiveness of the proposed method.

Adaptive Neural State-Feedback Tracking Control of Stochastic Nonlinear Switched Systems: An Average Dwell-Time Method

In this paper, the problem of adaptive neural state-feedback tracking control is considered for a class of stochastic nonstrict-feedback nonlinear switched systems with completely unknown nonlinearities. In the design procedure, the universal approximation capability of radial basis function neural networks is used for identifying the unknown compounded nonlinear functions, and a variable separation technique is employed to overcome the design difficulty caused by the nonstrict-feedback structure. The most outstanding novelty of this paper is that individual Lyapunov function of each subsystem is constructed by flexibly adopting the upper and lower bounds of the control gain functions of each subsystem. Furthermore, by combining the average dwell-time scheme and the adaptive backstepping design, a valid adaptive neural state-feedback controller design algorithm is presented such that all the signals of the switched closed-loop system are in probability semiglobally uniformly ultimately bounded, and the tracking error eventually converges to a small neighborhood of the origin in probability. Finally, the availability of the developed control scheme is verified by two simulation examples.