Model-Free Primitive-Based Iterative Learning Control Approach to Trajectory Tracking of MIMO Systems With Experimental Validation

Subspace identification method-based setpoints tracking control and its applications to the column cleaning process

As the last subprocess of copper flotation processes, the column cleaning process plays a decisive role in the tailing copper grade (TCG) and concentrate copper grade (CCG), which are the important factors in determining comprehensive economic indicators. Therefore, the problem of setpoints tracking control of TCG and CCG is particularly important. However, the unknown parameters in the column cleaning process bring great challenges to the problem of setpoints tracking. To overcome this problem, a state space model is constructed based on the two phase model of flotation. Due to the complexity of the column cleaning process, the state-space model matrices cannot be detected or calculated directly. Therefore, a deep autoencoder-based subspace identification method (SIM-DAE) is proposed to identify the state-space model matrices. Next, a Lyapunov-Krasovskii function is proposed to verify the stability and anti-interference performance of the identified system. Meanwhile, the state feedback controller is designed that the TCG and CCG can track with the setpoints. Finally, the effectiveness and feasibility of the proposed methods are verified by the data experiments and an industrial field platform.

Improving Tracking Accuracy for Repetitive Learning Systems by High-Order Extended State Observers

For systems executing repetitive tasks, how to realize the perfect tracking objective is generally desirable, for which an effective method called "iterative learning control (ILC)" emerges thanks to the incorporation of the repetitive execution of systems into an ILC design framework. However, nonrepetitive (iteration-varying) uncertainties are often inevitable in practice and greatly degrade the tracking accuracy of ILC, which has not been treated well, regardless of considerable robust ILC results. This motivates this article to develop a new design method to improve the tracking accuracy of ILC by adopting a high-order extended state observer (ESO) to address ill effects of nonrepetitive uncertainties and uncertain system models. With the designed ESO-based ILC, the robust tracking of any desired trajectory can be achieved such that the tracking error can be decreased to vary in a small bound depending continuously on the bounds of high-order variations of nonrepetitive uncertainties with respect to the iteration. It makes the tracking accuracy of ILC possible to be regulated through the design of ESO, of which the validity is demonstrated by including a simulation example.

Design and Analysis of Data-Driven Learning Control: An Optimization-Based Approach

Learning to perform perfect tracking tasks based on measurement data is desirable in the controller design of systems operating repetitively. This motivates this article to seek an optimization-based design and analysis approach for data-driven learning control systems by focusing on iterative learning control (ILC) of repetitive systems with unknown nonlinear time-varying dynamics. It is shown that perfect output tracking can be realized with updating inputs, where no explicit model knowledge but only measured input-output data are leveraged. In particular, adaptive updating strategies are proposed to obtain parameter estimations of nonlinearities. A double-dynamics analysis approach is applied to establish ILC convergence, together with boundedness of input, output, and estimated parameters, which benefits from employing properties of nonnegative matrices. Simulations are implemented to verify the validity of our optimization-based adaptive ILC.

A reinforcement learning-based near-optimal hierarchical approach for motion control: Design and experiment

As a data-driven design method, model-free optimal control based on reinforcement learning provides an effective way to find optimal control strategies. The design of model-free optimal control is sensitive to system data because it relies on data rather than detailed dynamic models. A prerequisite for generating applicable data is that the system must be open-loop stable (with a stable equilibrium point), which restricts the data-based control design methods in actual control problems and leads to rare experimental studies or verification in the literature. To improve this situation and enrich its applications, we propose a pre-stabilized mechanism and apply it to the motion control of a mechanical system together with a reinforcement learning-based model-free optimal control method, which constitutes a so-called hierarchical control structure. We design two real-time control experiments on an underactuated system to verify its effectiveness. The control results show that the proposed hierarchical control is quite promising in controlling this mechanical system, even though it is open-loop unstable with unknown dynamics.

Introducing a Novel Model-Free Multivariable Adaptive Neural Network Controller for Square MIMO Systems

In this study, a novel Multivariable Adaptive Neural Network Controller (MANNC) is developed for coupled model-free n-input n-output systems. The learning algorithm of the proposed controller does not rely on the model of a system and uses only the history of the system inputs and outputs. The system is considered as a 'black box' with no pre-knowledge of its internal structure. By online monitoring and possessing the system inputs and outputs, the parameters of the controller are adjusted. Using the accumulated gradient of the system error along with the Lyapunov stability analysis, the weights' adjustment convergence of the controller can be observed, and an optimal training number of the controller can be selected. The Lyapunov stability of the system is checked during the entire weight training process to enable the controller to handle any possible nonlinearities of the system. The effectiveness of the MANNC in controlling nonlinear square multiple-input multiple-output (MIMO) systems is demonstrated via three simulation studies covering the cases of a time-invariant nonlinear MIMO system, a time-variant nonlinear MIMO system, and a hybrid MIMO system, respectively. In each case, the performance of the MANNC is compared with that of a properly selected existing counterpart. Simulation results demonstrate that the proposed MANNC is capable of controlling various types of square MIMO systems with much improved performance over its existing counterpart. The unique properties of the MANNC will make it a suitable candidate for many industrial applications.

Hierarchical Cognitive Control for Unknown Dynamic Systems Tracking

A general control system tracking learning framework is proposed, by which an optimal learned tracking behavior called ‘primitive’ is extrapolated to new unseen trajectories without requiring relearning. This is considered intelligent behavior and strongly related to the neuro-motor cognitive control of biological (human-like) systems that deliver suboptimal executions for tasks outside of their current knowledge base, by using previously memorized experience. However, biological systems do not solve explicit mathematical equations for solving learning and prediction tasks. This stimulates the proposed hierarchical cognitive-like learning framework, based on state-of-the-art model-free control: (1) at the low-level L1, an approximated iterative Value Iteration for linearizing the closed-loop system (CLS) behavior by a linear reference model output tracking is first employed; (2) an experiment-driven Iterative Learning Control (EDILC) applied to the CLS from the reference input to the controlled output learns simple tracking tasks called ‘primitives’ in the secondary L2 level, and (3) the tertiary level L3 extrapolates the primitives’ optimal tracking behavior to new tracking tasks without trial-based relearning. The learning framework relies only on input-output system data to build a virtual state space representation of the underlying controlled system that is assumed to be observable. It has been shown to be effective by experimental validation on a representative, coupled, nonlinear, multivariable real-world system. Able to cope with new unseen scenarios in an optimal fashion, the hierarchical learning framework is an advance toward cognitive control systems.

Neural-Network-Based Iterative Learning Control for Multiple Tasks

Iterative learning control (ILC) can synthesize the feedforward control signal for the trajectory tracking control of a repetitive task, even when the system has strong nonlinear dynamics. This makes ILC be one of the most popular methods for trajectory tracking control. Restriction on a repetitive task, however, limits its application to multiple trajectories. This article proposes a neural-network-based ILC (NN-ILC) to deal with nonrepetitive tasks very effectively. A position-based ILC is designed to compensate the tracking error, based on which the multiple outputs of the ILC (ILC outputs) for multiple tasks are expressed as a function of the reference position, velocity, and acceleration. The proposed NN-ILC divides the ILC outputs of multiple tasks into two parts: the linear and nonlinear portions. The first part is expressed by a linear function, which is the linear portion of the function of the ILC outputs. The second part is expressed by a nonlinear function, which is estimated by complementary neural networks including a general neural network and a switching neural network. Finally, the two parts are combined and the ILC outputs of multiple tasks are expressed as a neural-network-based function. Two advantages of the proposed NN-ILC are emphasized. First, the ILC outputs of multiple tasks are compressed into a function by the proposed method, and thus, the memories can be saved. Second, in terms of generalizability, the neural-network-based function of the ILC outputs can easily predict position compensation for multiple tasks without extra iterative learning processes. Experimental results on a robot arm show that the proposed NN-ILC method can easily realize the ILC of multiple tasks. It can save memory comparing with the method of storing the data of multiple tasks and can predict the ILC output of any task, which can accelerate the iterative learning process.

Convergence Analysis of Robust Iterative Learning Control Against Nonrepetitive Uncertainties: System Equivalence Transformation

This article is concerned with the robust convergence analysis of iterative learning control (ILC) against nonrepetitive uncertainties, where the contradiction between convergence conditions for the output tracking error and the input signal (or error) is addressed. A system equivalence transformation (SET) is proposed for robust ILC such that given any desired reference trajectories, the output tracking problems for general nonsquare multi-input, multi-output (MIMO) systems can be equivalently transformed into those for the specific class of square MIMO systems with the same input and output numbers. As a benefit of SET, a unified condition is only needed to guarantee both the uniform boundedness of all system signals and the robust convergence of the output tracking error, which avoids causing the condition contradiction problem in implementing the double-dynamics analysis approach to ILC. Simulation examples are included to demonstrate the validity of our established robust ILC results.

Iterative Learning Model Predictive Control Based on Iterative Data-Driven Modeling

Iterative learning model predictive control (ILMPC) has been recognized as an effective approach to realize high-precision tracking for batch processes with repetitive nature because of its excellent learning ability and closed-loop stability property. However, as a model-based strategy, ILMPC suffers from the unavailability of accurate first principal model in many complex nonlinear batch systems. On account of the abundant process data, nonlinear dynamics of batch systems can be identified precisely along the trials by neural network (NN), making it enforceable to design a data-driven ILMPC. In this article, by using a control-affine feedforward neural network (CAFNN), the features in the process data of the former batch are extracted to form a nonlinear affine model for the controller design in the current batch. Based on the CAFNN model, the ILMPC is formulated in a tube framework to attenuate the influence of modeling errors and track the reference trajectory with sustained accuracy. Due to the control-affine structure, the gradients of the objective function can be analytically computed offline, so as to improve the online computational efficiency and optimization feasibility of the tube ILMPC. The robust stability and the convergence of the data-driven ILMPC system are analyzed theoretically. The simulation on a typical batch reactor verifies the effectiveness of the proposed control method.

Virtual State Feedback Reference Tuning and Value Iteration Reinforcement Learning for Unknown Observable Systems Control

In this paper, a novel Virtual State-feedback Reference Feedback Tuning (VSFRT) and Approximate Iterative Value Iteration Reinforcement Learning (AI-VIRL) are applied for learning linear reference model output (LRMO) tracking control of observable systems with unknown dynamics. For the observable system, a new state representation in terms of input/output (IO) data is derived. Consequently, the Virtual State Feedback Tuning (VRFT)-based solution is redefined to accommodate virtual state feedback control, leading to an original stability-certified Virtual State-Feedback Reference Tuning (VSFRT) concept. Both VSFRT and AI-VIRL use neural networks controllers. We find that AI-VIRL is significantly more computationally demanding and more sensitive to the exploration settings, while leading to inferior LRMO tracking performance when compared to VSFRT. It is not helped either by transfer learning the VSFRT control as initialization for AI-VIRL. State dimensionality reduction using machine learning techniques such as principal component analysis and autoencoders does not improve on the best learned tracking performance however it trades off the learning complexity. Surprisingly, unlike AI-VIRL, the VSFRT control is one-shot (non-iterative) and learns stabilizing controllers even in poorly, open-loop explored environments, proving to be superior in learning LRMO tracking control. Validation on two nonlinear coupled multivariable complex systems serves as a comprehensive case study.

Convergence Analysis of Saturated Iterative Learning Control Systems With Locally Lipschitz Nonlinearities

In this article, the robust trajectory tracking problem of iterative learning control (ILC) for uncertain nonlinear systems is considered, and the effects from locally Lipschitz nonlinearities, input saturations, and nonzero system relative degrees are treated. A saturated ILC algorithm is given, with the convergence analysis exploited using a composite energy function-based approach. It is shown that the tracking error can be guaranteed to converge both pointwisely and uniformly. Furthermore, the input updating signal can be ensured to eventually satisfy the input saturation requirements with increasing iterations. Two examples are given to demonstrate the validity of saturated ILC for systems with the relative degree of one, input saturation, and locally Lipschitz nonlinearity.

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

This work suggests a solution for the output reference model (ORM) tracking control problem, based on approximate dynamic programming. General nonlinear systems are included in a control system (CS) and subjected to state feedback. By linear ORM selection, indirect CS feedback linearization is obtained, leading to favorable linear behavior of the CS. The Value Iteration (VI) algorithm ensures model-free nonlinear state feedback controller learning, without relying on the process dynamics. From linear to nonlinear parameterizations, a reliable approximate VI implementation in continuous state-action spaces depends on several key parameters such as problem dimension, exploration of the state-action space, the state-transitions dataset size, and a suitable selection of the function approximators. Herein, we find that, given a transition sample dataset and a general linear parameterization of the Q-function, the ORM tracking performance obtained with an approximate VI scheme can reach the performance level of a more general implementation using neural networks (NNs). Although the NN-based implementation takes more time to learn due to its higher complexity (more parameters), it is less sensitive to exploration settings, number of transition samples, and to the selected hyper-parameters, hence it is recommending as the de facto practical implementation. Contributions of this work include the following: VI convergence is guaranteed under general function approximators; a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multivariable aerodynamic process; comparisons with an offline deep deterministic policy gradient solution; implementation details and further discussions on the obtained results.

Convergence Conditions for Solving Robust Iterative Learning Control Problems Under Nonrepetitive Model Uncertainties

Learning from saved measurement and control data to refine the performance of output tracking is the core feature of iterative learning control (ILC). Even though this implementation process of ILC does not need any model knowledge, ILC typically requires the strict repetitiveness of the control systems, especially on the plant models of them. The questions of interest in this paper are: 1) whether and how can robust ILC problems be solved with respect to the nonrepetitive (or iteration-dependent) model uncertainties and 2) can convergence conditions be developed with the effective contraction mapping (CM)-based approach to ILC? The answers to these questions are affirmative, and the CM-based approach is applicable to robust ILC that accommodates certain nonrepetitive uncertainties, especially in the plant models. In particular, an H_∞ -norm condition is proposed to ensure the robust ILC convergence, which can be solved to determine learning gain matrices. Simulation tests are performed to illustrate the validity of our presented H_∞ -based analysis results.

Data-Driven Model-Free Tracking Reinforcement Learning Control with VRFT-based Adaptive Actor-Critic

This paper proposes a neural network (NN)-based control scheme in an Adaptive Actor-Critic (AAC) learning framework designed for output reference model tracking, as a representative deep-learning application. The control learning scheme is model-free with respect to the process model. AAC designs usually require an initial controller to start the learning process; however, systematic guidelines for choosing the initial controller are not offered in the literature, especially in a model-free manner. Virtual Reference Feedback Tuning (VRFT) is proposed for obtaining an initially stabilizing NN nonlinear state-feedback controller, designed from input-state-output data collected from the process in open-loop setting. The solution offers systematic design guidelines for initial controller design. The resulting suboptimal state-feedback controller is next improved under the AAC learning framework by online adaptation of a critic NN and a controller NN. The mixed VRFT-AAC approach is validated on a multi-input multi-output nonlinear constrained coupled vertical two-tank system. Discussions on the control system behavior are offered together with comparisons with similar approaches.

Adaptive Learning Control for Nonlinear Systems With Randomly Varying Iteration Lengths

This paper proposes adaptive iterative learning control (ILC) schemes for continuous-time parametric nonlinear systems with iteration lengths that randomly vary. As opposed to the existing ILC works that feature nonuniform trial lengths, this paper is applicable to nonlinear systems that do not satisfy the globally Lipschitz continuous condition. In addition, this paper introduces a novel composite energy function based on newly defined virtual tracking error information for proving the asymptotical convergence. Both an original update algorithm and a projection-based update algorithm for estimating the unknown parameters are proposed. Extensions to cases with unknown input gains, iteration-varying tracking references, nonparametric uncertainty, high-order nonlinear systems, and multi-input-multi-output systems are all elaborated upon. Illustrative simulations are provided to verify the theoretical results.

Iterative Learning Control for a Flapping Wing Micro Aerial Vehicle Under Distributed Disturbances

This paper addresses a flexible micro aerial vehicle (MAV) under spatiotemporally varying disturbances, which is composed of a rigid body and two flexible wings. Based on Hamilton's principle, a distributed parameter system coupling in bending and twisting, is modeled. Two iterative learning control (ILC) schemes are designed to suppress the vibrations in bending and twisting, reject the distributed disturbances and regulate the displacement of the rigid body to track a prescribed constant trajectory. At the basis of composite energy function, the boundedness and the learning convergence are proved for the closed-loop MAV system. Simulation results are provided to illustrate the effectiveness of the proposed ILC laws.

Deterministic Convergence for Learning Control Systems Over Iteration-Dependent Tracking Intervals

This brief addresses the iterative learning control (ILC) problems for discrete-time systems subject to iteration-dependent tracking time intervals. A modified class of P-type ILC algorithms is proposed by properly defining an available modified output, for which robust convergence analysis is performed with an inductive approach. It is shown that if a persistent full-learning property is ensured, then a necessary and sufficient convergence condition of ILC can be derived to reach the perfect output tracking objective though the tracking time interval is iteration-dependent. That is, the tracking of ILC for iteration-dependent time intervals can be guaranteed in the same deterministic (not stochastic) convergence way as that of traditional ILC over a fixed time interval. Furthermore, the developed tracking results can be extended to admit iteration-dependent uncertainties in initial state and external disturbances. Simulation tests are also included to demonstrate the effectiveness of the modified P-type ILC.

Data-Driven Learning Control for Stochastic Nonlinear Systems: Multiple Communication Constraints and Limited Storage

This paper proposes a data-driven learning control method for stochastic nonlinear systems under random communication conditions, including data dropouts, communication delays, and packet transmission disordering. A renewal mechanism is added to the buffer to regulate the arrived packets, and a recognition mechanism is introduced to the controller for the selection of suitable update packets. Both intermittent and successive update schemes are proposed based on the conventional P-type iterative learning control algorithm, and are shown to converge to the desired input with probability one. The convergence and effectiveness of the proposed algorithms are verified by means of illustrative simulations.

Adaptive Boundary Iterative Learning Control for an Euler-Bernoulli Beam System With Input Constraint

This paper addresses the vibration control and the input constraint for an Euler-Bernoulli beam system under aperiodic distributed disturbance and aperiodic boundary disturbance. Hyperbolic tangent functions and saturation functions are adopted to tackle the input constraint. A restrained adaptive boundary iterative learning control (ABILC) law is proposed based on a time-weighted Lyapunov-Krasovskii-like composite energy function. In order to deal with the uncertainty of a system parameter and reject the external disturbances, three adaptive laws are designed and learned in the iteration domain. All the system states of the closed-loop system are proved to be bounded in each iteration. Along the iteration axis, the displacements asymptotically converge toward zero. Simulation results are provided to illustrate the effectiveness of the proposed ABILC scheme.

Data-driven model reference control of MIMO vertical tank systems with model-free VRFT and Q-Learning

This paper proposes a combined Virtual Reference Feedback Tuning-Q-learning model-free control approach, which tunes nonlinear static state feedback controllers to achieve output model reference tracking in an optimal control framework. The novel iterative Batch Fitted Q-learning strategy uses two neural networks to represent the value function (critic) and the controller (actor), and it is referred to as a mixed Virtual Reference Feedback Tuning-Batch Fitted Q-learning approach. Learning convergence of the Q-learning schemes generally depends, among other settings, on the efficient exploration of the state-action space. Handcrafting test signals for efficient exploration is difficult even for input-output stable unknown processes. Virtual Reference Feedback Tuning can ensure an initial stabilizing controller to be learned from few input-output data and it can be next used to collect substantially more input-state data in a controlled mode, in a constrained environment, by compensating the process dynamics. This data is used to learn significantly superior nonlinear state feedback neural networks controllers for model reference tracking, using the proposed Batch Fitted Q-learning iterative tuning strategy, motivating the original combination of the two techniques. The mixed Virtual Reference Feedback Tuning-Batch Fitted Q-learning approach is experimentally validated for water level control of a multi input-multi output nonlinear constrained coupled two-tank system. Discussions on the observed control behavior are offered.

Data-Driven Tracking Control With Adaptive Dynamic Programming for a Class of Continuous-Time Nonlinear Systems

A data-driven adaptive tracking control approach is proposed for a class of continuous-time nonlinear systems using a recent developed goal representation heuristic dynamic programming (GrHDP) architecture. The major focus of this paper is on designing a multivariable tracking scheme, including the filter-based action network (FAN) architecture, and the stability analysis in continuous-time fashion. In this design, the FAN is used to observe the system function, and then generates the corresponding control action together with the reference signals. The goal network will provide an internal reward signal adaptively based on the current system states and the control action. This internal reward signal is assigned as the input for the critic network, which approximates the cost function over time. We demonstrate its improved tracking performance in comparison with the existing heuristic dynamic programming (HDP) approach under the same parameter and environment settings. The simulation results of the multivariable tracking control on two examples have been presented to show that the proposed scheme can achieve better control in terms of learning speed and overall performance.

Model-free adaptive optimal controller design for aeroelastic system with input constraints

In this thesis, an innovative model-free adaptive control strategy based on a multi-observer technique that takes advantage of input/output measurement data is proposed for the aeroelastic system of the two degree of freedom pitch-plunge wing, and this unknown complicated nonlinear system is a general multi-input multi-output plant with input constraints. In this algorithm, the multi-observer technique is applied to estimate the value of the pseudopartial derivative parameter matrix in the approach of the compact form dynamic linearization designed to linearize the model of the two-dimensional wing-flap system with input constraints. At the same time, this model-free adaptive control method consists of the approximate internal model and the optimal controller. Moreover, this control scheme is based on the linear matrix inequalities, which is a kind of real-time computation. In the design process for controlling this two-dimensional wing-flap system in the condition that the control inputs are subjected to amplitude and change rate limits, the problem of the dynamic control is transformed into the optimization problem, which can minimize the performance index. Finally, simulation results for the two-dimensional wing-flap system with input constraints can demonstrate the availability and potential of the presented multi-observer-based model-free optimal control strategy for unknown nonlinear multi-input multi-output system with input saturation.