Data-Knowledge-Driven Self-Organizing Fuzzy Neural Network

Fuzzy neural networks (FNNs) hold the advantages of knowledge leveraging and adaptive learning, which have been widely used in nonlinear system modeling. However, it is difficult for FNNs to obtain the appropriate structure in the situation of insufficient data, which limits its generalization performance. To solve this problem, a data-knowledge-driven self-organizing FNN (DK-SOFNN) with a structure compensation strategy and a parameter reinforcement mechanism is proposed in this article. First, a structure compensation strategy is proposed to mine structural information from empirical knowledge to learn the structure of DK-SOFNN. Then, a complete model structure can be acquired by sufficient structural information. Second, a parameter reinforcement mechanism is developed to determine the parameter evolution direction of DK-SOFNN that is most suitable for the current model structure. Then, a robust model can be obtained by the interaction between parameters and dynamic structure. Finally, the proposed DK-SOFNN is theoretically analyzed on the fixed structure case and dynamic structure case. Then, the convergence conditions can be obtained to guide practical applications. The merits of DK-SOFNN are demonstrated by some benchmark problems and industrial applications.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Enhancement of the HILOMOT Algorithm with Modified EM and Modified PSO Algorithms for Nonlinear Systems Identification

Developing a mathematical model has become an inevitable need in studies of all disciplines. With advancements in technology, there is an emerging need to develop complex mathematical models. System identification is a popular way of constructing mathematical models of highly complex processes when an analytical model is not feasible. One of the many model architectures of system identification is to utilize a Local Model Network (LMN). Hierarchical Local Model Tree (HILOMOT) is an iterative LMN training algorithm that uses the axis-oblique split method to divide the input space hierarchically. The split positions of the local models directly influence the accuracy of the entire model. However, finding the best split positions of the local models presents a nonlinear optimization problem. This paper presents an optimized HILOMOT algorithm with enhanced Expectation–Maximization (EM) and Particle Swarm Optimization (PSO) algorithms which includes the normalization parameter and utilizes the reduced-parameter vector. Finally, the performance of the improved HILOMOT algorithm is compared with the existing algorithm by modeling the NOx emission model of a gas turbine and multiple nonlinear test functions of different orders and structures.

Interval Type-2 Fuzzy Neural Networks for Chaotic Time Series Prediction: A Concise Overview

Chaotic time series widely exists in nature and society (e.g., meteorology, physics, economics, etc.), which usually exhibits seemingly unpredictable features due to its inherent nonstationary and high complexity. Thankfully, multifarious advanced approaches have been developed to tackle the prediction issues, such as statistical methods, artificial neural networks (ANNs), and support vector machines. Among them, the interval type-2 fuzzy neural network (IT2FNN), which is a synergistic integration of fuzzy logic systems and ANNs, has received wide attention in the field of chaotic time series prediction. This paper begins with the structural features and superiorities of IT2FNN. Moreover, chaotic characters identification and phase-space reconstruction matters for prediction are presented. In addition, we also offer a comprehensive review of state-of-the-art applications of IT2FNN, with an emphasis on chaotic time series prediction and summarize their main contributions as well as some hardware implementations for computation speedup. Finally, this paper trends and extensions of this field, along with an outlook of future challenges are revealed. The primary objective of this paper is to serve as a tutorial or referee for interested researchers to have an overall picture on the current developments and identify their potential research direction to further investigation.

Kidney-inspired algorithm with reduced functionality treatment for classification and time series prediction

Optimization of an artificial neural network model through the use of optimization algorithms is the common method employed to search for an optimum solution for a broad variety of real-world problems. One such optimization algorithm is the kidney-inspired algorithm (KA) which has recently been proposed in the literature. The algorithm mimics the four processes performed by the kidneys: filtration, reabsorption, secretion, and excretion. However, a human with reduced kidney function needs to undergo additional treatment to improve kidney performance. In the medical field, the glomerular filtration rate (GFR) test is used to check the health of kidneys. The test estimates the amount of blood that passes through the glomeruli each minute. In this paper, we mimic this kidney function test and the GFR result is used to select a suitable step to add to the basic KA process. This novel imitation is designed for both minimization and maximization problems. In the proposed method, depends on GFR test result which is less than 15 or falls between 15 and 60 or is more than 60 a particular action is performed. These additional processes are applied as required with the aim of improving exploration of the search space and increasing the likelihood of the KA finding the optimum solution. The proposed method is tested on test functions and its results are compared with those of the basic KA. Its performance on benchmark classification and time series prediction problems is also examined and compared with that of other available methods in the literature. In addition, the proposed method is applied to a real-world water quality prediction problem. The statistical analysis of all these applications showed that the proposed method had a ability to improve the optimization outcome.

Self-Organizing RBF Neural Network Using an Adaptive Gradient Multiobjective Particle Swarm Optimization

One of the major obstacles in using radial basis function (RBF) neural networks is the convergence toward local minima instead of the global minima. For this reason, an adaptive gradient multiobjective particle swarm optimization (AGMOPSO) algorithm is designed to optimize both the structure and parameters of RBF neural networks in this paper. First, the AGMOPSO algorithm, based on a multiobjective gradient method and a self-adaptive flight parameters mechanism, is developed to improve the computation performance. Second, the AGMOPSO-based self-organizing RBF neural network (AGMOPSO-SORBF) can optimize the parameters (centers, widths, and weights), as well as determine the network size. The goal of AGMOPSO-SORBF is to find a tradeoff between the accuracy and the complexity of RBF neural networks. Third, the convergence analysis of AGMOPSO-SORBF is detailed to ensure the prerequisite of any successful applications. Finally, the merits of our proposed approach are verified on multiple numerical examples. The results indicate that the proposed AGMOPSO-SORBF achieves much better generalization capability and compact network structure than some other existing methods.

An Adaptive-PSO-Based Self-Organizing RBF Neural Network

In this paper, a self-organizing radial basis function (SORBF) neural network is designed to improve both accuracy and parsimony with the aid of adaptive particle swarm optimization (APSO). In the proposed APSO algorithm, to avoid being trapped into local optimal values, a nonlinear regressive function is developed to adjust the inertia weight. Furthermore, the APSO algorithm can optimize both the network size and the parameters of an RBF neural network simultaneously. As a result, the proposed APSO-SORBF neural network can effectively generate a network model with a compact structure and high accuracy. Moreover, the analysis of convergence is given to guarantee the successful application of the APSO-SORBF neural network. Finally, multiple numerical examples are presented to illustrate the effectiveness of the proposed APSO-SORBF neural network. The results demonstrate that the proposed method is more competitive in solving nonlinear problems than some other existing SORBF neural networks.

A Collective Neurodynamic Optimization Approach to Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) is an advanced method for nonnegative feature extraction, with widespread applications. However, the NMF solution often entails to solve a global optimization problem with a nonconvex objective function and nonnegativity constraints. This paper presents a collective neurodynamic optimization (CNO) approach to this challenging problem. The proposed collective neurodynamic system consists of a population of recurrent neural networks (RNNs) at the lower level and a particle swarm optimization (PSO) algorithm with wavelet mutation at the upper level. The RNNs act as search agents carrying out precise local searches according to their neurodynamics and initial conditions. The PSO algorithm coordinates and guides the RNNs with updated initial states toward global optimal solution(s). A wavelet mutation operator is added to enhance PSO exploration diversity. Through iterative interaction and improvement of the locally best solutions of RNNs and global best positions of the whole population, the population-based neurodynamic systems are almost sure able to achieve the global optimality for the NMF problem. It is proved that the convergence of the group-best state to the global optimal solution with probability one. The experimental results substantiate the efficacy and superiority of the CNO approach to bound-constrained global optimization with several benchmark nonconvex functions and NMF-based clustering with benchmark data sets in comparison with the state-of-the-art algorithms.

Barrier Function-Based Neural Adaptive Control With Locally Weighted Learning and Finite Neuron Self-Growing Strategy

This paper presents a new approach to construct neural adaptive control for uncertain nonaffine systems. By integrating locally weighted learning with barrier Lyapunov function (BLF), a novel control design method is presented to systematically address the two critical issues in neural network (NN) control field: one is how to fulfill the compact set precondition for NN approximation, and the other is how to use varying rather than a fixed NN structure to improve the functionality of NN control. A BLF is exploited to ensure the NN inputs to remain bounded during the entire system operation. To account for system nonlinearities, a neuron self-growing strategy is proposed to guide the process for adding new neurons to the system, resulting in a self-adjustable NN structure for better learning capabilities. It is shown that the number of neurons needed to accomplish the control task is finite, and better performance can be obtained with less number of neurons as compared with traditional methods. The salient feature of the proposed method also lies in the continuity of the control action everywhere. Furthermore, the resulting control action is smooth almost everywhere except for a few time instants at which new neurons are added. Numerical example illustrates the effectiveness of the proposed approach.

Adaptive Fault-Tolerant Control of Uncertain Nonlinear Large-Scale Systems With Unknown Dead Zone

In this paper, an adaptive neural fault-tolerant control scheme is proposed and analyzed for a class of uncertain nonlinear large-scale systems with unknown dead zone and external disturbances. To tackle the unknown nonlinear interaction functions in the large-scale system, the radial basis function neural network (RBFNN) is employed to approximate them. To further handle the unknown approximation errors and the effects of the unknown dead zone and external disturbances, integrated as the compounded disturbances, the corresponding disturbance observers are developed for their estimations. Based on the outputs of the RBFNN and the disturbance observer, the adaptive neural fault-tolerant control scheme is designed for uncertain nonlinear large-scale systems by using a decentralized backstepping technique. The closed-loop stability of the adaptive control system is rigorously proved via Lyapunov analysis and the satisfactory tracking performance is achieved under the integrated effects of unknown dead zone, actuator fault, and unknown external disturbances. Simulation results of a mass-spring-damper system are given to illustrate the effectiveness of the proposed adaptive neural fault-tolerant control scheme for uncertain nonlinear large-scale systems.

Impact of Noise on a Dynamical System: Prediction and Uncertainties from a Swarm-Optimized Neural Network

An artificial neural network (ANN) based on particle swarm optimization (PSO) was developed for the time series prediction. The hybrid ANN+PSO algorithm was applied on Mackey-Glass chaotic time series in the short-term x(t + 6). The performance prediction was evaluated and compared with other studies available in the literature. Also, we presented properties of the dynamical system via the study of chaotic behaviour obtained from the predicted time series. Next, the hybrid ANN+PSO algorithm was complemented with a Gaussian stochastic procedure (called stochastic hybrid ANN+PSO) in order to obtain a new estimator of the predictions, which also allowed us to compute the uncertainties of predictions for noisy Mackey-Glass chaotic time series. Thus, we studied the impact of noise for several cases with a white noise level (σ_N) from 0.01 to 0.1.

Nonlinear model predictive control based on collective neurodynamic optimization

In general, nonlinear model predictive control (NMPC) entails solving a sequential global optimization problem with a nonconvex cost function or constraints. This paper presents a novel collective neurodynamic optimization approach to NMPC without linearization. Utilizing a group of recurrent neural networks (RNNs), the proposed collective neurodynamic optimization approach searches for optimal solutions to global optimization problems by emulating brainstorming. Each RNN is guaranteed to converge to a candidate solution by performing constrained local search. By exchanging information and iteratively improving the starting and restarting points of each RNN using the information of local and global best known solutions in a framework of particle swarm optimization, the group of RNNs is able to reach global optimal solutions to global optimization problems. The essence of the proposed collective neurodynamic optimization approach lies in the integration of capabilities of global search and precise local search. The simulation results of many cases are discussed to substantiate the effectiveness and the characteristics of the proposed approach.

Neural Networks Predictive Controller Using an Adaptive Control Rate

A model predictive control design for nonlinear systems based on artificial neural networks is discussed. The Feedforward neural networks are used to describe the unknown nonlinear dynamics of the real system. The backpropagation algorithm is used, offline, to train the neural networks model. The optimal control actions are computed by solving a nonconvex optimization problem with the gradient method. In gradient method, the steepest descent is a sensible factor for convergence. Then, an adaptive variable control rate based on Lyapunov function candidate and asymptotic convergence of the predictive controller are proposed. The stability of the closed loop system based on the neural model is proved. In order to demonstrate the robustness of the proposed predictive controller under set-point and load disturbance, a simulation example is considered. A comparison of the control performance achieved with a Levenberg-Marquardt method is also provided to illustrate the effectiveness of the proposed controller.

Robust model predictive control of nonlinear systems with unmodeled dynamics and bounded uncertainties based on neural networks

This paper presents a neural network approach to robust model predictive control (MPC) for constrained discrete-time nonlinear systems with unmodeled dynamics affected by bounded uncertainties. The exact nonlinear model of underlying process is not precisely known, but a partially known nominal model is available. This partially known nonlinear model is first decomposed to an affine term plus an unknown high-order term via Jacobian linearization. The linearization residue combined with unmodeled dynamics is then modeled using an extreme learning machine via supervised learning. The minimax methodology is exploited to deal with bounded uncertainties. The minimax optimization problem is reformulated as a convex minimization problem and is iteratively solved by a two-layer recurrent neural network. The proposed neurodynamic approach to nonlinear MPC improves the computational efficiency and sheds a light for real-time implementability of MPC technology. Simulation results are provided to substantiate the effectiveness and characteristics of the proposed approach.

Real-time model predictive control using a self-organizing neural network

In this paper, a real-time model predictive control (RT-MPC) based on self-organizing radial basis function neural network (SORBFNN) is proposed for nonlinear systems. This RT-MPC has its simplicity in parallelism to model predictive control design and efficiency to deal with computational complexity. First, a SORBFNN with concurrent structure and parameter learning is developed as the predictive model of the nonlinear systems. The model performance can be significantly improved through SORBFNN, and the modeling error is uniformly ultimately bounded. Second, a fast gradient method (GM) is enhanced for the solution of optimal control problem. This proposed GM can reduce computational cost and suboptimize the RT-MPC online. Then, the conditions of the stability analysis and steady-state performance of the closed-loop systems are presented. Finally, numerical simulations reveal that the proposed control gives satisfactory tracking and disturbance rejection performances. Experimental results demonstrate its effectiveness.

Convergence analysis of particle swarm optimizer and its improved algorithm based on velocity differential evolution

This paper presents an analysis of the relationship of particle velocity and convergence of the particle swarm optimization. Its premature convergence is due to the decrease of particle velocity in search space that leads to a total implosion and ultimately fitness stagnation of the swarm. An improved algorithm which introduces a velocity differential evolution (DE) strategy for the hierarchical particle swarm optimization (H-PSO) is proposed to improve its performance. The DE is employed to regulate the particle velocity rather than the traditional particle position in case that the optimal result has not improved after several iterations. The benchmark functions will be illustrated to demonstrate the effectiveness of the proposed method.

Modeling and monitoring of dynamic processes

In this paper, a new online monitoring approach is proposed for handling the dynamic problem in industrial batch processes. Compared to conventional methods, its contributions are as follows: (1) multimodes are separated correctly since the cross-mode correlations are considered and the common information is extracted; (2) the expensive computing load is avoided since only the specific information is calculated when a mode is monitored online; and (3) after that, two different subspaces are separated, and the common and specific subspace models are built and analyzed, respectively. The monitoring is carried out in the subspace. The corresponding confidence regions are constructed according to their respective models.

Multistability of neural networks with time-varying delays and concave-convex characteristics

In this paper, stability of multiple equilibria of neural networks with time-varying delays and concave-convex characteristics is formulated and studied. Some sufficient conditions are obtained to ensure that an n-neuron neural network with concave-convex characteristics can have a fixed point located in the appointed region. By means of an appropriate partition of the n-dimensional state space, when nonlinear activation functions of an n-neuron neural network are concave or convex in 2k+2m-1 intervals, this neural network can have (2k+2m-1)n equilibrium points. This result can be applied to the multiobjective optimal control and associative memory. In particular, several succinct criteria are given to ascertain multistability of cellular neural networks. These stability conditions are the improvement and extension of the existing stability results in the literature. A numerical example is given to illustrate the theoretical findings via computer simulations.