A One-Layer Recurrent Neural Network for Pseudoconvex Optimization Problems With Equality and Inequality Constraints

A smooth gradient approximation neural network for general constrained nonsmooth nonconvex optimization problems

Nonsmooth nonconvex optimization problems are pivotal in engineering practice due to the inherent nonsmooth and nonconvex characteristics of many real-world complex systems and models. The nonsmoothness and nonconvexity of the objective and constraint functions bring great challenges to the design and convergence analysis of the optimization algorithms. This paper presents a smooth gradient approximation neural network for such optimization problems, in which a smooth approximation technique with time-varying control parameter is introduced for handling nonsmooth nonregular objective functions. In addition, a hard comparator function is introduced to ensure that the state solution of the proposed neural network remains within the nonconvex inequality constraint sets. Any accumulation point of the state solution of the proposed neural network is proved to be a stationary point of the nonconvex optimization under consideration. Furthermore, the neural network demonstrates the ability to find optimal solutions for some generalized convex optimization problems. Compared with the related neural networks, the constructed neural network has weaker convergence conditions and simpler algorithm structure. Simulation results and an application in optimizing condition number verify the practical applicability of the presented algorithm.

Adaptive penalty-based neurodynamic approach for nonsmooth interval-valued optimization problem

The complex and diverse practical background drives this paper to explore a new neurodynamic approach (NA) to solve nonsmooth interval-valued optimization problems (IVOPs) constrained by interval partial order and more general sets. On the one hand, to deal with the uncertainty of interval-valued information, the LU-optimality condition of IVOPs is established through a deterministic form. On the other hand, according to the penalty method and adaptive controller, the interval partial order constraint and set constraint are punished by one adaptive parameter, which is a key enabler for the feasibility of states while having a lower solution space dimension and avoiding estimating exact penalty parameters. Through nonsmooth analysis and Lyapunov theory, the proposed adaptive penalty-based neurodynamic approach (APNA) is proven to converge to an LU-solution of the considered IVOPs. Finally, the feasibility of the proposed APNA is illustrated by numerical simulations and an investment decision-making problem.

A Robust Coevolutionary Neural-Based Optimization Algorithm for Constrained Nonconvex Optimization

For nonconvex optimization problems, a routine is to assume that there is no perturbation when executing the solution task. Nevertheless, dealing with the perturbation in advance may increase the burden on the system and take up extra time. To remedy this weakness, we propose a robust coevolutionary neural-based optimization algorithm with inherent robustness based on the hybridization between the particle swarm optimization and a class of robust neural dynamics (RND). In this framework, every neural agent guided by the RND supersedes the place of the particle, mutually searches for the optimal solution, and stabilizes itself from different perturbations. The theoretical analysis ensures that the proposed algorithm is globally convergent with probability one. Besides, the effectiveness and robustness of the proposed approach are illustrated by illustrative examples compared with the existing methods. We further apply this proposed algorithm to the source localization and manipulability optimization of the redundant manipulator, simultaneously disposing of perturbation from the internal and exogenous system with satisfactory performance.

Zeroing Neural Network With Coefficient Functions and Adjustable Parameters for Solving Time-Variant Sylvester Equation

To solve the time-variant Sylvester equation, in 2013, Li et al. proposed the zeroing neural network with sign-bi-power function (ZNN-SBPF) model via constructing a nonlinear activation function. In this article, to further improve the convergence rate, the zeroing neural network with coefficient functions and adjustable parameters (ZNN-CFAP) model as a variation in zeroing neural network (ZNN) model is proposed. On the basis of the introduced coefficient functions, an appropriate ZNN-CFAP model can be chosen according to the error function. The high convergence rate of the ZNN-CFAP model can be achieved by choosing appropriate adjustable parameters. Moreover, the finite-time convergence property and convergence time upper bound of the ZNN-CFAP model are proved in theory. Computer simulations and numerical experiments are performed to illustrate the efficacy and validity of the ZNN-CFAP model in time-variant Sylvester equation solving. Comparative experiments among the ZNN-CFAP, ZNN-SBPF, and ZNN with linear function (ZNN-LF) models further substantiate the superiority of the ZNN-CFAP model in view of the convergence rate. Finally, the proposed ZNN-CFAP model is successfully applied to the tracking control of robot manipulator to verify its practicability.

Zhang equivalency of inequation-to-inequation type for constraints of redundant manipulators

In solving specific problems, physical laws and mathematical theorems directly express the connections between variables with equations/inequations. At times, it could be extremely hard or not viable to solve these equations/inequations directly. The PE (principle of equivalence) is a commonly applied pragmatic method across multiple fields. PE transforms the initial equations/inequations into simplified equivalent equations/inequations that are more manageable to solve, allowing researchers to achieve their objectives. The problem-solving process in many fields benefits from the use of PE. Recently, the ZE (Zhang equivalency) framework has surfaced as a promising approach for addressing time-dependent optimization problems. This ZEF (ZE framework) consolidates constraints at different tiers, demonstrating its capacity for the solving of time-dependent optimization problems. To broaden the application of ZEF in time-dependent optimization problems, specifically in the domain of motion planning for redundant manipulators, the authors systematically investigate the ZEF-I2I (ZEF of the inequation-to-inequation) type. The study concentrates on transforming constraints (i.e., joint constraints and obstacles avoidance depicted in different tiers) into consolidated constraints backed by rigorous mathematical derivations. The effectiveness and applicability of the ZEF-I2I are verified through two optimization motion planning schemes, which consolidate constraints in the velocity-tier and acceleration-tier. Schemes are required to accomplish the goal of repetitive motion planning within constraints. The firstly presented optimization motion planning schemes are then reformulated as two time-dependent quadratic programming problems. Simulative experiments conducted on the basis of a six-joint redundant manipulator confirm the outstanding effectiveness of the firstly presented ZEF-I2I in achieving the goal of motion planning within constraints.

A Projection Neural Network to Nonsmooth Constrained Pseudoconvex Optimization

In this article, a single-layer projection neural network based on penalty function and differential inclusion is proposed to solve nonsmooth pseudoconvex optimization problems with linear equality and convex inequality constraints, and the bound constraints, such as box and sphere types, in inequality constraints are processed by projection operator. By introducing the Tikhonov-like regularization method, the proposed neural network no longer needs to calculate the exact penalty parameters. Under mild assumptions, by nonsmooth analysis, it is proved that the state solution of the proposed neural network is always bounded and globally exists, and enters the constrained feasible region in a finite time, and never escapes from this region again. Finally, the state solution converges to an optimal solution for the considered optimization problem. Compared with some other existing neural networks based on subgradients, this algorithm eliminates the dependence on the selection of the initial point, which is a neural network model with a simple structure and low calculation load. Three numerical experiments and two application examples are used to illustrate the global convergence and effectiveness of the proposed neural network.

A subgradient-based neurodynamic algorithm to constrained nonsmooth nonconvex interval-valued optimization

In this paper, a subgradient-based neurodynamic algorithm is presented to solve the nonsmooth nonconvex interval-valued optimization problem with both partial order and linear equality constraints, where the interval-valued objective function is nonconvex, and interval-valued partial order constraint functions are convex. The designed neurodynamic system is constructed by a differential inclusion with upper semicontinuous right-hand side, whose calculation load is reduced by relieving penalty parameters estimation and complex matrix inversion. Based on nonsmooth analysis and the extension theorem of the solution of differential inclusion, it is obtained that the global existence and boundedness of state solution of neurodynamic system, as well as the asymptotic convergence of state solution to the feasible region and the set of LU-critical points of interval-valued nonconvex optimization problem. Several numerical experiments and the applications to emergency supplies distribution and nondeterministic fractional continuous static games are solved to illustrate the applicability of the proposed neurodynamic algorithm.

Growing Echo State Network With an Inverse-Free Weight Update Strategy

An echo state network (ESN) draws widespread attention and is applied in many scenarios. As the most typical approach for solving the ESN, the matrix inverse operation of high computational complexity is involved. However, in the modern big data era, addressing the heavy computational burden problem is necessary. In order to reduce the computational load, an inverse-free ESN (IFESN) is proposed for the first time in this article. Besides, an incremental IFESN is constructed to attain the network topology with theoretical proof on the training error's monotone decline property. Simulations and experiments are conducted on several numerical and real-world time-series benchmarks, and corresponding results indicate that the proposed model is superior to some existing models and possesses excellent practical application potential. The source code is publicly available at https://github.com/LongJin-lab/the-supplementary-file-for-CYB-E-2021-04-0944.

Explicit Linear Left-and-Right 5-Step Formulas With Zeroing Neural Network for Time-Varying Applications

In this article, being different from conventional time-discretization (simply called discretization) formulas, explicit linear left-and-right 5-step (ELLR5S) formulas with sixth-order precision are proposed. The general sixth-order ELLR5S formula with four variable parameters is developed first, and constraints of these four parameters are displayed to guarantee the zero stability, consistence, and convergence of the formula. Then, by choosing specific parameter values within constraints, eight specific sixth-order ELLR5S formulas are developed. The general sixth-order ELLR5S formula is further utilized to generate discrete zeroing neural network (DZNN) models for solving time-varying linear and nonlinear systems. For comparison, three conventional discretization formulas are also utilized. Theoretical analyses are presented to show the performance of ELLR5S formulas and DZNN models. Furthermore, abundant experiments, including three practical applications, that is, angle-of-arrival (AoA) localization and two redundant manipulators (PUMA560 manipulator and Kinova manipulator) control, are conducted. The synthesized results substantiate the efficacy and superiority of sixth-order ELLR5S formulas as well as the corresponding DZNN models.

Sparse Bayesian Learning Based on Collaborative Neurodynamic Optimization

Regression in a sparse Bayesian learning (SBL) framework is usually formulated as a global optimization problem with a nonconvex objective function and solved in a majorization-minimization framework where the solution quality and consistency depend heavily on the initial values of the used algorithm. In view of the shortcomings, this article presents an SBL algorithm based on collaborative neurodynamic optimization (CNO) for searching global optimal solutions to the global optimization problem. The CNO system consists of a population of recurrent neural networks (RNNs) where each RNN is convergent to a local optimum to the global optimization problem. Reinitialized repetitively via particle swarm optimization with exchanged local optima information, the RNNs iteratively improve their searching performance until reaching global convergence. The proposed CNO-based SBL algorithm is almost surely convergent to a global optimal solution to the formulated global optimization problem. Two applications with experimental results on sparse signal reconstruction and partial differential equation identification are elaborated to substantiate the superiority and efficacy of the proposed method in terms of solution optimality and consistency.

Robust k-WTA Network Generation, Analysis, and Applications to Multiagent Coordination

In this article, a robust k -winner-take-all ( k -WTA) neural network employing the saturation-allowed activation functions is designed and investigated to perform a k -WTA operation, and is shown to possess enhanced robustness to disturbance compared to existing k -WTA neural networks. Global convergence and robustness of the proposed k -WTA neural network are demonstrated through analysis and simulations. An application studied in detail is competitive multiagent coordination and dynamic task allocation, in which k active agents [among ] are allocated to execute a tracking task with the static m-k ones. This is implemented by adopting a distributed k -WTA network with limited communication, aided with a consensus filter. Simulation results demonstrating the system's efficacy and feasibility are presented.

A projection-based continuous-time algorithm for distributed optimization over multi-agent systems

Multi-agent systems are widely studied due to its ability of solving complex tasks in many fields, especially in deep reinforcement learning. Recently, distributed optimization problem over multi-agent systems has drawn much attention because of its extensive applications. This paper presents a projection-based continuous-time algorithm for solving convex distributed optimization problem with equality and inequality constraints over multi-agent systems. The distinguishing feature of such problem lies in the fact that each agent with private local cost function and constraints can only communicate with its neighbors. All agents aim to cooperatively optimize a sum of local cost functions. By the aid of penalty method, the states of the proposed algorithm will enter equality constraint set in fixed time and ultimately converge to an optimal solution to the objective problem. In contrast to some existed approaches, the continuous-time algorithm has fewer state variables and the testification of the consensus is also involved in the proof of convergence. Ultimately, two simulations are given to show the viability of the algorithm.

Two-Timescale Multilayer Recurrent Neural Networks for Nonlinear Programming

This article presents a neurodynamic approach to nonlinear programming. Motivated by the idea of sequential quadratic programming, a class of two-timescale multilayer recurrent neural networks is presented with neuronal dynamics in their output layer operating at a bigger timescale than in their hidden layers. In the two-timescale multilayer recurrent neural networks, the transient states in the hidden layer(s) undergo faster dynamics than those in the output layer. Sufficient conditions are derived on the convergence of the two-timescale multilayer recurrent neural networks to local optima of nonlinear programming problems. Simulation results of collaborative neurodynamic optimization based on the two-timescale neurodynamic approach on global optimization problems with nonconvex objective functions or constraints are discussed to substantiate the efficacy of the two-timescale neurodynamic approach.

A one-layer recurrent neural network for nonsmooth pseudoconvex optimization with quasiconvex inequality and affine equality constraints

As two important types of generalized convex functions, pseudoconvex and quasiconvex functions appear in many practical optimization problems. The lack of convexity poses some difficulties in solving pseudoconvex optimization with quasiconvex constraint functions. In this paper, we propose a one-layer recurrent neural network for solving such problems. We prove that the state of the proposed neural network is convergent from the feasible region to an optimal solution of the given optimization problem. We show that the proposed neural network has several advantages over the existing neural networks for pseudoconvex optimization. Specifically, the proposed neural network is applicable to optimization problems with quasiconvex inequality constraints as well as affine equality constraints. In addition, parameter matrix inversion is avoided and some assumptions on the objective function and inequality constraints in existing results are relaxed. We demonstrate the superior performance and characteristics of the proposed neural network with simulation results in three numerical examples.

Smoothing neural network for L0 regularized optimization problem with general convex constraints

In this paper, we propose a neural network modeled by a differential inclusion to solve a class of discontinuous and nonconvex sparse regression problems with general convex constraints, whose objective function is the sum of a convex but not necessarily differentiable loss function and L₀ regularization. We construct a smoothing relaxation function of L₀ regularization and propose a neural network to solve the considered problem. We prove that the solution of proposed neural network with any initial point satisfying linear equality constraints is global existent, bounded and reaches the feasible region in finite time and remains there thereafter. Moreover, the solution of proposed neural network is its slow solution and any accumulation point of it is a Clarke stationary point of the brought forward nonconvex smoothing approximation problem. In the box-constrained case, all accumulation points of the solution own a unified lower bound property and have a common support set. Except for a special case, any accumulation point of the solution is a local minimizer of the considered problem. In particular, the proposed neural network has a simple structure than most existing neural networks for solving the locally Lipschitz continuous but nonsmooth nonconvex problems. Finally, we give some numerical experiments to show the efficiency of proposed neural network.

A Compact Neural Network for Fused Lasso Signal Approximator

The fused lasso signal approximator (FLSA) is a vital optimization problem with extensive applications in signal processing and biomedical engineering. However, the optimization problem is difficult to solve since it is both nonsmooth and nonseparable. The existing numerical solutions implicate the use of several auxiliary variables in order to deal with the nondifferentiable penalty. Thus, the resulting algorithms are both time- and memory-inefficient. This paper proposes a compact neural network to solve the FLSA. The neural network has a one-layer structure with the number of neurons proportionate to the dimension of the given signal, thanks to the utilization of consecutive projections. The proposed neural network is stable in the Lyapunov sense and is guaranteed to converge globally to the optimal solution of the FLSA. Experiments on several applications from signal processing and biomedical engineering confirm the reasonable performance of the proposed neural network.

A Novel Multiagent Neurodynamic Approach to Constrained Distributed Convex Optimization

This paper considers a class of distributed convex optimization problems with constraints and gives a novel multiagent neurodynamic approach in continuous-time form. The considered distributed optimization is to search for a minimizer of the summation of nonsmooth convex functions on some agents, which have local general constraints. The proposed approach solves the objective function of each agent individually, and the state solutions of all agents reach consensus asymptotically under mild assumptions. In particular, the existence and boundedness of the global state solution to the dynamical system are guaranteed. Moreover, the state solution reaches the feasible region of equivalent optimization problem asymptotically and the output of each agent is convergent to the optimal solution set of the primal distributed problem. In contrast to the existing methods in a distributed manner, the proposed approach is more convenient for general constrained distributed problems and has low structure complexity which could narrow the bandwidth of communication. Finally, the proposed neurodynamic approach is applied to two numerical examples and a class of power system optimal load-sharing problems to support the theoretical results and its efficiency.

A Finite-Time Convergent and Noise-Rejection Recurrent Neural Network and Its Discretization for Dynamic Nonlinear Equations Solving

The so-called zeroing neural network (ZNN) is an effective recurrent neural network for solving dynamic problems including the dynamic nonlinear equations. There exist numerous unperturbed ZNN models that can converge to the theoretical solution of solvable nonlinear equations in infinity long or finite time. However, when these ZNN models are perturbed by external disturbances, the convergence performance would be dramatically deteriorated. To overcome this issue, this paper for the first time proposes a finite-time convergent ZNN with the noise-rejection capability to endure disturbances and solve dynamic nonlinear equations in finite time. In theory, the finite-time convergence and noise-rejection properties of the finite-time convergent and noise-rejection ZNN (FTNRZNN) are rigorously proved. For potential digital hardware realization, the discrete form of the FTNRZNN model is established based on a recently developed five-step finite difference rule to guarantee a high computational accuracy. The numerical results demonstrate that the discrete-time FTNRZNN can reject constant external noises. When perturbed by dynamic bounded or unbounded linear noises, the discrete-time FTNRZNN achieves the smallest steady-state errors in comparison with those generated by other discrete-time ZNN models that have no or limited ability to handle these noises. Discrete models of the FTNRZNN and the other ZNNs are comparatively applied to redundancy resolution of a robotic arm with superior positioning accuracy of the FTNRZNN verified.

A neurodynamic optimization approach for complex-variables programming problem

A neural network model upon differential inclusion is designed for solving the complex-variables convex programming, and the chain rule for real-valued function with the complex-variables is established in this paper. The model does not need to choose penalty parameters when applied to practical problems, which makes it easier to design. The result is obtained that its state reaches the feasible region in finite time. Furthermore, the convergence for its state to an optimal solution is proved. Some typical examples are shown for the effectiveness of the designed model.

Two Projection Neural Networks With Reduced Model Complexity for Nonlinear Programming

Recent reports show that projection neural networks with a low-dimensional state space can enhance computation speed obviously. This paper proposes two projection neural networks with reduced model dimension and complexity (RDPNNs) for solving nonlinear programming (NP) problems. Compared with existing projection neural networks for solving NP, the proposed two RDPNNs have a low-dimensional state space and low model complexity. Under the condition that the Hessian matrix of the associated Lagrangian function is positive semi-definite and positive definite at each Karush-Kuhn-Tucker point, the proposed two RDPNNs are proven to be globally stable in the sense of Lyapunov and converge globally to a point satisfying the reduced optimality condition of NP. Therefore, the proposed two RDPNNs are theoretically guaranteed to solve convex NP problems and a class of nonconvex NP problems. Computed results show that the proposed two RDPNNs have a faster computation speed than the existing projection neural networks for solving NP problems.

A Novel Neural Network for Solving Nonsmooth Nonconvex Optimization Problems

In this paper, a novel recurrent neural network (RNN) is presented to deal with a kind of nonsmooth nonconvex optimization problem in which the objective function may be nonsmooth and nonconvex, and the constraints include linear equations and convex inequations. Under certain suitable assumptions, from an arbitrary initial state, each solution to the proposed RNN exists globally and is bounded, and it enters the feasible region within a limited time. Moreover, the solution to the RNN with an arbitrary initial state can converge to the critical point set of the optimization problem. In particular, the RNN does not need the following: 1) abounded feasible region; 2) the computation of an exact penalty parameter; or 3) the initial state being chosen from a given bounded set. Numerical experiments are provided to show the effectiveness and advantages of the RNN.

A Novel Neurodynamic Approach to Constrained Complex-Variable Pseudoconvex Optimization

Complex-variable pseudoconvex optimization has been widely used in numerous scientific and engineering optimization problems. A neurodynamic approach is proposed in this paper for complex-variable pseudoconvex optimization problems subject to bound and linear equality constraints. An efficient penalty function is introduced to guarantee the boundedness of the state of the presented neural network, and make the state enter the feasible region of the considered optimization in finite time and stay there thereafter. The state is also shown to be convergent to an optimal point of the considered optimization. Compared with other neurodynamic approaches, the presented neural network does not need any penalty parameters, and has lower model complexity. Furthermore, some additional assumptions in other existing related neural networks are also removed in this paper, such as the assumption that the objective function is lower bounded over the equality constraint set and so on. Finally, some numerical examples and an application in beamforming formulation are provided.

A Complex Varying-Parameter Convergent-Differential Neural-Network for Solving Online Time-Varying Complex Sylvester Equation

A novel recurrent neural network, which is named as complex varying-parameter convergent-differential neural network (CVP-CDNN), is proposed in this paper for solving the time-varying complex Sylvester equation. Two kinds of CVP-CDNNs (i.e., CVP-CDNN Type I and Type II) are illustrated and proved to be effective. The proposed CVP-CDNNs can achieve super-exponential performance if the linear activation function is used. Some activation functions are considered for searching the better performance of the CVP-CDNN and the finite time convergence property of the CVP-CDNN with sign-bi-power activation function is testified. The convergence time of the CVP-CDNN with sign-bi-power activation function is shorter than complex fixed-parameter convergent-differential neural network (CFP-CDNN). Moreover, compared with traditional CFP-CDNN, better convergence performances of novel CVP-CDNN are verified by computer simulation comparisons.

A generalized neural network for distributed nonsmooth optimization with inequality constraint

In this paper, a generalized neural network with a novel auxiliary function is proposed to solve a distributed non-differentiable optimization over a multi-agent network. The constructed auxiliary function can ensure that the state solution of proposed neural network is bounded, and enters the inequality constraint set in finite time. Furthermore, the proposed neural network is demonstrated to reach consensus and ultimately converges to the optimal solution under several mild assumptions. Compared with the existing methods, the neural network proposed in this paper has a simple structure with a low amount of state variables, and does not depend on projection operator method for constrained distributed optimization. Finally, two numerical simulations and an application in power system are delineated to show the characteristics and practicability of the presented neural network.

Recurrent Neural Network Model: A New Strategy to Solve Fuzzy Matrix Games

This paper aims to investigate the fuzzy constrained matrix game (MG) problems using the concepts of recurrent neural networks (RNNs). To the best of our knowledge, this paper is the first in attempting to find a solution for fuzzy game problems using RNN models. For this purpose, a fuzzy game problem is reformulated into a weighting problem. Then, the Karush-Kuhn-Tucker (KKT) optimality conditions are provided for the weighting problem. The KKT conditions are used to propose the RNN model. Moreover, the Lyapunov stability and the global convergence of the RNN model are also confirmed. Finally, three illustrative examples are presented to demonstrate the effectiveness of this approach. The obtained results are compared with the results obtained by the previous approaches for solving fuzzy constrained MG.

A New Varying-Parameter Recurrent Neural-Network for Online Solution of Time-Varying Sylvester Equation

Solving Sylvester equation is a common algebraic problem in mathematics and control theory. Different from the traditional fixed-parameter recurrent neural networks, such as gradient-based recurrent neural networks or Zhang neural networks, a novel varying-parameter recurrent neural network, [called varying-parameter convergent-differential neural network (VP-CDNN)] is proposed in this paper for obtaining the online solution to the time-varying Sylvester equation. With time passing by, this kind of new varying-parameter neural network can achieve super-exponential performance. Computer simulation comparisons between the fixed-parameter neural networks and the proposed VP-CDNN via using different kinds of activation functions demonstrate that the proposed VP-CDNN has better convergence and robustness properties.

A Neurodynamic Model to Solve Nonlinear Pseudo-Monotone Projection Equation and Its Applications

In this paper, a neurodynamic model is given to solve nonlinear pseudo-monotone projection equation. Under pseudo-monotonicity condition and Lipschitz continuous condition, the projection neurodynamic model is proved to be stable in the sense of Lyapunov, globally convergent, globally asymptotically stable, and globally exponentially stable. Also, we show that, our new neurodynamic model is effective to solve the nonconvex optimization problems. Moreover, since monotonicity is a special case of pseudo-monotonicity and also since a co-coercive mapping is Lipschitz continuous and monotone, and a strongly pseudo-monotone mapping is pseudo-monotone, the neurodynamic model can be applied to solve a broader classes of constrained optimization problems related to variational inequalities, pseudo-convex optimization problem, linear and nonlinear complementarity problems, and linear and convex quadratic programming problems. Finally, several illustrative examples are stated to demonstrate the effectiveness and efficiency of our new neurodynamic model.