Analysis and Application of Matrix-Form Neural Networks for Fast Matrix-Variable Convex Optimization

Matrix-variable optimization is a generalization of vector-variable optimization and has been found to have many important applications. To reduce computation time and storage requirement, this article presents two matrix-form recurrent neural networks (RNNs), one continuous-time model and another discrete-time model, for solving matrix-variable optimization problems with linear constraints. The two proposed matrix-form RNNs have low complexity and are suitable for parallel implementation in terms of matrix state space. The proposed continuous-time matrix-form RNN can significantly generalize existing continuous-time vector-form RNN. The proposed discrete-time matrix-form RNN can be effectively used in blind image restoration, where the storage requirement and computational cost are largely reduced. Theoretically, the two proposed matrix-form RNNs are guaranteed to be globally convergent to the optimal solution under mild conditions. Computed results show that the proposed matrix-form RNN-based algorithm is superior to related vector-form RNN and matrix-form RNN-based algorithms, in terms of computation time.

A proximal neurodynamic model for a system of non-linear inverse mixed variational inequalities

In this article, we introduce a system of non-linear inverse mixed variational inequalities (SNIMVIs). We propose a proximal neurodynamic model (PNDM) for solving SNIMVIs, leveraging proximal mappings. The uniqueness of the continuous solution for the PNDM is proved by assuming Lipschitz continuity. Moreover, we establish the global asymptotic stability of equilibrium points of the PNDM, contingent upon Lipschitz continuity and strong monotonicity. Additionally, an iterative algorithm involving proximal mappings for solving the SNIMVIs is presented. Finally, we provide illustrative examples to support our main findings. Furthermore, we provide an example where the SNIMVIs violate the strong monotonicity condition and exhibit the divergence nature of the trajectories of the corresponding PNDM.

Novel Continuous- and Discrete-Time Neural Networks for Solving Quadratic Minimax Problems With Linear Equality Constraints

This article presents two novel continuous- and discrete-time neural networks (NNs) for solving quadratic minimax problems with linear equality constraints. These two NNs are established based on the conditions of the saddle point of the underlying function. For the two NNs, a proper Lyapunov function is constructed so that they are stable in the sense of Lyapunov, and will converge to some saddle point(s) for any starting point under some mild conditions. Compared with the existing NNs for solving quadratic minimax problems, the proposed NNs require weaker stability conditions. The validity and transient behavior of the proposed models are illustrated by some simulation results.

Zhang equivalency of inequation-to-inequation type for constraints of redundant manipulators

In solving specific problems, physical laws and mathematical theorems directly express the connections between variables with equations/inequations. At times, it could be extremely hard or not viable to solve these equations/inequations directly. The PE (principle of equivalence) is a commonly applied pragmatic method across multiple fields. PE transforms the initial equations/inequations into simplified equivalent equations/inequations that are more manageable to solve, allowing researchers to achieve their objectives. The problem-solving process in many fields benefits from the use of PE. Recently, the ZE (Zhang equivalency) framework has surfaced as a promising approach for addressing time-dependent optimization problems. This ZEF (ZE framework) consolidates constraints at different tiers, demonstrating its capacity for the solving of time-dependent optimization problems. To broaden the application of ZEF in time-dependent optimization problems, specifically in the domain of motion planning for redundant manipulators, the authors systematically investigate the ZEF-I2I (ZEF of the inequation-to-inequation) type. The study concentrates on transforming constraints (i.e., joint constraints and obstacles avoidance depicted in different tiers) into consolidated constraints backed by rigorous mathematical derivations. The effectiveness and applicability of the ZEF-I2I are verified through two optimization motion planning schemes, which consolidate constraints in the velocity-tier and acceleration-tier. Schemes are required to accomplish the goal of repetitive motion planning within constraints. The firstly presented optimization motion planning schemes are then reformulated as two time-dependent quadratic programming problems. Simulative experiments conducted on the basis of a six-joint redundant manipulator confirm the outstanding effectiveness of the firstly presented ZEF-I2I in achieving the goal of motion planning within constraints.

A Proximal Neurodynamic Network With Fixed-Time Convergence for Equilibrium Problems and Its Applications

This article proposes a novel fixed-time converging proximal neurodynamic network (FXPNN) via a proximal operator to deal with equilibrium problems (EPs). A distinctive feature of the proposed FXPNN is its better transient performance in comparison to most existing proximal neurodynamic networks. It is shown that the FXPNN converges to the solution of the corresponding EP in fixed-time under some mild conditions. It is also shown that the settling time of the FXPNN is independent of initial conditions and the fixed-time interval can be prescribed, unlike existing results with asymptotical or exponential convergence. Moreover, the proposed FXPNN is applied to solve composition optimization problems (COPs), l1 -regularized least-squares problems, mixed variational inequalities (MVIs), and variational inequalities (VIs). It is further shown, in the case of solving COPs, that the fixed-time convergence can be established via the Polyak-Lojasiewicz condition, which is a relaxation of the more demanding convexity condition. Finally, numerical examples are presented to validate the effectiveness and advantages of the proposed neurodynamic network.

An RNN-Based Algorithm for Decentralized-Partial-Consensus Constrained Optimization

This technical note proposes a decentralized-partial-consensus optimization (DPCO) problem with inequality constraints. The partial-consensus matrix originating from the Laplacian matrix is constructed to tackle the partial-consensus constraints. A continuous-time algorithm based on multiple interconnected recurrent neural networks (RNNs) is derived to solve the optimization problem. In addition, based on nonsmooth analysis and Lyapunov theory, the convergence of continuous-time algorithm is further proved. Finally, several examples demonstrate the effectiveness of main results.

A Novel Fixed-Time Converging Neurodynamic Approach to Mixed Variational Inequalities and Applications

This article proposes a novel fixed-time converging forward-backward-forward neurodynamic network (FXFNN) to deal with mixed variational inequalities (MVIs). A distinctive feature of the FXFNN is its fast and fixed-time convergence, in contrast to conventional forward-backward-forward neurodynamic network and projected neurodynamic network. It is shown that the solution of the proposed FXFNN exists uniquely and converges to the unique solution of the corresponding MVIs in fixed time under some mild conditions. It is also shown that the fixed-time convergence result obtained for the FXFNN is independent of initial conditions, unlike most of the existing asymptotical and exponential convergence results. Furthermore, the proposed FXFNN is applied in solving sparse recovery problems, variational inequalities, nonlinear complementarity problems, and min-max problems. Finally, numerical and experimental examples are presented to validate the effectiveness of the proposed neurodynamic network.

A collaborative neurodynamic optimization algorithm to traveling salesman problem

This paper proposed a collaborative neurodynamic optimization (CNO) method to solve traveling salesman problem (TSP). First, we construct a Hopfield neural network (HNN) with $$n \times n$$ n × n neurons for the n cities. Second, to ensure the convergence of continuous HNN (CHNN), we reformulate TSP to satisfy the convergence condition of CHNN and solve TSP by CHNN. Finally, a population of CHNNs is used to search for local optimal solutions of TSP and the globally optimal solution is obtained using particle swarm optimization. Experimental results show the effectiveness of the CNO approach for solving TSP.

Hash Bit Selection via Collaborative Neurodynamic Optimization With Discrete Hopfield Networks

Hash bit selection (HBS) aims to find the most discriminative and informative hash bits from a hash pool generated by using different hashing algorithms. It is usually formulated as a binary quadratic programming problem with an information-theoretic objective function and a string-length constraint. In this article, it is equivalently reformulated in the form of a quadratic unconstrained binary optimization problem by augmenting the objective function with a penalty function. The reformulated problem is solved via collaborative neurodynamic optimization (CNO) with a population of classic discrete Hopfield networks. The two most important hyperparameters of the CNO approach are determined based on Monte Carlo test results. Experimental results on three benchmark data sets are elaborated to substantiate the superiority of the collaborative neurodynamic approach to several existing methods for HBS.

A projection-based continuous-time algorithm for distributed optimization over multi-agent systems

Multi-agent systems are widely studied due to its ability of solving complex tasks in many fields, especially in deep reinforcement learning. Recently, distributed optimization problem over multi-agent systems has drawn much attention because of its extensive applications. This paper presents a projection-based continuous-time algorithm for solving convex distributed optimization problem with equality and inequality constraints over multi-agent systems. The distinguishing feature of such problem lies in the fact that each agent with private local cost function and constraints can only communicate with its neighbors. All agents aim to cooperatively optimize a sum of local cost functions. By the aid of penalty method, the states of the proposed algorithm will enter equality constraint set in fixed time and ultimately converge to an optimal solution to the objective problem. In contrast to some existed approaches, the continuous-time algorithm has fewer state variables and the testification of the consensus is also involved in the proof of convergence. Ultimately, two simulations are given to show the viability of the algorithm.

Two-Timescale Multilayer Recurrent Neural Networks for Nonlinear Programming

This article presents a neurodynamic approach to nonlinear programming. Motivated by the idea of sequential quadratic programming, a class of two-timescale multilayer recurrent neural networks is presented with neuronal dynamics in their output layer operating at a bigger timescale than in their hidden layers. In the two-timescale multilayer recurrent neural networks, the transient states in the hidden layer(s) undergo faster dynamics than those in the output layer. Sufficient conditions are derived on the convergence of the two-timescale multilayer recurrent neural networks to local optima of nonlinear programming problems. Simulation results of collaborative neurodynamic optimization based on the two-timescale neurodynamic approach on global optimization problems with nonconvex objective functions or constraints are discussed to substantiate the efficacy of the two-timescale neurodynamic approach.

An inertial neural network approach for robust time-of-arrival localization considering clock asynchronization

This paper presents an inertial neural network to solve the source localization optimization problem with l₁-norm objective function based on the time of arrival (TOA) localization technique. The convergence and stability of the inertial neural network are analyzed by the Lyapunov function method. An inertial neural network iterative approach is further used to find a better solution among the solutions with different inertial parameters. Furthermore, the clock asynchronization is considered in the TOA l₁-norm model for more general real applications, and the corresponding inertial neural network iterative approach is addressed. The numerical simulations and real data are both considered in the experiments. In the simulation experiments, the noise contains uncorrelated zero-mean Gaussian noise and uniform distributed outliers. In the real experiments, the data is obtained by using the ultra wide band (UWB) technology hardware modules. Whether or not there is clock asynchronization, the results show that the proposed approach always can find a more accurate source position compared with some of the existing algorithms, which implies that the proposed approach is more effective than the compared ones.

Multivehicle Task Assignment Based on Collaborative Neurodynamic Optimization With Discrete Hopfield Networks

This article presents a collaborative neurodynamic optimization (CNO) approach to multivehicle task assignments (TAs). The original combinatorial quadratic optimization problem for TA is reformulated as a quadratic unconstrained binary optimization (QUBO) problem with a quadratic utility function and a penalty function for handling load capacity and cooperation constraints. In the framework of CNO with a population of discrete Hopfield networks (DHNs), a TA algorithm is proposed for solving the formulated QUBO problem. Superior experimental results in four typical multivehicle operation scenarios are reported to substantiate the efficacy of the proposed neurodynamics-based TA approach.

Cardinality-constrained portfolio selection based on collaborative neurodynamic optimization

Portfolio optimization is one of the most important investment strategies in financial markets. It is practically desirable for investors, especially high-frequency traders, to consider cardinality constraints in portfolio selection, to avoid odd lots and excessive costs such as transaction fees. In this paper, a collaborative neurodynamic optimization approach is presented for cardinality-constrained portfolio selection. The expected return and investment risk in the Markowitz framework are scalarized as a weighted Chebyshev function and the cardinality constraints are equivalently represented using introduced binary variables as an upper bound. Then cardinality-constrained portfolio selection is formulated as a mixed-integer optimization problem and solved by means of collaborative neurodynamic optimization with multiple recurrent neural networks repeatedly repositioned using a particle swarm optimization rule. The distribution of resulting Pareto-optimal solutions is also iteratively perfected by optimizing the weights in the scalarized objective functions based on particle swarm optimization. Experimental results with stock data from four major world markets are discussed to substantiate the superior performance of the collaborative neurodynamic approach to several exact and metaheuristic methods.

A proximal neurodynamic model for solving inverse mixed variational inequalities

This paper proposes a proximal neurodynamic model (PNDM) for solving inverse mixed variational inequalities (IMVIs) based on the proximal operator. It is shown that the PNDM has a unique continuous solution under the condition of Lipschitz continuity (L-continuity). It is also shown that the equilibrium point of the proposed PNDM is asymptotically stable or exponentially stable under some mild conditions. Finally, three numerical examples are presented to illustrate effectiveness of the proposed PNDM.

Stability analysis of genetic regulatory networks via a linear parameterization approach

This paper investigates the problem of finite-time stability (FTS) for a class of delayed genetic regulatory networks with reaction-diffusion terms. In order to fully utilize the system information, a linear parameterization method is proposed. Firstly, by applying the Lagrange’s mean-value theorem, the linear parameterization method is applied to transform the nonlinear system into a linear one with time-varying bounded uncertain terms. Secondly, a new generalized convex combination lemma is proposed to dispose the relationship of bounded uncertainties with respect to their boundaries. Thirdly, sufficient conditions are established to ensure the FTS by resorting to Lyapunov Krasovskii theory, convex combination technique, Jensen’s inequality, linear matrix inequality, etc. Finally, the simulation verifications indicate the validity of the theoretical results.

Neurodynamical classifiers with low model complexity

The recently proposed Minimal Complexity Machine (MCM) finds a hyperplane classifier by minimizing an upper bound on the Vapnik-Chervonenkis (VC) dimension. The VC dimension measures the capacity or model complexity of a learning machine. Vapnik's risk formula indicates that models with smaller VC dimension are expected to show improved generalization. On many benchmark datasets, the MCM generalizes better than SVMs and uses far fewer support vectors than the number used by SVMs. In this paper, we describe a neural network that converges to the MCM solution. We employ the MCM neurodynamical system as the final layer of a neural network architecture. Our approach also optimizes the weights of all layers in order to minimize the objective, which is a combination of a bound on the VC dimension and the classification error. We illustrate the use of this model for robust binary and multi-class classification. Numerical experiments on benchmark datasets from the UCI repository show that the proposed approach is scalable and accurate, and learns models with improved accuracies and fewer support vectors.

Relaxed Inertial Tseng’s Type Method for Solving the Inclusion Problem with Application to Image Restoration

The relaxed inertial Tseng-type method for solving the inclusion problem involving a maximally monotone mapping and a monotone mapping is proposed in this article. The study modifies the Tseng forward-backward forward splitting method by using both the relaxation parameter, as well as the inertial extrapolation step. The proposed method follows from time explicit discretization of a dynamical system. A weak convergence of the iterates generated by the method involving monotone operators is given. Moreover, the iterative scheme uses a variable step size, which does not depend on the Lipschitz constant of the underlying operator given by a simple updating rule. Furthermore, the proposed algorithm is modified and used to derive a scheme for solving a split feasibility problem. The proposed schemes are used in solving the image deblurring problem to illustrate the applicability of the proposed methods in comparison with the existing state-of-the-art methods.

A Novel Neural Network for Generally Constrained Variational Inequalities

This paper presents a novel neural network for solving generally constrained variational inequality problems by constructing a system of double projection equations. By defining proper convex energy functions, the proposed neural network is proved to be stable in the sense of Lyapunov and converges to an exact solution of the original problem for any starting point under the weaker cocoercivity condition or the monotonicity condition of the gradient mapping on the linear equation set. Furthermore, two sufficient conditions are provided to ensure the stability of the proposed neural network for a special case. The proposed model overcomes some shortcomings of existing continuous-time neural networks for constrained variational inequality, and its stability only requires some monotonicity conditions of the underlying mapping and the concavity of nonlinear inequality constraints on the equation set. The validity and transient behavior of the proposed neural network are demonstrated by some simulation results.

A Two-Time-Scale Neurodynamic Approach to Constrained Minimax Optimization

This paper presents a two-time-scale neurodynamic approach to constrained minimax optimization using two coupled neural networks. One of the recurrent neural networks is used for minimizing the objective function and another is used for maximization. It is shown that the coupled neurodynamic systems operating in two different time scales work well for minimax optimization. The effectiveness and characteristics of the proposed approach are illustrated using several examples. Furthermore, the proposed approach is applied for H_∞ model predictive control.

Optimal Formation of Multirobot Systems Based on a Recurrent Neural Network

The optimal formation problem of multirobot systems is solved by a recurrent neural network in this paper. The desired formation is described by the shape theory. This theory can generate a set of feasible formations that share the same relative relation among robots. An optimal formation means that finding one formation from the feasible formation set, which has the minimum distance to the initial formation of the multirobot system. Then, the formation problem is transformed into an optimization problem. In addition, the orientation, scale, and admissible range of the formation can also be considered as the constraints in the optimization problem. Furthermore, if all robots are identical, their positions in the system are exchangeable. Then, each robot does not necessarily move to one specific position in the formation. In this case, the optimal formation problem becomes a combinational optimization problem, whose optimal solution is very hard to obtain. Inspired by the penalty method, this combinational optimization problem can be approximately transformed into a convex optimization problem. Due to the involvement of the Euclidean norm in the distance, the objective function of these optimization problems are nonsmooth. To solve these nonsmooth optimization problems efficiently, a recurrent neural network approach is employed, owing to its parallel computation ability. Finally, some simulations and experiments are given to validate the effectiveness and efficiency of the proposed optimal formation approach.

A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach

This paper proposes a generalized Hopfield network for solving general constrained convex optimization problems. First, the existence and the uniqueness of solutions to the generalized Hopfield network in the Filippov sense are proved. Then, the Lie derivative is introduced to analyze the stability of the network using a differential inclusion. The optimality of the solution to the nonsmooth constrained optimization problems is shown to be guaranteed by the enhanced Fritz John conditions. The convergence rate of the generalized Hopfield network can be estimated by the second-order derivative of the energy function. The effectiveness of the proposed network is evaluated on several typical nonsmooth optimization problems and used to solve the hierarchical and distributed model predictive control four-tank benchmark.

Nonsmooth Neural Network for Convex Time-Dependent Constraint Satisfaction Problems

This paper introduces a nonsmooth (NS) neural network that is able to operate in a time-dependent (TD) context and is potentially useful for solving some classes of NS-TD problems. The proposed network is named nonsmooth time-dependent network (NTN) and is an extension to a TD setting of a previous NS neural network for programming problems. Suppose C(t), t ≥ 0, is a nonempty TD convex feasibility set defined by TD inequality constraints. The constraints are in general NS (nondifferentiable) functions of the state variables and time. NTN is described by the subdifferential with respect to the state variables of an NS-TD barrier function and a vector field corresponding to the unconstrained dynamics. This paper shows that for suitable values of the penalty parameter, the NTN dynamics displays two main phases. In the first phase, any solution of NTN not starting in C(0) at t=0 is able to reach the moving set C(·) in finite time th , whereas in the second phase, the solution tracks the moving set, i.e., it stays within C(t) for all subsequent times t ≥ t(h). NTN is thus able to find an exact feasible solution in finite time and also to provide an exact feasible solution for subsequent times. This new and peculiar dynamics displayed by NTN is potentially useful for addressing some significant TD signal processing tasks. As an illustration, this paper discusses a number of examples where NTN is applied to the solution of NS-TD convex feasibility problems.

Neurodynamics-Based Robust Pole Assignment for High-Order Descriptor Systems

In this paper, a neurodynamic optimization approach is proposed for synthesizing high-order descriptor linear systems with state feedback control via robust pole assignment. With a new robustness measure serving as the objective function, the robust eigenstructure assignment problem is formulated as a pseudoconvex optimization problem. A neurodynamic optimization approach is applied and shown to be capable of maximizing the robust stability margin for high-order singular systems with guaranteed optimality and exact pole assignment. Two numerical examples and vehicle vibration control application are discussed to substantiate the efficacy of the proposed approach.

Robust pole assignment for synthesizing feedback control systems using recurrent neural networks

This paper presents a neurodynamic optimization approach to robust pole assignment for synthesizing linear control systems via state and output feedback. The problem is formulated as a pseudoconvex optimization problem with robustness measure: i.e., the spectral condition number as the objective function and linear matrix equality constraints for exact pole assignment. Two coupled recurrent neural networks are applied for solving the formulated problem in real time. In contrast to existing approaches, the exponential convergence of the proposed neurodynamics to global optimal solutions can be guaranteed even with lower model complexity in terms of the number of variables. Simulation results of the proposed neurodynamic approach for 11 benchmark problems are reported to demonstrate its superiority.

A one-layer projection neural network for nonsmooth optimization subject to linear equalities and bound constraints

This paper presents a one-layer projection neural network for solving nonsmooth optimization problems with generalized convex objective functions and subject to linear equalities and bound constraints. The proposed neural network is designed based on two projection operators: linear equality constraints, and bound constraints. The objective function in the optimization problem can be any nonsmooth function which is not restricted to be convex but is required to be convex (pseudoconvex) on a set defined by the constraints. Compared with existing recurrent neural networks for nonsmooth optimization, the proposed model does not have any design parameter, which is more convenient for design and implementation. It is proved that the output variables of the proposed neural network are globally convergent to the optimal solutions provided that the objective function is at least pseudoconvex. Simulation results of numerical examples are discussed to demonstrate the effectiveness and characteristics of the proposed neural network.