A one-layer recurrent neural network for constrained pseudoconvex optimization and its application for dynamic portfolio optimization

A smooth gradient approximation neural network for general constrained nonsmooth nonconvex optimization problems

Nonsmooth nonconvex optimization problems are pivotal in engineering practice due to the inherent nonsmooth and nonconvex characteristics of many real-world complex systems and models. The nonsmoothness and nonconvexity of the objective and constraint functions bring great challenges to the design and convergence analysis of the optimization algorithms. This paper presents a smooth gradient approximation neural network for such optimization problems, in which a smooth approximation technique with time-varying control parameter is introduced for handling nonsmooth nonregular objective functions. In addition, a hard comparator function is introduced to ensure that the state solution of the proposed neural network remains within the nonconvex inequality constraint sets. Any accumulation point of the state solution of the proposed neural network is proved to be a stationary point of the nonconvex optimization under consideration. Furthermore, the neural network demonstrates the ability to find optimal solutions for some generalized convex optimization problems. Compared with the related neural networks, the constructed neural network has weaker convergence conditions and simpler algorithm structure. Simulation results and an application in optimizing condition number verify the practical applicability of the presented algorithm.

Adaptive penalty-based neurodynamic approach for nonsmooth interval-valued optimization problem

The complex and diverse practical background drives this paper to explore a new neurodynamic approach (NA) to solve nonsmooth interval-valued optimization problems (IVOPs) constrained by interval partial order and more general sets. On the one hand, to deal with the uncertainty of interval-valued information, the LU-optimality condition of IVOPs is established through a deterministic form. On the other hand, according to the penalty method and adaptive controller, the interval partial order constraint and set constraint are punished by one adaptive parameter, which is a key enabler for the feasibility of states while having a lower solution space dimension and avoiding estimating exact penalty parameters. Through nonsmooth analysis and Lyapunov theory, the proposed adaptive penalty-based neurodynamic approach (APNA) is proven to converge to an LU-solution of the considered IVOPs. Finally, the feasibility of the proposed APNA is illustrated by numerical simulations and an investment decision-making problem.

Supervised Feature Selection via Collaborative Neurodynamic Optimization

As a crucial part of machine learning and pattern recognition, feature selection aims at selecting a subset of the most informative features from the set of all available features. In this article, supervised feature selection is at first formulated as a mixed-integer optimization problem with an objective function of weighted feature redundancy and relevancy subject to a cardinality constraint on the number of selected features. It is equivalently reformulated as a bound-constrained mixed-integer optimization problem by augmenting the objective function with a penalty function for realizing the cardinality constraint. With additional bilinear and linear equality constraints for realizing the integrality constraints, it is further reformulated as a bound-constrained biconvex optimization problem with two more penalty terms. Two collaborative neurodynamic optimization (CNO) approaches are proposed for solving the formulated and reformulated feature selection problems. One of the proposed CNO approaches uses a population of discrete-time recurrent neural networks (RNNs), and the other use a pair of continuous-time projection networks operating concurrently on two timescales. Experimental results on 13 benchmark datasets are elaborated to substantiate the superiority of the CNO approaches to several mainstream methods in terms of average classification accuracy with three commonly used classifiers.

An event-triggered collaborative neurodynamic approach to distributed global optimization

In this paper, we propose an event-triggered collaborative neurodynamic approach to distributed global optimization in the presence of nonconvexity. We design a projection neural network group consisting of multiple projection neural networks coupled via a communication network. We prove the convergence of the projection neural network group to Karush-Kuhn-Tucker points of a given global optimization problem. To reduce communication bandwidth consumption, we adopt an event-triggered mechanism to liaise with other neural networks in the group with the Zeno behavior being precluded. We employ multiple projection neural network groups for scattered searches and re-initialize their states using a meta-heuristic rule in the collaborative neurodynamic optimization framework. In addition, we apply the collaborative neurodynamic approach for distributed optimal chiller loading in a heating, ventilation, and air conditioning system.

A Projection Neural Network to Nonsmooth Constrained Pseudoconvex Optimization

In this article, a single-layer projection neural network based on penalty function and differential inclusion is proposed to solve nonsmooth pseudoconvex optimization problems with linear equality and convex inequality constraints, and the bound constraints, such as box and sphere types, in inequality constraints are processed by projection operator. By introducing the Tikhonov-like regularization method, the proposed neural network no longer needs to calculate the exact penalty parameters. Under mild assumptions, by nonsmooth analysis, it is proved that the state solution of the proposed neural network is always bounded and globally exists, and enters the constrained feasible region in a finite time, and never escapes from this region again. Finally, the state solution converges to an optimal solution for the considered optimization problem. Compared with some other existing neural networks based on subgradients, this algorithm eliminates the dependence on the selection of the initial point, which is a neural network model with a simple structure and low calculation load. Three numerical experiments and two application examples are used to illustrate the global convergence and effectiveness of the proposed neural network.

Decentralized Robust Portfolio Optimization Based on Cooperative-Competitive Multiagent Systems

This article addresses decentralized robust portfolio optimization based on multiagent systems. Decentralized robust portfolio optimization is first formulated as two distributed minimax optimization problems in a Markowitz return-risk framework. Cooperative-competitive multiagent systems are developed and applied for solving the formulated problems. The multiagent systems are shown to be able to reach consensuses in the expected stock prices and convergence in investment allocations through both intergroup and intragroup interactions. Experimental results of the multiagent systems with stock data from four major markets are elaborated to substantiate the efficacy of multiagent systems for decentralized robust portfolio optimization.

Sparse signal reconstruction via collaborative neurodynamic optimization

In this paper, we formulate a mixed-integer problem for sparse signal reconstruction and reformulate it as a global optimization problem with a surrogate objective function subject to underdetermined linear equations. We propose a sparse signal reconstruction method based on collaborative neurodynamic optimization with multiple recurrent neural networks for scattered searches and a particle swarm optimization rule for repeated repositioning. We elaborate on experimental results to demonstrate the outperformance of the proposed approach against ten state-of-the-art algorithms for sparse signal reconstruction.

Two-Timescale Multilayer Recurrent Neural Networks for Nonlinear Programming

This article presents a neurodynamic approach to nonlinear programming. Motivated by the idea of sequential quadratic programming, a class of two-timescale multilayer recurrent neural networks is presented with neuronal dynamics in their output layer operating at a bigger timescale than in their hidden layers. In the two-timescale multilayer recurrent neural networks, the transient states in the hidden layer(s) undergo faster dynamics than those in the output layer. Sufficient conditions are derived on the convergence of the two-timescale multilayer recurrent neural networks to local optima of nonlinear programming problems. Simulation results of collaborative neurodynamic optimization based on the two-timescale neurodynamic approach on global optimization problems with nonconvex objective functions or constraints are discussed to substantiate the efficacy of the two-timescale neurodynamic approach.

A one-layer recurrent neural network for nonsmooth pseudoconvex optimization with quasiconvex inequality and affine equality constraints

As two important types of generalized convex functions, pseudoconvex and quasiconvex functions appear in many practical optimization problems. The lack of convexity poses some difficulties in solving pseudoconvex optimization with quasiconvex constraint functions. In this paper, we propose a one-layer recurrent neural network for solving such problems. We prove that the state of the proposed neural network is convergent from the feasible region to an optimal solution of the given optimization problem. We show that the proposed neural network has several advantages over the existing neural networks for pseudoconvex optimization. Specifically, the proposed neural network is applicable to optimization problems with quasiconvex inequality constraints as well as affine equality constraints. In addition, parameter matrix inversion is avoided and some assumptions on the objective function and inequality constraints in existing results are relaxed. We demonstrate the superior performance and characteristics of the proposed neural network with simulation results in three numerical examples.

Cardinality-constrained portfolio selection based on collaborative neurodynamic optimization

Portfolio optimization is one of the most important investment strategies in financial markets. It is practically desirable for investors, especially high-frequency traders, to consider cardinality constraints in portfolio selection, to avoid odd lots and excessive costs such as transaction fees. In this paper, a collaborative neurodynamic optimization approach is presented for cardinality-constrained portfolio selection. The expected return and investment risk in the Markowitz framework are scalarized as a weighted Chebyshev function and the cardinality constraints are equivalently represented using introduced binary variables as an upper bound. Then cardinality-constrained portfolio selection is formulated as a mixed-integer optimization problem and solved by means of collaborative neurodynamic optimization with multiple recurrent neural networks repeatedly repositioned using a particle swarm optimization rule. The distribution of resulting Pareto-optimal solutions is also iteratively perfected by optimizing the weights in the scalarized objective functions based on particle swarm optimization. Experimental results with stock data from four major world markets are discussed to substantiate the superior performance of the collaborative neurodynamic approach to several exact and metaheuristic methods.

Smoothing neural network for L0 regularized optimization problem with general convex constraints

In this paper, we propose a neural network modeled by a differential inclusion to solve a class of discontinuous and nonconvex sparse regression problems with general convex constraints, whose objective function is the sum of a convex but not necessarily differentiable loss function and L₀ regularization. We construct a smoothing relaxation function of L₀ regularization and propose a neural network to solve the considered problem. We prove that the solution of proposed neural network with any initial point satisfying linear equality constraints is global existent, bounded and reaches the feasible region in finite time and remains there thereafter. Moreover, the solution of proposed neural network is its slow solution and any accumulation point of it is a Clarke stationary point of the brought forward nonconvex smoothing approximation problem. In the box-constrained case, all accumulation points of the solution own a unified lower bound property and have a common support set. Except for a special case, any accumulation point of the solution is a local minimizer of the considered problem. In particular, the proposed neural network has a simple structure than most existing neural networks for solving the locally Lipschitz continuous but nonsmooth nonconvex problems. Finally, we give some numerical experiments to show the efficiency of proposed neural network.

Minimax and Biobjective Portfolio Selection Based on Collaborative Neurodynamic Optimization

Portfolio selection is one of the important issues in financial investments. This article is concerned with portfolio selection based on collaborative neurodynamic optimization. The classic Markowitz mean-variance (MV) framework and its variant mean conditional value-at-risk (CVaR) are formulated as minimax and biobjective portfolio selection problems. Neurodynamic approaches are then applied for solving these optimization problems. For each of the problems, multiple neural networks work collaboratively to characterize the efficient frontier by means of particle swarm optimization (PSO)-based weight optimization. Experimental results with stock data from four major markets show the performance and characteristics of the collaborative neurodynamic approaches to the portfolio optimization problems.

A continuous-time neurodynamic approach and its discretization for distributed convex optimization over multi-agent systems

Distributed optimization problem (DOP) over multi-agent systems, which can be described by minimizing the sum of agents' local objective functions, has recently attracted widespread attention owing to its applications in diverse domains. In this paper, inspired by penalty method and subgradient descent method, a continuous-time neurodynamic approach is proposed for solving a DOP with inequality and set constraints. The state of continuous-time neurodynamic approach exists globally and converges to an optimal solution of the considered DOP. Comparisons reveal that the proposed neurodynamic approach can not only resolve more general convex DOPs, but also has lower dimension of solution space. Additionally, the discretization of the neurodynamic approach is also introduced for the convenience of implementation in practice. The iteration sequence of discrete-time method is also convergent to an optimal solution of DOP from any initial point. The effectiveness of the neurodynamic approach is verified by simulation examples and an application in L₁-norm minimization problem in the end.

Two-timescale neurodynamic approaches to supervised feature selection based on alternative problem formulations

Feature selection is a crucial step in data processing and machine learning. While many greedy and sequential feature selection approaches are available, a holistic neurodynamics approach to supervised feature selection is recently developed via fractional programming by minimizing feature redundancy and maximizing relevance simultaneously. In view that the gradient of the fractional objective function is also fractional, alternative problem formulations are desirable to obviate the fractional complexity. In this paper, the fractional programming problem formulation is equivalently reformulated as bilevel and bilinear programming problems without using any fractional function. Two two-timescale projection neural networks are adapted for solving the reformulated problems. Experimental results on six benchmark datasets are elaborated to demonstrate the global convergence and high classification performance of the proposed neurodynamic approaches in comparison with six mainstream feature selection approaches.

A neurodynamic optimization approach to supervised feature selection via fractional programming

Feature selection is an important issue in machine learning and data mining. Most existing feature selection methods are greedy in nature thus are prone to sub-optimality. Though some global feature selection methods based on unsupervised redundancy minimization can potentiate clustering performance improvements, their efficacy for classification may be limited. In this paper, a neurodynamics-based holistic feature selection approach is proposed via feature redundancy minimization and relevance maximization. An information-theoretic similarity coefficient matrix is defined based on multi-information and entropy to measure feature redundancy with respect to class labels. Supervised feature selection is formulated as a fractional programming problem based on the similarity coefficients. A neurodynamic approach based on two one-layer recurrent neural networks is developed for solving the formulated feature selection problem. Experimental results with eight benchmark datasets are discussed to demonstrate the global convergence of the neural networks and superiority of the proposed neurodynamic approach to several existing feature selection methods in terms of classification accuracy, precision, recall, and F-measure.

A neurodynamic optimization approach for complex-variables programming problem

A neural network model upon differential inclusion is designed for solving the complex-variables convex programming, and the chain rule for real-valued function with the complex-variables is established in this paper. The model does not need to choose penalty parameters when applied to practical problems, which makes it easier to design. The result is obtained that its state reaches the feasible region in finite time. Furthermore, the convergence for its state to an optimal solution is proved. Some typical examples are shown for the effectiveness of the designed model.

A Discrete-Time Neurodynamic Approach to Sparsity-Constrained Nonnegative Matrix Factorization

Sparsity is a desirable property in many nonnegative matrix factorization (NMF) applications. Although some level of sparseness of NMF solutions can be achieved by using regularization, the resulting sparsity depends highly on the regularization parameter to be valued in an ad hoc way. In this letter we formulate sparse NMF as a mixed-integer optimization problem with sparsity as binary constraints. A discrete-time projection neural network is developed for solving the formulated problem. Sufficient conditions for its stability and convergence are analytically characterized by using Lyapunov's method. Experimental results on sparse feature extraction are discussed to substantiate the superiority of this approach to extracting highly sparse features.

Projected Primal-Dual Dynamics for Distributed Constrained Nonsmooth Convex Optimization

A distributed nonsmooth convex optimization problem subject to a general type of constraint, including equality and inequality as well as bounded constraints, is studied in this paper for a multiagent network with a fixed and connected communication topology. To collectively solve such a complex optimization problem, primal-dual dynamics with projection operation are investigated under optimal conditions. For the nonsmooth convex optimization problem, a framework under the LaSalle's invariance principle from nonsmooth analysis is established, where the asymptotic stability of the primal-dual dynamics at an optimal solution is guaranteed. For the case where inequality and bounded constraints are not involved and the objective function is twice differentiable and strongly convex, the globally exponential convergence of the primal-dual dynamics is established. Finally, two simulations are provided to verify and visualize the theoretical results.

A Novel Neurodynamic Approach to Constrained Complex-Variable Pseudoconvex Optimization

Complex-variable pseudoconvex optimization has been widely used in numerous scientific and engineering optimization problems. A neurodynamic approach is proposed in this paper for complex-variable pseudoconvex optimization problems subject to bound and linear equality constraints. An efficient penalty function is introduced to guarantee the boundedness of the state of the presented neural network, and make the state enter the feasible region of the considered optimization in finite time and stay there thereafter. The state is also shown to be convergent to an optimal point of the considered optimization. Compared with other neurodynamic approaches, the presented neural network does not need any penalty parameters, and has lower model complexity. Furthermore, some additional assumptions in other existing related neural networks are also removed in this paper, such as the assumption that the objective function is lower bounded over the equality constraint set and so on. Finally, some numerical examples and an application in beamforming formulation are provided.

A generalized neural network for distributed nonsmooth optimization with inequality constraint

In this paper, a generalized neural network with a novel auxiliary function is proposed to solve a distributed non-differentiable optimization over a multi-agent network. The constructed auxiliary function can ensure that the state solution of proposed neural network is bounded, and enters the inequality constraint set in finite time. Furthermore, the proposed neural network is demonstrated to reach consensus and ultimately converges to the optimal solution under several mild assumptions. Compared with the existing methods, the neural network proposed in this paper has a simple structure with a low amount of state variables, and does not depend on projection operator method for constrained distributed optimization. Finally, two numerical simulations and an application in power system are delineated to show the characteristics and practicability of the presented neural network.

A Neurodynamic Approach to Distributed Optimization With Globally Coupled Constraints

In this paper, a distributed neurodynamic approach is proposed for constrained convex optimization. The objective function is a sum of local convex subproblems, whereas the constraints of these subproblems are coupled. Each local objective function is minimized individually with the proposed neurodynamic optimization approach. Through information exchange between connected neighbors only, all nodes can reach consensus on the Lagrange multipliers of all global equality and inequality constraints, and the decision variables converge to the global optimum in a distributed manner. Simulation results of two power system cases are discussed to substantiate the effectiveness and characteristics of the proposed approach.

A Collaborative Neurodynamic Approach to Multiple-Objective Distributed Optimization

This paper is concerned with multiple-objective distributed optimization. Based on objective weighting and decision space decomposition, a collaborative neurodynamic approach to multiobjective distributed optimization is presented. In the approach, a system of collaborative neural networks is developed to search for Pareto optimal solutions, where each neural network is associated with one objective function and given constraints. Sufficient conditions are derived for ascertaining the convergence to a Pareto optimal solution of the collaborative neurodynamic system. In addition, it is proved that each connected subsystem can generate a Pareto optimal solution when the communication topology is disconnected. Then, a switching-topology-based method is proposed to compute multiple Pareto optimal solutions for discretized approximation of Pareto front. Finally, simulation results are discussed to substantiate the performance of the collaborative neurodynamic approach. A portfolio selection application is also given.

Kinematic Control of Redundant Manipulators Using Neural Networks

Redundancy resolution is a critical problem in the control of robotic manipulators. Recurrent neural networks (RNNs), as inherently parallel processing models for time-sequence processing, are potentially applicable for the motion control of manipulators. However, the development of neural models for high-accuracy and real-time control is a challenging problem. This paper identifies two limitations of the existing RNN solutions for manipulator control, i.e., position error accumulation and the convex restriction on the projection set, and overcomes them by proposing two modified neural network models. Our method allows nonconvex sets for projection operations, and control error does not accumulate over time in the presence of noise. Unlike most works in which RNNs are used to process time sequences, the proposed approach is model-based and training-free, which makes it possible to achieve fast tracking of reference signals with superior robustness and accuracy. Theoretical analysis reveals the global stability of a system under the control of the proposed neural networks. Simulation results confirm the effectiveness of the proposed control method in both the position regulation and tracking control of redundant PUMA 560 manipulators.

A One-Layer Recurrent Neural Network for Pseudoconvex Optimization Problems With Equality and Inequality Constraints

Pseudoconvex optimization problem, as an important nonconvex optimization problem, plays an important role in scientific and engineering applications. In this paper, a recurrent one-layer neural network is proposed for solving the pseudoconvex optimization problem with equality and inequality constraints. It is proved that from any initial state, the state of the proposed neural network reaches the feasible region in finite time and stays there thereafter. It is also proved that the state of the proposed neural network is convergent to an optimal solution of the related problem. Compared with the related existing recurrent neural networks for the pseudoconvex optimization problems, the proposed neural network in this paper does not need the penalty parameters and has a better convergence. Meanwhile, the proposed neural network is used to solve three nonsmooth optimization problems, and we make some detailed comparisons with the known related conclusions. In the end, some numerical examples are provided to illustrate the effectiveness of the performance of the proposed neural network.

A Two-Time-Scale Neurodynamic Approach to Constrained Minimax Optimization

This paper presents a two-time-scale neurodynamic approach to constrained minimax optimization using two coupled neural networks. One of the recurrent neural networks is used for minimizing the objective function and another is used for maximization. It is shown that the coupled neurodynamic systems operating in two different time scales work well for minimax optimization. The effectiveness and characteristics of the proposed approach are illustrated using several examples. Furthermore, the proposed approach is applied for H_∞ model predictive control.

Convergence of batch gradient learning with smoothing regularization and adaptive momentum for neural networks

This paper presents new theoretical results on the backpropagation algorithm with smoothing [Formula: see text] regularization and adaptive momentum for feedforward neural networks with a single hidden layer, i.e., we show that the gradient of error function goes to zero and the weight sequence goes to a fixed point as n (n is iteration steps) tends to infinity, respectively. Also, our results are more general since we do not require the error function to be quadratic or uniformly convex, and neuronal activation functions are relaxed. Moreover, compared with existed algorithms, our novel algorithm can get more sparse network structure, namely it forces weights to become smaller during the training and can eventually removed after the training, which means that it can simply the network structure and lower operation time. Finally, two numerical experiments are presented to show the characteristics of the main results in detail.

A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach

This paper proposes a generalized Hopfield network for solving general constrained convex optimization problems. First, the existence and the uniqueness of solutions to the generalized Hopfield network in the Filippov sense are proved. Then, the Lie derivative is introduced to analyze the stability of the network using a differential inclusion. The optimality of the solution to the nonsmooth constrained optimization problems is shown to be guaranteed by the enhanced Fritz John conditions. The convergence rate of the generalized Hopfield network can be estimated by the second-order derivative of the energy function. The effectiveness of the proposed network is evaluated on several typical nonsmooth optimization problems and used to solve the hierarchical and distributed model predictive control four-tank benchmark.

A Neurodynamic Approach for Real-Time Scheduling via Maximizing Piecewise Linear Utility

In this paper, we study a set of real-time scheduling problems whose objectives can be expressed as piecewise linear utility functions. This model has very wide applications in scheduling-related problems, such as mixed criticality, response time minimization, and tardiness analysis. Approximation schemes and matrix vectorization techniques are applied to transform scheduling problems into linear constraint optimization with a piecewise linear and concave objective; thus, a neural network-based optimization method can be adopted to solve such scheduling problems efficiently. This neural network model has a parallel structure, and can also be implemented on circuits, on which the converging time can be significantly limited to meet real-time requirements. Examples are provided to illustrate how to solve the optimization problem and to form a schedule. An approximation ratio bound of 0.5 is further provided. Experimental studies on a large number of randomly generated sets suggest that our algorithm is optimal when the set is nonoverloaded, and outperforms existing typical scheduling strategies when there is overload. Moreover, the number of steps for finding an approximate solution remains at the same level when the size of the problem (number of jobs within a set) increases.