Discrete-time neural network for fast solving large linear L1 estimation problems and its application to image restoration

Novel Continuous- and Discrete-Time Neural Networks for Solving Quadratic Minimax Problems With Linear Equality Constraints

This article presents two novel continuous- and discrete-time neural networks (NNs) for solving quadratic minimax problems with linear equality constraints. These two NNs are established based on the conditions of the saddle point of the underlying function. For the two NNs, a proper Lyapunov function is constructed so that they are stable in the sense of Lyapunov, and will converge to some saddle point(s) for any starting point under some mild conditions. Compared with the existing NNs for solving quadratic minimax problems, the proposed NNs require weaker stability conditions. The validity and transient behavior of the proposed models are illustrated by some simulation results.

Two Recurrent Neural Networks With Reduced Model Complexity for Constrained l₁-Norm Optimization

Because of the robustness and sparsity performance of least absolute deviation (LAD or l1 ) optimization, developing effective solution methods becomes an important topic. Recurrent neural networks (RNNs) are reported to be capable of effectively solving constrained l1 -norm optimization problems, but their convergence speed is limited. To accelerate the convergence, this article introduces two RNNs, in form of continuous- and discrete-time systems, for solving l1 -norm optimization problems with linear equality and inequality constraints. The RNNs are theoretically proven to be globally convergent to optimal solutions without any condition. With reduced model complexity, the two RNNs can significantly expedite constrained l1 -norm optimization. Numerical simulation results show that the two RNNs spend much less computational time than related RNNs and numerical optimization algorithms for linearly constrained l1 -norm optimization.

Matrix-Form Neural Networks for Complex-Variable Basis Pursuit Problem With Application to Sparse Signal Reconstruction

In this article, a continuous-time complex-valued projection neural network (CCPNN) in a matrix state space is first proposed for a general complex-variable basis pursuit problem. The proposed CCPNN is proved to be stable in the sense of Lyapunov and to be globally convergent to the optimal solution under the condition that the sensing matrix is not row full rank. Furthermore, an improved discrete-time complex projection neural network (IDCPNN) is proposed by discretizing the CCPNN model. The proposed IDCPNN consists of a two-step stop strategy to reduce the calculational cost. The proposed IDCPNN is theoretically guaranteed to be global convergent to the optimal solution. Finally, the proposed IDCPNN is applied to the reconstruction of sparse signals based on compressed sensing. Computed results show that the proposed IDCPNN is superior to related complex-valued neural networks and conventional basis pursuit algorithms in terms of solution quality and computation time.

A Compact Neural Network for Fused Lasso Signal Approximator

The fused lasso signal approximator (FLSA) is a vital optimization problem with extensive applications in signal processing and biomedical engineering. However, the optimization problem is difficult to solve since it is both nonsmooth and nonseparable. The existing numerical solutions implicate the use of several auxiliary variables in order to deal with the nondifferentiable penalty. Thus, the resulting algorithms are both time- and memory-inefficient. This paper proposes a compact neural network to solve the FLSA. The neural network has a one-layer structure with the number of neurons proportionate to the dimension of the given signal, thanks to the utilization of consecutive projections. The proposed neural network is stable in the Lyapunov sense and is guaranteed to converge globally to the optimal solution of the FLSA. Experiments on several applications from signal processing and biomedical engineering confirm the reasonable performance of the proposed neural network.

Neurodynamical classifiers with low model complexity

The recently proposed Minimal Complexity Machine (MCM) finds a hyperplane classifier by minimizing an upper bound on the Vapnik-Chervonenkis (VC) dimension. The VC dimension measures the capacity or model complexity of a learning machine. Vapnik's risk formula indicates that models with smaller VC dimension are expected to show improved generalization. On many benchmark datasets, the MCM generalizes better than SVMs and uses far fewer support vectors than the number used by SVMs. In this paper, we describe a neural network that converges to the MCM solution. We employ the MCM neurodynamical system as the final layer of a neural network architecture. Our approach also optimizes the weights of all layers in order to minimize the objective, which is a combination of a bound on the VC dimension and the classification error. We illustrate the use of this model for robust binary and multi-class classification. Numerical experiments on benchmark datasets from the UCI repository show that the proposed approach is scalable and accurate, and learns models with improved accuracies and fewer support vectors.

Two Projection Neural Networks With Reduced Model Complexity for Nonlinear Programming

Recent reports show that projection neural networks with a low-dimensional state space can enhance computation speed obviously. This paper proposes two projection neural networks with reduced model dimension and complexity (RDPNNs) for solving nonlinear programming (NP) problems. Compared with existing projection neural networks for solving NP, the proposed two RDPNNs have a low-dimensional state space and low model complexity. Under the condition that the Hessian matrix of the associated Lagrangian function is positive semi-definite and positive definite at each Karush-Kuhn-Tucker point, the proposed two RDPNNs are proven to be globally stable in the sense of Lyapunov and converge globally to a point satisfying the reduced optimality condition of NP. Therefore, the proposed two RDPNNs are theoretically guaranteed to solve convex NP problems and a class of nonconvex NP problems. Computed results show that the proposed two RDPNNs have a faster computation speed than the existing projection neural networks for solving NP problems.

Linear Regression Supporting Vector Machine and Hybrid LOG Filter-Based Image Restoration

The image restoration (IR) technique is a part of image processing to improve the quality of an image that is affected by noise and blur. Thus, IR is required to attain a better quality of image. In this paper, IR is performed using linear regression-based support vector machine (LR-SVM). This LR-SVM has two steps: training and testing. The training and testing stages have a distinct windowing process for extracting blocks from the images. The LR-SVM is trained through a block-by-block training sequence. The extracted block-by-block values of images are used to enhance the classification process of IR. In training, the imperfections on the image are easily identified by setting the target vectors as the original images. Then, the noisy image is given at LR-SVM testing, based on the original image restored from the dictionary. Finally, the image block from the testing stage is enhanced using the hybrid Laplacian of Gaussian (HLOG) filter. The denoising of the HLOG filter provides enhanced results by using block-by-block values. This proposed approach is named as LR-SVM-HLOG. A dataset used in this LR-SVM-HLOG method is the Berkeley Segmentation Database. The performance of LR-SVM-HLOG was analyzed as peak signal-to-noise ratio (PSNR) and structural similarity index. The PSNR values of the house and pepper image (color image) are 40.82 and 36.56 dB, respectively, which are higher compared to the inter- and intra-block sparse estimation method and block matching and three-dimensional filtering for color images at 20% noise.

Alternative continuous- and discrete-time neural networks for image restoration

This paper presents alternative continuous- and discrete-time neural networks for image restoration in real time by introducing new vectors and transforming its optimization conditions into a system of double projection equations. The proposed neural networks are shown to be stable in the sense of Lyapunov and convergent for any starting point. Compared with the existing neural networks for image restoration, the proposed models have the least neurons, a one-layer structure and the faster convergence, and is suitable to parallel implementation. The validity and transient behaviour of the proposed neural network is demonstrated by numerical examples.

Robust Regression Estimation Based on Low-Dimensional Recurrent Neural Networks

The robust Huber's M-estimator is widely used in signal and image processing, classification, and regression. From an optimization point of view, Huber's M-estimation problem is often formulated as a large-sized quadratic programming (QP) problem in view of its nonsmooth cost function. This paper presents a generalized regression estimator which minimizes a reduced-sized QP problem. The generalized regression estimator may be viewed as a significant generalization of several robust regression estimators including Huber's M-estimator. The performance of the generalized regression estimator is analyzed in terms of robustness and approximation accuracy. Furthermore, two low-dimensional recurrent neural networks (RNNs) are introduced for robust estimation. The two RNNs have low model complexity and enhanced computational efficiency. Finally, the experimental results of two examples and an application to image restoration are presented to substantiate superior performance of the proposed method over conventional algorithms for robust regression estimation in terms of approximation accuracy and convergence rate.

Modified Primal-Dual Neural Networks for Motion Control of Redundant Manipulators With Dynamic Rejection of Harmonic Noises

In recent decades, primal-dual neural networks, as a special type of recurrent neural networks, have received great success in real-time manipulator control. However, noises are usually ignored when neural controllers are designed based on them, and thus, they may fail to perform well in the presence of intensive noises. Harmonic noises widely exist in real applications and can severely affect the control accuracy. This work proposes a novel primal-dual neural network design that directly takes noise control into account. By taking advantage of the fact that the unknown amplitude and phase information of a harmonic signal can be eliminated from its dynamics, our deliberately designed neural controller is able to reach the accurate tracking of reference trajectories in a noisy environment. Theoretical analysis and extensive simulations show that the proposed controller stabilizes the control system polluted by harmonic noises and converges the position tracking error to zero. Comparisons show that our proposed solution consistently and significantly outperforms the existing primal-dual neural solutions as well as feedforward neural one and adaptive neural one for redundancy resolution of manipulators.

Augmented Lagrange Programming Neural Network for Localization Using Time-Difference-of-Arrival Measurements

A commonly used measurement model for locating a mobile source is time-difference-of-arrival (TDOA). As each TDOA measurement defines a hyperbola, it is not straightforward to compute the mobile source position due to the nonlinear relationship in the measurements. This brief exploits the Lagrange programming neural network (LPNN), which provides a general framework to solve nonlinear constrained optimization problems, for the TDOA-based localization. The local stability of the proposed LPNN solution is also analyzed. Simulation results are included to evaluate the localization accuracy of the LPNN scheme by comparing with the state-of-the-art methods and the optimality benchmark of Cramér-Rao lower bound.

Lagrange Programming Neural Network for TOA-Based Localization with Clock Asynchronization and Sensor Location Uncertainties

Source localization based on time of arrival (TOA) measurements in the presence of clock asynchronization and sensor position uncertainties is investigated in this paper. Different from the traditional numerical algorithms, a neural circuit named Lagrange programming neural network (LPNN) is employed to tackle the nonlinear and nonconvex constrained optimization problem of source localization. With the augmented term, two types of neural networks are developed from the original maximum likelihood functions based on the general framework provided by LPNN. The convergence and local stability of the proposed neural networks are analyzed in this paper. In addition, the Cramér-Rao lower bound is also derived as a benchmark in the presence of clock asynchronization and sensor position uncertainties. Simulation results verify the superior performance of the proposed LPNN over the traditional numerical algorithms and its robustness to resist the impact of a high level of measurement noise, clock asynchronization, as well as sensor position uncertainties.

A Neurodynamic Model to Solve Nonlinear Pseudo-Monotone Projection Equation and Its Applications

In this paper, a neurodynamic model is given to solve nonlinear pseudo-monotone projection equation. Under pseudo-monotonicity condition and Lipschitz continuous condition, the projection neurodynamic model is proved to be stable in the sense of Lyapunov, globally convergent, globally asymptotically stable, and globally exponentially stable. Also, we show that, our new neurodynamic model is effective to solve the nonconvex optimization problems. Moreover, since monotonicity is a special case of pseudo-monotonicity and also since a co-coercive mapping is Lipschitz continuous and monotone, and a strongly pseudo-monotone mapping is pseudo-monotone, the neurodynamic model can be applied to solve a broader classes of constrained optimization problems related to variational inequalities, pseudo-convex optimization problem, linear and nonlinear complementarity problems, and linear and convex quadratic programming problems. Finally, several illustrative examples are stated to demonstrate the effectiveness and efficiency of our new neurodynamic model.

Kinematic Control of Redundant Manipulators Using Neural Networks

Redundancy resolution is a critical problem in the control of robotic manipulators. Recurrent neural networks (RNNs), as inherently parallel processing models for time-sequence processing, are potentially applicable for the motion control of manipulators. However, the development of neural models for high-accuracy and real-time control is a challenging problem. This paper identifies two limitations of the existing RNN solutions for manipulator control, i.e., position error accumulation and the convex restriction on the projection set, and overcomes them by proposing two modified neural network models. Our method allows nonconvex sets for projection operations, and control error does not accumulate over time in the presence of noise. Unlike most works in which RNNs are used to process time sequences, the proposed approach is model-based and training-free, which makes it possible to achieve fast tracking of reference signals with superior robustness and accuracy. Theoretical analysis reveals the global stability of a system under the control of the proposed neural networks. Simulation results confirm the effectiveness of the proposed control method in both the position regulation and tracking control of redundant PUMA 560 manipulators.

Lagrange Programming Neural Network for Nondifferentiable Optimization Problems in Sparse Approximation

The major limitation of the Lagrange programming neural network (LPNN) approach is that the objective function and the constraints should be twice differentiable. Since sparse approximation involves nondifferentiable functions, the original LPNN approach is not suitable for recovering sparse signals. This paper proposes a new formulation of the LPNN approach based on the concept of the locally competitive algorithm (LCA). Unlike the classical LCA approach which is able to solve unconstrained optimization problems only, the proposed LPNN approach is able to solve the constrained optimization problems. Two problems in sparse approximation are considered. They are basis pursuit (BP) and constrained BP denoise (CBPDN). We propose two LPNN models, namely, BP-LPNN and CBPDN-LPNN, to solve these two problems. For these two models, we show that the equilibrium points of the models are the optimal solutions of the two problems, and that the optimal solutions of the two problems are the equilibrium points of the two models. Besides, the equilibrium points are stable. Simulations are carried out to verify the effectiveness of these two LPNN models.

State Estimation of Discrete-Time Switched Neural Networks With Multiple Communication Channels

In this paper, the state estimation problem for a class of discrete-time switched neural networks with modal persistent dwell time (MPDT) switching and mixed time delays is investigated. The considered switching law, not only generalizes the commonly studied dwell-time (DT) and average DT (ADT) switchings, but also further attaches mode-dependency to the persistent DT (PDT) switching that is shown to be more general. Multiple communication channels, which include one primary channel and multiredundant channels, are considered to coexist for the state estimation of underlying switched neural networks. The desired mode-dependent filters are designed such that the resulting filtering error system is exponentially mean-square stable with a guaranteed nonweighted generalized H₂ performance index. It is verified that better filtering performance index can be achieved as the number of channels to be used increases. The potential and effectiveness of the developed theoretical results are demonstrated via a numerical example.

Principal Component 2-D Long Short-Term Memory for Font Recognition on Single Chinese Characters

Chinese character font recognition (CCFR) has received increasing attention as the intelligent applications based on optical character recognition becomes popular. However, traditional CCFR systems do not handle noisy data effectively. By analyzing in detail the basic strokes of Chinese characters, we propose that font recognition on a single Chinese character is a sequence classification problem, which can be effectively solved by recurrent neural networks. For robust CCFR, we integrate a principal component convolution layer with the 2-D long short-term memory (2DLSTM) and develop principal component 2DLSTM (PC-2DLSTM) algorithm. PC-2DLSTM considers two aspects: 1) the principal component layer convolution operation helps remove the noise and get a rational and complete font information and 2) simultaneously, 2DLSTM deals with the long-range contextual processing along scan directions that can contribute to capture the contrast between character trajectory and background. Experiments using the frequently used CCFR dataset suggest the effectiveness of PC-2DLSTM compared with other state-of-the-art font recognition methods.

A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach

This paper proposes a generalized Hopfield network for solving general constrained convex optimization problems. First, the existence and the uniqueness of solutions to the generalized Hopfield network in the Filippov sense are proved. Then, the Lie derivative is introduced to analyze the stability of the network using a differential inclusion. The optimality of the solution to the nonsmooth constrained optimization problems is shown to be guaranteed by the enhanced Fritz John conditions. The convergence rate of the generalized Hopfield network can be estimated by the second-order derivative of the energy function. The effectiveness of the proposed network is evaluated on several typical nonsmooth optimization problems and used to solve the hierarchical and distributed model predictive control four-tank benchmark.

A Bi-Projection Neural Network for Solving Constrained Quadratic Optimization Problems

In this paper, a bi-projection neural network for solving a class of constrained quadratic optimization problems is proposed. It is proved that the proposed neural network is globally stable in the sense of Lyapunov, and the output trajectory of the proposed neural network will converge globally to an optimal solution. Compared with existing projection neural networks (PNNs), the proposed neural network has a very small model size owing to its bi-projection structure. Furthermore, an application to data fusion shows that the proposed neural network is very effective. Numerical results demonstrate that the proposed neural network is much faster than the existing PNNs.

Energy-to-peak state estimation for Markov jump RNNs with time-varying delays via nonsynchronous filter with nonstationary mode transitions

In this paper, the problem of energy-to-peak state estimation for a class of discrete-time Markov jump recurrent neural networks (RNNs) with randomly occurring nonlinearities (RONs) and time-varying delays is investigated. A practical phenomenon of nonsynchronous jumps between RNNs modes and desired mode-dependent filters is considered, and a nonstationary mode transition among the filters is used to model the nonsynchronous jumps to different degrees that are also mode dependent. The RONs are used to model a class of sector-like nonlinearities that occur in a probabilistic way according to a Bernoulli sequence. The time-varying delays are supposed to be mode dependent and unknown, but with known lower and upper bounds a priori. Sufficient conditions on the existence of the nonsynchronous filters are obtained such that the filtering error system is stochastically stable and achieves a prescribed energy-to-peak performance index. Further to the recent study on the class of nonsynchronous estimation problem, a monotonicity is observed in obtaining filtering performance index, while changing the degree of nonsynchronous jumps. A numerical example is presented to verify the theoretical findings.

Blind image quality assessment via deep learning

This paper investigates how to blindly evaluate the visual quality of an image by learning rules from linguistic descriptions. Extensive psychological evidence shows that humans prefer to conduct evaluations qualitatively rather than numerically. The qualitative evaluations are then converted into the numerical scores to fairly benchmark objective image quality assessment (IQA) metrics. Recently, lots of learning-based IQA models are proposed by analyzing the mapping from the images to numerical ratings. However, the learnt mapping can hardly be accurate enough because some information has been lost in such an irreversible conversion from the linguistic descriptions to numerical scores. In this paper, we propose a blind IQA model, which learns qualitative evaluations directly and outputs numerical scores for general utilization and fair comparison. Images are represented by natural scene statistics features. A discriminative deep model is trained to classify the features into five grades, corresponding to five explicit mental concepts, i.e., excellent, good, fair, poor, and bad. A newly designed quality pooling is then applied to convert the qualitative labels into scores. The classification framework is not only much more natural than the regression-based models, but also robust to the small sample size problem. Thorough experiments are conducted on popular databases to verify the model's effectiveness, efficiency, and robustness.

New discrete-time recurrent neural network proposal for quadratic optimization with general linear constraints

In this brief, the quadratic problem with general linear constraints is reformulated using the Wolfe dual theory, and a very simple discrete-time recurrent neural network is proved to be able to solve it. Conditions that guarantee global convergence of this network to the constrained minimum are developed. The computational complexity of the method is analyzed, and experimental work is presented that shows its high efficiency.