A new synthesis approach for feedback neural networks based on the perceptron training algorithm

Toward Optimal Synthesis of Discrete-Time Hopfield Neural Network

In this research brief, the relationship between eigenvectors (with {+1, -1} components) of a synaptic weight matrix W and the stable/anti-stable states of discrete-time Hopfield associative memory (HAM) is established. Also, the synthesis of W with desired stable/anti-stable states using spectral representation of W in even/odd dimension is discussed when the threshold vector is a non-zero vector. Freedom in choice of eigenvalues is capitalized to improve the noise immunity of the Hopfield neural network (HNN). Also, the problem of optimal synthesis of Hopfield Associative memory is formulated.

Analysis and Design of Multivalued High-Capacity Associative Memories Based on Delayed Recurrent Neural Networks

This article aims at analyzing and designing the multivalued high-capacity-associative memories based on recurrent neural networks with both asynchronous and distributed delays. In order to increase storage capacities, multivalued activation functions are introduced into associative memories. The stored patterns are retrieved by external input vectors instead of initial conditions, which can guarantee accurate associative memories by avoiding spurious equilibrium points. Some sufficient conditions are proposed to ensure the existence, uniqueness, and global exponential stability of the equilibrium point of neural networks with mixed delays. For neural networks with n neurons, m -dimensional input vectors, and 2k -valued activation functions, the autoassociative memories have (2k)ⁿ storage capacities and heteroassociative memories have min {(2k)ⁿ,(2k)^m} storage capacities. That is, the storage capacities of designed associative memories in this article are obviously higher than the 2ⁿ and min {2ⁿ,2^m} storage capacities of the conventional ones. Three examples are given to support the theoretical results.

A New Local Bipolar Autoassociative Memory Based on External Inputs of Discrete Recurrent Neural Networks With Time Delay

In this paper, local bipolar auto-associative memories are presented based on discrete recurrent neural networks with a class of gain type activation function. The weight parameters of neural networks are acquired by a set of inequalities without the learning procedure. The global exponential stability criteria are established to ensure the accuracy of the restored patterns by considering time delays and external inputs. The proposed methodology is capable of effectively overcoming spurious memory patterns and achieving memory capacity. The effectiveness, robustness, and fault-tolerant capability are validated by simulated experiments.In this paper, local bipolar auto-associative memories are presented based on discrete recurrent neural networks with a class of gain type activation function. The weight parameters of neural networks are acquired by a set of inequalities without the learning procedure. The global exponential stability criteria are established to ensure the accuracy of the restored patterns by considering time delays and external inputs. The proposed methodology is capable of effectively overcoming spurious memory patterns and achieving memory capacity. The effectiveness, robustness, and fault-tolerant capability are validated by simulated experiments.

New accurate and flexible design procedure for a stable KWTA continuous time network

The classical continuous time recurrent (Hopfield) network is considered and adapted to K -winner-take-all operation. The neurons are of sigmoidal type with a controllable gain G, an amplitude m and interconnected by the conductance p. The network is intended to process one by one a sequence of lists, each of them with N distinct elements, each of them squeezed to [0,I] admission interval, each of them having an imposed minimum separation between elements z(min). The network carries out: 1) a matching dynamic process between the order of list elements and the order of outputs, and 2) a binary type steady-state separation between K and K+1 outputs, the former surpassing a +ξ threshold and the later falling under the -ξ threshold. As a result, the machine will signal the ranks of the K largest elements of the list. To achieve 1), the initial condition of processing phase has to be placed in a computable θ -vicinity of zero-state. This requires a resetting procedure after each list. To achieve 2) the bias current M has to be within a certain interval computable from circuit parameters. In addition, the steady-state should be asymptotically stable. To these goals, we work with high gain and exploit the sigmoid properties and network symmetry. The various inequality type constraints between parameters are shown to be compatible and a neat synthesis procedure, simple and flexible, is given for the tanh sigmoid. It starts with the given parameters N, K, I, z(min), m and computes simple bounds of p, G, ξ, θ, and M. Numerical tests and comments reveal qualities and shortcomings of the method.

Multiobjective algebraic synthesis of neural control systems by implicit model following

The advantages brought about by using classical linear control theory in conjunction with neural approximators have long been recognized in the literature. In particular, using linear controllers to obtain the starting neural control design has been shown to be a key step for the successful development and implementation of adaptive-critic neural controllers. Despite their adaptive capabilities, neural controllers are often criticized for not providing the same performance and stability guarantees as classical linear designs. Therefore, this paper develops an algebraic synthesis procedure for designing dynamic output-feedback neural controllers that are closed-loop stable and meet the same performance objectives as any classical linear design. The performance synthesis problem is addressed by deriving implicit model-following algebraic relationships between model matrices, obtained from the classical design, and the neural control parameters. Additional linear matrix inequalities (LMIs) conditions for closed-loop exponential stability of the neural controller are derived using existing integral quadratic constraints (IQCs) for operators with repeated slope-restricted nonlinearities. The approach is demonstrated by designing a recurrent neural network controller for a highly maneuverable tailfin-controlled missile that meets multiple design objectives, including pole placement for transient tuning, H(infinity) and H(2) performance in the presence of parameter uncertainty, and command-input tracking.

Improved one-shot learning for feedforward associative memories with application to composite pattern association

The local identical index (LII) associative memory (AM) proposed by the authors in a previous paper is a one-shot feedforward structure designed to exhibit no spurious attractors. In this paper we relax the latter design constraint in exchange for enlarged basins of attraction and we develop a family of modified LII AM networks that exhibit improved performance, particularly in memorizing highly correlated patterns. The new algorithm meets the requirement of no spurious attractors only in a local sense. Finally, we show that the modified LII family of networks can accommodate composite patterns of any size by storing (memorizing) only the basic (prime) prototype patterns. The latter property translates to low learning complexity and a simple network structure with significant memory savings. Simulation studies and comparisons illustrate and support the the optical developments.

Analysis and Design of Associative Memories Based on Recurrent Neural Networks with Linear Saturation Activation Functions and Time-Varying Delays

In this letter, some sufficient conditions are obtained to guarantee recurrent neural networks with linear saturation activation functions, and time-varying delays have multiequilibria located in the saturation region and the boundaries of the saturation region. These results on pattern characterization are used to analyze and design autoassociative memories, which are directly based on the parameters of the neural networks. Moreover, a formula for the numbers of spurious equilibria is also derived. Four design procedures for recurrent neural networks with linear saturation activation functions and time-varying delays are developed based on stability results. Two of these procedures allow the neural network to be capable of learning and forgetting. Finally, simulation results demonstrate the validity and characteristics of the proposed approach.

Associative memory design using support vector machines

The relation existing between support vector machines (SVMs) and recurrent associative memories is investigated. The design of associative memories based on the generalized brain-state-in-a-box (GBSB) neural model is formulated as a set of independent classification tasks which can be efficiently solved by standard software packages for SVM learning. Some properties of the networks designed in this way are evidenced, like the fact that surprisingly they follow a generalized Hebb's law. The performance of the SVM approach is compared to existing methods with nonsymmetric connections, by some design examples.

Designing Asymmetric Hopfield-Type Associative Memory With Higher Order Hamming Stability

The problem of optimal asymmetric Hopfield-type associative memory (HAM) design based on perceptron-type learning algorithms is considered. It is found that most of the existing methods considered the design problem as either 1) finding optimal hyperplanes according to normal distance from the prototype vectors to the hyperplane surface or 2) obtaining weight matrix W = [w(ij)] by solving a constraint optimization problem. In this paper, we show that since the state space of the HAM consists of only bipolar patterns, i.e., V = (v1, v2, . . ., vN)T E {-1, +1}N, the basins of attraction around each prototype (training) vector should be expanded by using Hamming distance measure. For this reason, in this paper, the design problem is considered from a different point of view. Our idea is to systematically increase the size of the training set according to the desired basin of attraction around each prototype vector. We name this concept the higher order Hamming stability and show that conventional minimum-overlap algorithm can be modified to incorporate this concept. Experimental results show that the recall capability as well as the number of spurious memories are all improved by using the proposed method. Moreover, it is well known that setting all self-connections wiiVi to zero has the effect of reducing the number of spurious memories in state space. From the experimental results, we find that the basin width around each prototype vector can be enlarged by allowing nonzero diagonal elements on learning of the weight matrix W. If the magnitude of w(ii) is small for all i, then the condition w(ii) = OVi can be relaxed without seriously affecting the number of spurious memories in the state space. Therefore, the method proposed in this paper can be used to increase the basin width around each prototype vector with the cost of slightly increasing the number of spurious memories in the state space.

A synthesis procedure for associative memories based on space-varying cellular neural networks

In this paper, we consider the problem of realizing associative memories via space-varying CNNs (cellular neural networks). Based on some known results and a newly derived theorem for the CNN model, we propose a synthesis procedure for obtaining a space-varying CNN that can store given bipolar vectors with certain desirable properties. The major part of our synthesis procedure consists of solving generalized eigenvalue problems and/or linear matrix inequality problems, which can be efficiently solved by recently developed interior point methods. The validity of the proposed approach is illustrated by a design example.