Dynamic heteroassociative neural memories

A new bidirectional heteroassociative memory encompassing correlational, competitive and topological properties

In this paper, we present a new recurrent bidirectional model that encompasses correlational, competitive and topological model properties. The simultaneous use of many classes of network behaviors allows for the unsupervised learning/categorization of perceptual patterns (through input compression) and the concurrent encoding of proximities in a multidimensional space. All of these operations are achieved within a common learning operation, and using a single set of defining properties. It is shown that the model can learn categories by developing prototype representations strictly from exposition to specific exemplars. Moreover, because the model is recurrent, it can reconstruct perfect outputs from incomplete and noisy patterns. Empirical exploration of the model's properties and performance shows that its ability for adequate clustering stems from: (1) properly distributing connection weights, and (2) producing a weight space with a low dispersion level (or higher density). In addition, since the model uses a sparse representation (k-winners), the size of topological neighborhood can be fixed, and no longer requires a decrease through time as was the case with classic self-organizing feature maps. Since the model's learning and transmission parameters are independent from learning trials, the model can develop stable fixed points in a constrained topological architecture, while being flexible enough to learn novel patterns.

A feedforward bidirectional associative memory

In contrast to conventional feedback bidirectional associative Memory (BAM) network models, a feedforward BAM network is developed based on a one-shot design algorithm of O(p(2)(n+m)) computational complexity, where p is the number of prototype pairs and n, m are the dimensions of the input/output bipolar vectors. The feedforward BAM is an n-p-m three-layer network of McCulloch-Pitts neurons with storage capacity 2(min{m,n}) and guaranteed perfect bidirectional recall. The overall network design procedure is fully scalable in the sense that any number p= or <2(min{m,n}) of bidirectional associations can be implemented. The prototype patterns may be arbitrarily correlated. With respect to inference performance, it is shown that the Hamming attractive radius of each prototype reaches the maximum possible value. Simulation studies and comparisons illustrate and support these theoretical developments.

A sequential dynamic heteroassociative memory for multistep pattern recognition and one-to-many association

Bidirectional associative memories (BAMs) have been widely used for auto and heteroassociative learning. However, few research efforts have addressed the issue of multistep vector pattern recognition. We propose a model that can perform multi step pattern recognition without the need for a special learning algorithm, and with the capacity to learn more than two pattern series in the training set. The model can also learn pattern series of different lengths and, contrarily to previous models, the stimuli can be composed of gray-level images. The paper also shows that by adding an extra autoassociative layer, the model can accomplish one-to-many association, a task that was exclusive to feedforward networks with context units and error backpropagation learning.

Improving Leung's bidirectional learning rule for associative memories

Leung (1994) introduced a perceptron-like learning rule to enhance the recall performance of bidirectional associative memories (BAMs). He proved that his so-called bidirectional learning scheme always yields a solution within a finite number of learning iterations in case that a solution exists. Unfortunately, in the setting of Leung a solution only exists in case that the training set is strongly linear separable by hyperplanes through the origin. We extend Leung's approach by considering conditionally strong linear separable sets allowing separating hyperplanes not containing the origin. Moreover, we deal with BAMs, which are generalized by defining so-called dilation and translation parameters enlarging their capacity, while leaving their complexity almost unaffected. The whole approach leads to a generalized bidirectional learning rule which generates BAMs with dilation and translation that perform perfectly on the training set in a case that the latter satisfies the conditionally strong linear separability assumption. Therefore, in the sense of Leung, we conclude with an optimal learning strategy which contains Leung's initial idea as a special case.

Improved bidirectional retrieval of sparse patterns stored by Hebbian learning

The Willshaw model is asymptotically the most efficient neural associative memory (NAM), but its finite version is hampered by high retrieval errors. Iterative retrieval has been proposed in a large number of different models to improve performance in auto-association tasks. In this paper, bidirectional retrieval for the hetero-associative memory task is considered: we define information efficiency as a general performance measure for bidirectional associative memory (BAM) and determine its asymptotic bound for the bidirectional Willshaw model. For the finite Willshaw model, an efficient new bidirectional retrieval strategy is proposed, the appropriate combinatorial model analysis is derived, and implications of the proposed sparse BAM for applications and brain theory are discussed. The distribution of the dendritic sum in the finite Willshaw model given by Buckingham and Willshaw [Buckingham, J., & Willshaw, D. (1992). Performance characteristics of associative nets. Network, 3, 407-414] allows no fast numerical evaluation. We derive a combinatorial formula with a highly reduced evaluation time that is used in the improved error analysis of the basic model and for estimation of the retrieval error in the naive model extension, where bidirectional retrieval is employed in the hetero-associative Willshaw model. The analysis rules out the naive BAM extension as a promising improvement. A new bidirectional retrieval algorithm - called crosswise bidirectional (CB) retrieval - is presented. The cross talk error is significantly reduced without employing more complex learning procedures or dummy augmentation in the pattern coding, as proposed in other refined BAM models [Wang, Y. F., Cruz, J. B., & Mulligan, J. H. (1990). Two coding strategies for bidirectional associative memory. IEEE Trans. Neural Networks, 1(1), 81-92; Leung, C.-S., Chan, L.-W., & Lai, E. (1995). Stability, capacity and statistical dynamics of second-order bidirectional associative memory. IEEE Trans. Syst. Man Cybern., 25(10), 1414-1424]. The improved performance of CB retrieval is shown by a combinatorial analysis of the first step and by simulation experiments: it allows very efficient hetero-associative mapping, as well as auto-associative completion for sparse patterns - the experimentally achieved information efficiency is close to the asymptotic bound. The different retrieval methods in the hetero-associative Willshaw matrix are discussed as Boolean linear optimization problems. The improved BAM model opens interesting new perspectives, for instance, in information retrieval it allows efficient data access providing segmentation of ambiguous user input, relevance feedback and relevance ranking. Finally, we discuss BAM models as functional models for reciprocal cortico-cortical pathways, and the implication of this for a more flexible version of Hebbian cell-assemblies.

Complexity preserving increase of the capacity of bidirectional associative memories by dilation and translation

In this paper, we show how to increase the capacity of Kosko-type bidirectional associative memories by introducing dilation and translation parameters in the pattern space. The essential point of this approach is that the increase of the capacity of the networks almost doesn't affect their complexity. The general idea of the proposed strategy is to tune the networks in such a way that they are better suited to store patterns which are highly correlated without touching their ability to recover easy-to-store information. A detailed example at the end of the paper shows how our approach works in practice.

Adaptive associative memories capable of pattern segmentation

This paper presents an adaptive type of associative memory (AAM) that can separate patterns from composite inputs which might be degraded by deficiency or noise and that can recover incomplete or noisy single patterns. The behavior of AAM is analyzed in terms of stability, giving the stable solutions (results of recall), and the recall of spurious memories (the undesired solutions) is shown to be greatly reduced compared with earlier types of associative memory that can perform pattern segmentation. Two conditions that guarantee the nonexistence of undesired solutions are also given. Results of computer experiments show that the performance of AAM is much better than that of the earlier types of associative memory in terms of pattern segmentation and pattern recovery.

Weighted learning of bidirectional associative memories by global minimization

A weighted learning algorithm for bidirectional associative memories (BAMs) by means of global minimization, where each desired pattern is weighted, is described. According to the cost function that measures the goodness of the BAM, the learning algorithm is formulated as a global minimization problem and solved by a gradient descent rule. The learning approach guarantees not only that each desired pattern is stored as a stable state, but also that the basin of attraction is constructed as large as possible around each desired pattern. The existence of the weights, the asymptotic stability of each desired pattern and its basin of attraction, and the convergence of the proposed learning algorithm are investigated in an analytic way. A large number of computer experiments are reported to demonstrate the efficiency of the learning rule.

Adaptive Ho-Kashyap rules for perceptron training

Three adaptive versions of the Ho-Kashyap perceptron training algorithm are derived based on gradient descent strategies. These adaptive Ho-Kashyap (AHK) training rules are comparable in their complexity to the LMS and perceptron training rules and are capable of adaptively forming linear discriminant surfaces that guarantee linear separability and of positioning such surfaces for maximal classification robustness. In particular, a derived version called AHK II is capable of adaptively identifying critical input vectors lying close to class boundaries in linearly separable problems. The authors extend this algorithm as AHK III, which adds the capability of fast convergence to linear discriminant surfaces which are good approximations for nonlinearly separable problems. This is achieved by a simple built-in unsupervised strategy which allows for the adaptive grading and discarding of input vectors causing nonseparability. Performance comparisons with LMS and perceptron training are presented.

Exact associative neural memory dynamics utilizing Boolean matrices

The exact dynamics of shallow loaded associative neural memories are generated and characterized. The Boolean matrix analysis approach is employed for the efficient generation of all possible state transition trajectories for parallel updated binary-state dynamic associative memories (DAMs). General expressions for the size of the basin of attraction of fundamental and oscillatory memories and the number of oscillatory and stable states are derived for discrete synchronous Hopfield DAMs loaded with one, two, or three even-dimensionality bipolar memory vectors having the same mutual Hamming distances between them. Spurious memories are shown to occur only if the number of stored patterns exceeds two in an even-dimensionality Hopfield memory. The effects of odd- versus even-dimensionality memory vectors on DAM dynamics and the effects of memory pattern encoding on DAM performance are tested. An extension of the Boolean matrix dynamics characterization technique to other, more complex DAMs is presented.