Information storage and retrieval in synchronous neural networks

External Stimuli on Neural Networks: Analytical and Numerical Approaches

Based on the behavior of living beings, which react mostly to external stimuli, we introduce a neural-network model that uses external patterns as a fundamental tool for the process of recognition. In this proposal, external stimuli appear as an additional field, and basins of attraction, representing memories, arise in accordance with this new field. This is in contrast to the more-common attractor neural networks, where memories are attractors inside well-defined basins of attraction. We show that this procedure considerably increases the storage capabilities of the neural network; this property is illustrated by the standard Hopfield model, which reveals that the recognition capacity of our model may be enlarged, typically, by a factor 102. The primary challenge here consists in calibrating the influence of the external stimulus, in order to attenuate the noise generated by memories that are not correlated with the external pattern. The system is analyzed primarily through numerical simulations. However, since there is the possibility of performing analytical calculations for the Hopfield model, the agreement between these two approaches can be tested—matching results are indicated in some cases. We also show that the present proposal exhibits a crucial attribute of living beings, which concerns their ability to react promptly to changes in the external environment. Additionally, we illustrate that this new approach may significantly enlarge the recognition capacity of neural networks in various situations; with correlated and non-correlated memories, as well as diluted, symmetric, or asymmetric interactions (synapses). This demonstrates that it can be implemented easily on a wide diversity of models.

Escaping from cycles through a glass transition

A random walk is performed over a disordered media composed of N sites random and uniformly distributed inside a d-dimensional hypercube. The walker cannot remain in the same site and hops to one of its n neighboring sites with a transition probability that depends on the distance D between sites according to a cost function E(D). The stochasticity level is parametrized by a formal temperature T. In the case T=0, the walk is deterministic and ergodicity is broken: the phase space is divided in a O(N) number of attractor basins of two-cycles that trap the walker. For d=1, analytic results indicate the existence of a glass transition at T(1)=1/2 as N--> infinity. Below T1, the average trapping time in two-cycles diverges and an out-of-equilibrium behavior appears. Similar glass transitions occur in higher dimensions when the right cost function is chosen. We also present some results for the statistics of distances for Poisson spatial point processes.

Results from the Gardner-Derrida-Mottishaw theory of associative memory

General computable formulas for overlap and the other parameters are derived from the Gardner-Derrida-Mottishaw theory of associative memory on the Little-Hopfield model. The overlap is expressed in terms of integral of many-dimensional Gaussian functions. A method of approximation is developed to make numerical computation easy. It is shown that the numerical results are totally in good agreement with simulation.

Notions of Associative Memory and Sparse Coding

This paper summarizes associative memory models and sparse representation of memory in these models. Important properties of the associative memory models are their storage capacity, basin of attraction, and the existence of spurious memories. Sparse coding and nonmonotonic output functions are proposed to improve them. Sparsely coded associative memory model has an extremely large storage capacity which diverges as the mean firing rate of memory patterns approaches 0. The storage capacity strongly depends on the shape of the output function as well as the mean firing rate, even in the case of monotonic output functions. Dynamical properties of the model are analyzed by means of a statistical neurodynamical method. We emphasize the necessity of a feedback mechanism to control the mean firing rate in the recall process. Recently, there have been some experimental results suggesting its existence in the brain. On the other hand, it has been shown that the storage capacity can be markedly improved by replacing the usual monotonic output function with a nonmonotonic one. Another remarkable property of the model with the nonmonotonic neurons is that it seems to have no, or almost no, spurious memory. An associative memory model using nonmonotonic modules with a feedforward inhibition is discussed. The modules consist of two types of threshold units, each of which has a different threshold and can be considered as a biologically plausible representation of the nonmonotonic output function. The above model is compared with the monotonic one. The difference in the storage capacity between the two models becomes small when the sparse patterns are stored. Finally, we discuss the biological plausibility of the discussed associative memory models and sparse coding. Copyright 1996 Elsevier Science Ltd.