Hopfield model associative memory with nonzero-diagonal terms in memory matrix

Bayesian retrieval in associative memories with storage errors

It is well known that for finite-sized networks, onestep retrieval in the autoassociative Willshaw net is a suboptimal way to extract the information stored in the synapses. Iterative retrieval strategies are much better, but have hitherto only had heuristic justification. We show how they emerge naturally from considerations of probabilistic inference under conditions of noisy and partial input and a corrupted weight matrix. We start from the conditional probability distribution over possible patterns for retrieval. This contains all possible information that is available to an observer of the network and the initial input. Since this distribution is over exponentially many patterns, we use it to develop two approximate, but tractable, iterative retrieval methods. One performs maximum likelihood inference to find the single most likely pattern, using the (negative log of the) conditional probability as a Lyapunov function for retrieval. In physics terms, if storage errors are present, then the modified iterative update equations contain an additional antiferromagnetic interaction term and site dependent threshold values. The second method makes a mean field assumption to optimize a tractable estimate of the full conditional probability distribution. This leads to iterative mean field equations which can be interpreted in terms of a network of neurons with sigmoidal responses but with the same interactions and thresholds as in the maximum likelihood update equations. In the absence of storage errors, both models become very similiar to the Willshaw model, where standard retrieval is iterated using a particular form of linear threshold strategy.

Optical implementation of terminal-attractor-based associative memory

For the purpose of reducing the spurious states in a Hopfield neural net for associative memory, we invoke terminal attractors. In achieving the optical implementation of the terminal-attractor-based associative memory (TABAM) as described in this paper, we first prove the existence of the terminal-attractor model with binary neuron representation. We then present several one- and two-dimensional optical architectures for the TABAM. Finally, as an example, we experimentally demonstrate an inner-product optical neural model using liquid-crystal spatial light modulators with engineering approximations and discuss how to apply the inner-product model to build a two-dimensional parallel processing TABAM.

Neural network model for unequally distributed neuron states

A modified neural network model suitable for patterns with unequally distributed neuron states is proposed. The storage capacity and content addressability are greatly improved by adding a linear modification term to the interconnection weights of the Hopfield model. Computer simulations were performed to demonstrate the robustness of the modified model compared to the original. A gratingmodulated holographic hybrid system was employed for an optical demonstration. Experimental results are shown.

On the Characteristics of the Autoassociative Memory with Nonzero-Diagonal Terms in the Memory Matrix

A statistical method is applied to explore the unique characteristics of a certain class of neural network autoassociative memory with N neurons and first-order synaptic interconnections. The memory matrix is constructed to store M = αN vectors based on the outer-product learning algorithm. We theoretically prove that, by setting all the diagonal terms of the memory matrix to be M and letting the input error ratio ρ = 0, the probability of successful recall P_r steadily decreases as α increases, but as α increases past 1.0, P_r begins to increase slowly. When 0 < ρ ≤ 0.5, the network exhibits strong error-correction capability if α ≤ 0.15 and this capability is shown to rapidly decrease as α increases. The network essentially loses all its error-correction capability at α = 2, regardless of the value of ρ. When 0 < ρ ≤ 0.5, and under the constraint of P_r > 0.99, the tradeoff between the number of stable states and their attraction force is analyzed and the memory capacity is shown to be 0.15N at best.

Large-scale two-dimensional optical Hopfield associative memory using an incoherent optical free-space interconnection

An optically implemented two-dimensional Hopfield associative memory using an optical free-space interconnection constructed by time-domain polarization switching is described. The operation for 2304-element memory is also demonstrated.

Combined associative memories

One of the major problems with conventional associative memory models is that there exists a set of stable false and oscillatory states in addition to the true memories in the models. To solve this problem, a combined associative memory model is proposed. Digital simulation results show that the model is capable of picking out the false memories generated by conventional models and is more reliable in comparison with the conventional models.

Projection method formulations of Hopfield-type associative memory neural networks

We study certain neural network architectures, which admit to optical implementations, from the point of view of projection methods. This approach enables us to uncover some interesting facts about the stable states of such networks. For example, a Hopfield-type associative content-addressable memory (ACAM) using a hard-limiter nonlinearity has three kinds of possible stable states. If the hard-limiter is replaced by a unity-slope saturation nonlinearity, there are only two kinds of stable states. These and other results demonstrate the utility of using projection methods in analyzing the dynamic behavior of neural networks.

Optical implementation of the Hopfield neural network using multiple fiber nets

Associative memory based on the Hopfield model utilizing interconnections by multiple optical fiber nets is proposed and demonstrated experimentally. The coupling ratio from fiber to fiber represents the synaptic weight of the connection between units. The remarkable feature of the optical fiber neural networks is that global connections between 2-D units can be easily achieved, resulting in simultaneous multiplication of the weight matrix and input vector. In the experiment, multiple fiber nets having 25(2) connections for 5 x 5 binary patterns are prepared, and three patterns are stored and successfully retrieved.

Adaptive neural network in a hybrid optical/electronic architecture using lateral inhibition

We report the optical implementation of a neural network based on a nearest matched filter algorithm and extensive lateral inhibition. Extremely rapid learning is demonstrated in pattern recognition and autonomous control applications, without introducing processing artifacts such as spurious states and ambiguous solutions. The optical implementation is achieved with a reconfigurable, bipolar mask-type crossbar switch based on an inexpensive liquid crystal spatial light modulator.

Updating optical pseudoinverse associative memories

Selected algorithms for adding to and deleting from optical pseudoinverse associative memories are presented and compared. New realizations of pseudoinverse updating methods using vector inner product matrix bordering and reduced-dimensionality Karhunen-Loeve approximations (which have been used for updating optical filters) are described in the context of associative memories. Greville's theorem is reviewed and compared with the Widrow-Hoff algorithm. Kohonen's gradient projection method is expressed in a different form suitable for optical implementation. The data matrix memory is also discussed for comparison purposes. Memory size, speed and ease of updating, and key vector requirements are the comparison criteria used.

Statistical performance of outer-product associative memory models

A figure of merit, the probability a bit is correct after update, is used to evaluate the performance of randomly coded outer-product associative memory models. Networks with bipolar binary states and nonzero diagonal connections are shown to yield the best performance with respect to this figure of merit. A surprising result is that an all-positive network, one with binary states and positive connections, is superior to a standard Hopfield style network with binary states and bipolar connections. A prescription for the optimal threshold point for the all-positive network is given.

Reconfigurable bipolar analog optical crossbar switch

A differential technique is proposed for obtaining bipolar analog {-1 . . . 0 . . . 1} inputs, outputs, and connection weights in a mask-type optical crossbar switch. The technique compensates for the limitations of current component technology and permits the use of inexpensive, low-contrast spatial light modulators for reconfigurable switching matrices. The conclusions of this work may be extended to design improvements in a wide variety of optical computing implementations.

Optical inner-product implementation of neural networks models

The Hopfield neural network model is described in terms of inner product, i.e., as a matched filtering step followed by a pattern synthesis step. Optical implementation by two cascaded coherent filtering setups with holographic matched filters is described. Suitable encoding of information in the form of bipolar (positive or negative) amplitudes in one hologram and of non-negative amplitudes in the other allows one to deal only with non-negative quantities in the input and output planes, thereby avoiding use of multiple channels and coherent detection. The performance of this scheme is analytically evaluated. The above coding can moreover be adapted to the cases of nonzero average memorized states and to the higher-order models associated to Hopfield's algorithm. Numerical simulations and experimental results are presented to illustrate the analysis.