Deep associative neural network for associative memory based on unsupervised representation learning

This paper presents a deep associative neural network (DANN) based on unsupervised representation learning for associative memory. In brain, the knowledge is learnt by associating different types of sensory data, such as image and voice. The associative memory models which imitate such a learning process have been studied for decades but with simpler architectures they fail to deal with large scale complex data as compared with deep neural networks. Therefore, we define a deep architecture consisting of a perception layer and hierarchical propagation layers. To learn the network parameters, we define a probabilistic model for the whole network inspired from unsupervised representation learning models. The model is optimized by a modified contrastive divergence algorithm with a novel iterated sampling process. After training, given a new data or corrupted data, the correct label or corrupted part is associated by the network. The DANN is able to achieve many machine learning problems, including not only classification, but also depicting the data given a label and recovering corrupted images. Experiments on MNIST digits and CIFAR-10 datasets demonstrate the learning capability of the proposed DANN.

Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than 200× lower energy as compared with a 45-nm CMOS ASIC design.

Performance analysis and comparison of a minimum interconnections direct storage model with traditional neural bidirectional memories

This study proposes an efficient and improved model of a direct storage bidirectional memory, improved bidirectional associative memory (IBAM), and emphasises the use of nanotechnology for efficient implementation of such large-scale neural network structures at a considerable lower cost reduced complexity, and less area required for implementation. This memory model directly stores the X and Y associated sets of M bipolar binary vectors in the form of (MxN(x)) and (MxN(y)) memory matrices, requires O(N) or about 30% of interconnections with weight strength ranging between +/-1, and is computationally very efficient as compared to sequential, intraconnected and other bidirectional associative memory (BAM) models of outer-product type that require O(N(2)) complex interconnections with weight strength ranging between +/-M. It is shown that it is functionally equivalent to and possesses all attributes of a BAM of outer-product type, and yet it is simple and robust in structure, very large scale integration (VLSI), optical and nanotechnology realisable, modular and expandable neural network bidirectional associative memory model in which the addition or deletion of a pair of vectors does not require changes in the strength of interconnections of the entire memory matrix. The analysis of retrieval process, signal-to-noise ratio, storage capacity and stability of the proposed model as well as of the traditional BAM has been carried out. Constraints on and characteristics of unipolar and bipolar binaries for improved storage and retrieval are discussed. The simulation results show that it has log(e) N times higher storage capacity, superior performance, faster convergence and retrieval time, when compared to traditional sequential and intraconnected bidirectional memories.

Reduced order multiport parallel and multidirectional neural associative memories

This paper proposes multiport parallel and multidirectional intraconnected associative memories of outer product type with reduced interconnections. Some new reduced order memory architectures such as k-directional and k-port parallel memories are suggested. These architectures are, also, very suitable for implementation of spatio-temporal sequences and multiassociative memories. It is shown that in the proposed memory architectures, a substational reduction in interconnections is achieved if the actual length of original N-bit long vectors is subdivided into k sublengths. Using these sublengths, submemory matrices, T ( s ) or W ( s ), are computed, which are then intraconnected to form k-port parallel or k-directional memories. The subdivisions of N-bit long vectors into k sublengths save ((k-1) x 100) / k % of interconnections. It is shown, by means of an example, that more than 80% reduction in interconnections is achieved. Minimum limit in bits on k as well as maximum limit on subdivisions in k is determined. The topologies of reduced interconnectivity developed in this paper are symmetric in structure and can be used to scale up to larger systems. The underlying principal of construction, storage and retrieval processes of such associative memories has been analyzed. The effect of complexity of different levels of reduced interconnectivity on the quality of retrieval, signal to noise ratio, and storage capacity has been investigated. The model possesses analogies to biological neural structures and digital parallel port memories commonly used in parallel and multiprocessing systems.

Generation and reshaping of sequences in neural systems

The generation of informational sequences and their reorganization or reshaping is one of the most intriguing subjects for both neuroscience and the theory of autonomous intelligent systems. In spite of the diversity of sequential activities of sensory, motor, and cognitive neural systems, they have many similarities from the dynamical point of view. In this review we discus the ideas, models, and mathematical image of sequence generation and reshaping on different levels of the neural hierarchy, i.e., the role of a sensory network dynamics in the generation of a motor program (hunting swimming of marine mollusk Clione), olfactory dynamical coding, and sequential learning and decision making. Analysis of these phenomena is based on the winnerless competition principle. The considered models can be a basis for the design of biologically inspired autonomous intelligent systems.

A sequential dynamic heteroassociative memory for multistep pattern recognition and one-to-many association

Bidirectional associative memories (BAMs) have been widely used for auto and heteroassociative learning. However, few research efforts have addressed the issue of multistep vector pattern recognition. We propose a model that can perform multi step pattern recognition without the need for a special learning algorithm, and with the capacity to learn more than two pattern series in the training set. The model can also learn pattern series of different lengths and, contrarily to previous models, the stimuli can be composed of gray-level images. The paper also shows that by adding an extra autoassociative layer, the model can accomplish one-to-many association, a task that was exclusive to feedforward networks with context units and error backpropagation learning.

Non-homogenous neural networks with chaotic recursive nodes: Connectivity and multi-assemblies structures in recursive processing elements architectures

This paper addresses recurrent neural architectures based on bifurcating nodes that exhibit chaotic dynamics, with local dynamics defined by first order parametric recursions. In the studied architectures, logistic recursive nodes interact through parametric coupling, they self organize, and the network evolves to global spatio-temporal period-2 attractors that encode stored patterns. The performance of associative memories arrangements is measured through the average error in pattern recovery, under several levels of prompting noise. The impact of the synaptic connections magnitude on architecture performance is analyzed in detail, through pattern recovery performance measures and basin of attraction characterization. The importance of a planned choice of the synaptic connections scale in RPEs architectures is shown. A strategy for minimizing pattern recovery degradation when the number of stored patterns increases is developed. Experimental results show the success of such strategy. Mechanisms for allowing the studied associative networks to deal with asynchronous changes in input patterns, and tools for the interconnection between different associative assemblies are developed. Finally, coupling in heterogeneous assemblies with diverse recursive maps is analyzed, and the associated synaptic connections are equated.