Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

Pattern recognition in the neocognitron is improved by neuronal adaptation

We demonstrate that equipping the neurons of Fukushima's neocognitron with the phenomenon that a neuron decreases its activity when repeatedly stimulated (adaptation) markedly improves the pattern discriminatory power of the network. By means of adaptation, circuits for extracting discriminating features develop preferentially. In the original neocognitron, in contrast, features shared by different patterns are preferentially learned, as connections required for extracting them are more frequently reinforced.

Pattern association and retrieval in a continuous neural system

This paper studies the behavior of a large body of neurons in the continuum limit. A mathematical characterization of such systems is obtained by approximating the inverse input-output nonlinearity of a cell (or an assembly of cells) by three adjustable linearized sections. The associative spatio-temporal patterns for storage in the neural system are obtained by using approaches analogous to solving space-time field equations in physics. A noise-reducing equation is also derived from this neural model. In addition, conditions that make a noisy pattern retrievable are identified. Based on these analyses, a visual cortex model is proposed and an exact characterization of the patterns that are storable in this cortex is obtained. Furthermore, we show that this model achieves pattern association that is invariant to scaling, translation, rotation and mirror-reflection.

Optical correlator production system neural net

A new neural net is described that can easily and cost-effectively accommodate multiple objects in the field of view in parallel. The use of a correlator achieves shift invariance and accommodates multiple objects in parallel. Distortion-invariant filters provide aspect-invariant distortion. Symbolic encoding, the use of generic object parts, and a production system neural net allow large class problems to be addressed. Optical laboratory data on the production system inputs are provided and emphasized. Test data assume binary inputs, although analog (probability) input neurons are possible.

Direct and Indirect Cooperation between Temporal and Parietal Networks for Invariant Visual Recognition

A new type of biologically inspired multilayered network is proposed to model the properties of the primate visual system with respect to invariant visual recognition (IVR). This model is based on 10 major neurobiological and psychological constraints. The first five constraints shape the architecture and properties of the network.

1. The network model has a Y-like double-branched multilayered architecture, with one input (the retina) and two parallel outputs, the “What” and the “Where,” which model, respectively, the temporal pathway, specialized for “object” identification, and the parietal pathway specialized for “spatial” localization.

2. Four processing layers are sufficient to model the main functional steps of primate visual system that transform the retinal information into prototypes (object-centered reference frame) in the “What” branch and into an oculomotor command in the “Where” branch.

3. The distribution of receptive field sizes within and between the two functional pathways provides an appropriate tradeoff between discrimination and invariant recognition capabilities.

4. The two outputs are represented by a population coding: the ocular command is computed as a population vector in the “Where” branch and the prototypes are coded in a “semidistributed” way in the “What” branch. In the intermediate associative steps, processing units learn to associate prototypes (through feedback connections) to component features (through feedforward ones).

5. The basic processing units of the network do not model single cells but model the local neuronal circuits that combine different information flows organized in separate cortical layers.

Such a biologically constrained model shows shift-invariant and size-invariant capabilities that resemble those of humans (psychological constraints):

6. During the Learning session, a set of patterns (26 capital letters and 2 geometric figures) are presented to the network: a single presentation of each pattern in one position (at the center) and with one size is sufficient to learn the corresponding prototypes (internal representations).

These patterns are thus presented in widely varying new sizes and positions during the Recognition session:

7. The “What” branch of the network succeeds in immediate recognition for patterns presented in the central zone of the retina with the learned size.

8. The recognition by the “What” branch is resistant to changes in size within a limited range of variation related to the distribution of receptive field (RF) sizes in the successive processing steps of this pathway.

9. Even when ocular movements are not allowed, the recognition capabilities of the “What” branch are unaffected by changing positions around the learned one. This significant shift-invariance of the “What” branch is also related to the distribution of RF sizes.

10. When varying both sizes and locations, the “What” and the “Where” branches cooperate for recognition: the location coding in the “Where” branch can command, under the control of the “What” branch, an ocular movement efficient to reset peripheral patterns toward the central zone of the retina until successful recognition.

This model results in predictions about anatomical connections and physiological interactions between temporal and parietal cortices.

Learning Invariance from Transformation Sequences

The visual system can reliably identify objects even when the retinal image is transformed considerably by commonly occurring changes in the environment. A local learning rule is proposed, which allows a network to learn to generalize across such transformations. During the learning phase, the network is exposed to temporal sequences of patterns undergoing the transformation. An application of the algorithm is presented in which the network learns invariance to shift in retinal position. Such a principle may be involved in the development of the characteristic shift invariance property of complex cells in the primary visual cortex, and also in the development of more complicated invariance properties of neurons in higher visual areas.

Simulating Visual Attention

Selective visual attention serializes the processing of stimulus data to make efficient use of limited processing resources in the human visual system. This paper describes a connectionist network that exhibits a variety of attentional phenomena reported by Treisman, Wolford, Duncan, and others. As demonstrated in several simulations, a hierarchical, multiscale network that uses feature arrays with strong lateral inhibitory connections provides responses in agreement with a number of prominent behaviors associated with visual attention. The overall network design is consistent with a range of data reported in the psychological literature, and with neurophysiol-ogical characteristics of primate vision.

Subtracting incoherent optical neuron model: analysis, experiment, and applications

To fully use the advantages of optics in optical neural networks, an incoherent optical neuron (ION) model is proposed. The main purpose of this model is to provide for the requisite subtraction of signals without the phase sensitivity of a fully coherent system and without the cumbrance of photon-electron conversion and electronic subtraction. The ION model can subtract inhibitory from excitatory neuron inputs by using two device responses. Functionally it accommodates positive and negative weights, excitatory and inhibitory inputs, non-negative neuron outputs, and can be used in a variety of neural network models. This technique can implement conventional inner-product neuron units and Grossberg's mass action law neuron units. Some implementation considerations, such as the effect of nonlinearities on device response, noise, and fan-in/fan-out capability, are discussed and simulated by computer. An experimental demonstration of optical excitation and inhibition on a 2-D array of neuron units using a single Hughes liquid crystal light valve is also reported.

Backpropagation Applied to Handwritten Zip Code Recognition

The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.

Unsupervised Learning

What use can the brain make of the massive flow of sensory information that occurs without any associated rewards or punishments? This question is reviewed in the light of connectionist models of unsupervised learning and some older ideas, namely the cognitive maps and working models of Tolman and Craik, and the idea that redundancy is important for understanding perception (Attneave 1954), the physiology of sensory pathways (Barlow 1959), and pattern recognition (Watanabe 1960). It is argued that (1) The redundancy of sensory messages provides the knowledge incorporated in the maps or models. (2) Some of this knowledge can be obtained by observations of mean, variance, and covariance of sensory messages, and perhaps also by a method called “minimum entropy coding.” (3) Such knowledge may be incorporated in a model of “what usually happens” with which incoming messages are automatically compared, enabling unexpected discrepancies to be immediately identified. (4) Knowledge of the sort incorporated into such a filter is a necessary prerequisite of ordinary learning, and a representation whose elements are independent makes it possible to form associations with logical functions of the elements, not just with the elements themselves.

Neural network model for selective attention in visual pattern recognition and associative recall

A neural network model of selective attention is discussed. When two patterns or more are presented simultaneously, the model successively pays selective attention to each one, segmenting it from the rest and recognizing it separately. In the presence of noise or defects, the model can recall the complete pattern in which the noise has been eliminated and the defects corrected. These operations can be successfully performed regardless of deformation of the input patterns. This is an improved version of the earlier model proposed by the author: the ability of segmentation is improved by lateral inhibition.

Self-organizing neural network models for visual pattern recognition

Two neural network models for visual pattern recognition are discussed. The first model, called a "neocognitron", is a hierarchical multilayered network which has only afferent synaptic connections. It can acquire the ability to recognize patterns by "learning-without-a-teacher": the repeated presentation of a set of training patterns is sufficient, and no information about the categories of the patterns is necessary. The cells of the highest stage eventually become "gnostic cells", whose response shows the final result of the pattern-recognition of the network. Pattern recognition is performed on the basis of similarity in shape between patterns, and is not affected by deformation, nor by changes in size, nor by shifts in the position of the stimulus pattern. The second model has not only afferent but also efferent synaptic connections, and is endowed with the function of selective attention. The afferent and the efferent signals interact with each other in the hierarchical network: the efferent signals, that is, the signals for selective attention, have a facilitating effect on the afferent signals, and at the same time, the afferent signals gate efferent signal flow. When a complex figure, consisting of two patterns or more, is presented to the model, it is segmented into individual patterns, and each pattern is recognized separately. Even if one of the patterns to which the models is paying selective attention is affected by noise or defects, the model can "recall" the complete pattern from which the noise has been eliminated and the defects corrected.

Computing with neural circuits: a model

A new conceptual framework and a minimization principle together provide an understanding of computation in model neural circuits. The circuits consist of nonlinear graded-response model neurons organized into networks with effectively symmetric synaptic connections. The neurons represent an approximation to biological neurons in which a simplified set of important computational properties is retained. Complex circuits solving problems similar to those essential in biology can be analyzed and understood without the need to follow the circuit dynamics in detail. Implementation of the model with electronic devices will provide a class of electronic circuits of novel form and function.

A synaptic modification algorithm in consideration of the generation of rhythmic oscillation in a ring neural network

In consideration of the generation of bursts of nerve impulses (that is, rhythmic oscillation in impulse density) in the ring neural network, a synaptic modification algorithm is newly proposed. Rhythmic oscillation generally occurs in the regular ring network with feedback inhibition and in fact such signals can be observed in the real nervous system. Since, however, various additional connections can cause a disturbance which easily extinguishes the rhythmic oscillation in the network, some function for maintaining the rhythmic oscillation is to be expected to exist in the synapses if such signals play an important part in the nervous system. Our preliminary investigation into the rhythmic oscillation in the regular ring network has led to the selection of the parameters, that is, the average membrane potential (AMP) and the average impulse density (AID) in the synaptic modification algorithm, where the decrease of synaptic strength is supposed to be essential. This synaptic modification algorithm using AMP and AID enables both the rhythmic oscillation and the nonoscillatory state to be dealt with in the algorithm without distinction. Simulation demonstrates cases in which the algorithm catches and holds the rhythmic oscillation in the disturbed ring network where the rhythmic oscillation was previously extinguished.

A hierarchical neural network model for associative memory

A hierarchical neural network model with feedback interconnections, which has the function of associative memory and the ability to recognize patterns, is proposed. The model consists of a hierarchical multi-layered network to which efferent connections are added, so as to make positive feedback loops in pairs with afferent connections. The cell-layer at the initial stage of the network is the input layer which receives the stimulus input and at the same time works as an output layer for associative recall. The deepest layer is the output layer for pattern-recognition. Pattern-recognition is performed hierarchically by integrating information by converging afferent paths in the network. For the purpose of associative recall, the integrated information is again distributed to lower-order cells by diverging efferent paths. These two operations progress simultaneously in the network. If a fragment of a training pattern is presented to the network which has completed its self-organization, the entire pattern will gradually be recalled in the initial layer. If a stimulus consisting of a number of training patterns superposed is presented, one pattern gradually becomes predominant in the recalled output after competition between the patterns, and the others disappear. At about the same time when the recalled pattern reaches a steady state in the initial layer, in the deepest layer of the network, a response is elicited from the cell corresponding to the category of the finally-recalled pattern. Once a steady state has been reached, the response of the network is automatically extinguished by inhibitory signals from a steadiness-detecting cell.(ABSTRACT TRUNCATED AT 250 WORDS)

A neural network model for cognitive activity

A consideration of the storage of information as an energized neuronal state leads to the development of a new type of neural network model which is capable of pattern recognition, concept formation and recognition of patterns of events in time. The network consists of several layers of cells, each cell representing by connections from the lower levels some combination of features or concepts. Information travels toward higher layers by such connections during an association phase, and then reverses during a recognition phase, where higher-order concepts can redirect the flow to more appropriate elements, revising the perception of the environment. This permits a more efficient method of distinguishing closely-related patterns and also permits the formation of negative associations, which is a likely requirement for formation of "abstract" concepts.