Neural networks for computation: number representations and programming complexity

Parallel, self-organizing, hierarchical neural networks

A new neural-network architecture called the parallel, self-organizing, hierarchical neural network (PSHNN) is presented. The new architecture involves a number of stages in which each stage can be a particular neural network (SNN). At the end of each stage, error detection is carried out, and a number of input vectors are rejected. Between two stages there is a nonlinear transformation of input vectors rejected by the previous stage. The new architecture has many desirable properties, such as optimized system complexity (in the sense of minimized self-organizing number of stages), high classification accuracy, minimized learning and recall times, and truly parallel architectures in which all stages operate simultaneously without waiting for data from other stages during testing. The experiments performed indicated the superiority of the new architecture over multilayered networks with back-propagation training.

Neural networks for shortest path computation and routing in computer networks

The application of neural networks to the optimum routing problem in packet-switched computer networks, where the goal is to minimize the network-wide average time delay, is addressed. Under appropriate assumptions, the optimum routing algorithm relies heavily on shortest path computations that have to be carried out in real time. For this purpose an efficient neural network shortest path algorithm that is an improved version of previously suggested Hopfield models is proposed. The general principles involved in the design of the proposed neural network are discussed in detail. Its computational power is demonstrated through computer simulations. One of the main features of the proposed model is that it will enable the routing algorithm to be implemented in real time and also to be adaptive to changes in link costs and network topology.

Optimality in mono- and multisensory map formation

In the struggle for survival in a complex and dynamic environment, nature has developed a multitude of sophisticated sensory systems. In order to exploit the information provided by these sensory systems, higher vertebrates reconstruct the spatio-temporal environment from each of the sensory systems they have at their disposal. That is, for each modality the animal computes a neuronal representation of the outside world, a monosensory neuronal map. Here we present a universal framework that allows to calculate the specific layout of the involved neuronal network by means of a general mathematical principle, viz., stochastic optimality. In order to illustrate the use of this theoretical framework, we provide a step-by-step tutorial of how to apply our model. In so doing, we present a spatial and a temporal example of optimal stimulus reconstruction which underline the advantages of our approach. That is, given a known physical signal transmission and rudimental knowledge of the detection process, our approach allows to estimate the possible performance and to predict neuronal properties of biological sensory systems. Finally, information from different sensory modalities has to be integrated so as to gain a unified perception of reality for further processing, e.g., for distinct motor commands. We briefly discuss concepts of multimodal interaction and how a multimodal space can evolve by alignment of monosensory maps.

"Optimal" Hopfield network for combinatorial optimization with linear cost function

An "optimal" Hopfield network is presented for many of combinatorial optimization problems with linear cost function. It is proved that a vertex of the network state hypercube is asymptotically stable if and only if it is an optimal solution to the problem. That is, one can always obtain an optimal solution whenever the network converges to a vertex. In this sense, this network can be called the "optimal" Hopfield network. It is also shown through simulations of assignment problems that this network obtains optimal or nearly optimal solutions more frequently than other familiar Hopfield networks.

Scheduling multiprocessor job with resource and timing constraints using neural networks

The Hopfield neural network is extensively applied to obtaining an optimal/feasible solution in many different applications such as the traveling salesman problem (TSP), a typical discrete combinatorial problem. Although providing rapid convergence to the solution, TSP frequently converges to a local minimum. Stochastic simulated annealing is a highly effective means of obtaining an optimal solution capable of preventing the local minimum. This important feature is embedded into a Hopfield neural network to derive a new technique, i.e., mean field annealing. This work applies the Hopfield neural network and the normalized mean field annealing technique, respectively, to resolve a multiprocessor problem (known to be a NP-hard problem) with no process migration, constrained times (execution time and deadline) and limited resources. Simulation results demonstrate that the derived energy function works effectively for this class of problems.

Improvements of Complex-Valued Hopfield Associative Memory by Using Generalized Projection Rules

In this letter, new design methods for the complex-valued multistate Hopfield associative memories (CVHAMs) are presented. We show that the well-known projection rule proposed by Personnaz et al. can be generalized to complex domain such that the weight matrix of the CVHAM can be designed by using a simple and effective method. The stability of the proposed CVHAM is analyzed by using energy function approach which shows that in synchronous update mode the proposed model is guaranteed to converge to a fixed point from any given initial state. Moreover, the projection geometry of the generalized projection rule (GPR) is discussed. In order to enhance the recall capability, a strategy of eliminating the spurious memories is also reported. The validity and the performance of the proposed methods are investigated by computer simulation.

A novel neural network for the synthesis of antennas and microwave devices

A novel artificial neural network (SYNTHESIS-ANN) is presented, which has been designed for computationally intensive problems and applied to the optimization of antennas and microwave devices. The antenna example presented is optimized with respect to voltage standing-wave ratio, bandwidth, and frequency of operation. A simple microstrip transmission line problem is used to further describe the ANN effectiveness, in which microstrip line width is optimized with respect to line impedance. The ANNs exploit a unique number representation of input and output data in conjunction with a more standard neural network architecture. An ANN consisting of a heteroassociative memory provided a very efficient method of computing necessary geometrical values for the antenna when used in conjunction with a new randomization process. The number representation used provides significant insight into this new method of fault-tolerant computing. Further work is needed to evaluate the potential of this new paradigm.

Grating projection system for surface contour measurement

A grating projection system is a low-cost surface contour measurement technique that can be applied to a wide range of applications. There has been a resurgence of interest in the technique in recent years because of developments in computer hardware and image processing algorithms. We describe a method that projects a phase-shifted grating through a lens on an object surface. The deformed grating image on the object surface is captured by a CCD camera for subsequent analysis. Phase variation is achieved by a linear translation stage on which the grating is mounted. We compare the experimental results with the test results using a mechanical stylus method.

A multivalued bidirectional associative memory operating on a complex domain

A bidirectional associative memory (BAM) with complex states and connection weights is investigated in this paper. The states are represented by quantization values defined on the unit circle of the complex plane. A given Lyapunov function indicates that the proposed complex domain BAM (CDBAM) is bidirectionally stable no matter which operation (synchronous or asynchronous) is used. We also prove that all the equilibrium (fixed) points of CDBAM correspond to local energy minima so that the design problem can be solved by a gradient descent algorithm. Finally, a gradient descent algorithm in the complex domain is derived to design the weight matrix of CDBAM. Several computer simulations are performed to illustrate the validity, capacity, attractivity and the applications of the CDBAM.

Power distribution in two-dimensional optical network channels

The power distribution in two-dimensional optical network channels is analyzed. The maximum number of allowable channels as determined by the characteristics of optical detector is identified, in particular, for neural-network and wavelet-transform applications.

Existence and stability of a unique equilibrium in continuous-valued discrete-time asynchronous Hopfield neural networks

It is shown that the assumption of D-stability of the interconnection matrix, together with the standard assumptions on the activation functions, guarantee the existence of a unique equilibrium under a synchronous mode of operation as well as a class of asynchronous modes. For the synchronous mode, these assumptions are also shown to imply local asymptotic stability of the equilibrium. For the asynchronous mode of operation, two results are derived. First, it is shown that symmetry and stability of the interconnection matrix guarantee local asymptotic stability of the equilibrium under a class of asynchronous modes-this is referred to as local absolute asymptotic stability. Second, it is shown that, under the standard assumptions, if the nonnegative matrix whose elements are the absolute values of the corresponding elements of the interconnection matrix is stable, then the equilibrium is globally absolutely asymptotically stable under a class of asynchronous modes. The results obtained are discussed from the points of view of their applications, robustness, and their relationship to earlier results.

Neural networks for process scheduling in real-time communication systems

This paper presents the use of Hopfield-type neural networks for process scheduling in the area of factory automation, where bus-based communication systems, called FieldBuses, are widely used to connect sensors and actuators to the control systems. We show how it overcomes the problem of the computational complexity of the algorithmic solution. The neural model proposed allows several processes to be scheduled simultaneously; the time required is polynomial with respect to the number of processes being scheduled. This feature allows real-time process scheduling and makes it possible for the scheduling table to adapt to changes in process control features. The paper presents the neural model for process scheduling and assesses its computational complexity, pointing out the drastic reduction in the time needed to generate a schedule as compared with the algorithmic scheduling solution. Finally, the authors propose an on-line scheduling strategy based on the neural model which can achieve real-time adaptation of the scheduling table to changes in the manufacturing environment.

Improvement on performance of modified Hopfield neural network for image restoration

By analyzing the same inequality ||u*||(1)=/<(1/2)trace(T), the authors conclude that a severely blurred image is generally restored less accurately than a mildly blurred one by the modified Hopfield neural network. This conclusion is the opposite of the statement made in Paik and Katsaggelos (1992). The authors also propose an improved new algorithm. Simulation results show that the SNRs of the images restored by the algorithm are higher by 3 to 8 db than those restored by the algorithm in Paik and Katsaggelos and the streaks in the restored images are obviously suppressed by the algorithm.

Reversal-input superposing technique for all-optical neural networks

The proposed technique for optical neural networks can perform all the neural operations in a positive range. Bipolar weights of the neurons are represented by unipolar weights with a positive constant. By superposing the reversal inputs to the weighted sums, we can perform subtraction in a neuron by the nonlinear output function with a negative offset constant. This means that the number of processing elements needed in the proposed system is the same as that of neurons in the original neural network model. An experimental neural system is demonstrated for verification of this technique. The Hopfield model is adapted as an example of the neural networks implemented in the experimental neural system.

Sequential and parallel image restoration: neural network implementations

Sequential and parallel image restoration algorithms and their implementations on neural networks are proposed. For images degraded by linear blur and contaminated by additive white Gaussian noise, maximum a posteriori (MAP) estimation and regularization theory lead to the same high dimension convex optimization problem. The commonly adopted strategy (in using neural networks for image restoration) is to map the objective function of the optimization problem into the energy of a predefined network, taking advantage of its energy minimization properties. Departing from this approach, we propose neural implementations of iterative minimization algorithms which are first proved to converge. The developed schemes are based on modified Hopfield (1985) networks of graded elements, with both sequential and parallel updating schedules. An algorithm supported on a fully standard Hopfield network (binary elements and zero autoconnections) is also considered. Robustness with respect to finite numerical precision is studied, and examples with real images are presented.

Gray-level discrete associative memory

A gray-level discrete associative-memory neural network based on object decomposition and composition is presented. By decomposing a gray-level pattern into bipolar/binary subpatterns, a gray-level discrete associative memory can be constructed from the composition of the subpattern channel results. Preprocessing for removing dc bias and normalizing the gray-level scale is performed on the input gray-level pattern. This eliminates the mismatching and saturation problems caused by bias level, which shifts the pattern gray levels throughout the pattern. Computer-simulation and optical-experimental results for a gray-level interpattern association model are shown to be consistent with the theoretical model.

Image restoration using a modified Hopfield network

A modified Hopfield neural network model for regularized image restoration is presented. The proposed network allows negative autoconnections for each neuron. A set of algorithms using the proposed neural network model is presented, with various updating modes: sequential updates; n-simultaneous updates; and partially asynchronous updates. The sequential algorithm is shown to converge to a local minimum of the energy function after a finite number of iterations. Since an algorithm which updates all n neurons simultaneously is not guaranteed to converge, a modified algorithm is presented, which is called a greedy algorithm. Although the greedy algorithm is not guaranteed to converge to a local minimum, the l (1) norm of the residual at a fixed point is bounded. A partially asynchronous algorithm is presented, which allows a neuron to have a bounded time delay to communicate with other neurons. Such an algorithm can eliminate the synchronization overhead of synchronous algorithms.

Hopfield model with multistate neurons and its optoelectronic implementation

A Hopfield model with multistate neurons is described. This model is able to deal with multivalued problems such as restoring a degraded image with gray level equivalent to that produced by the two-states model, but needs many less neurons and interconnections. The performance of this model is compared with that of the linear model, and it is concluded that the multistate neuron model can produce convergence more quickly. Finally, a hybrid system for the implementation of this model is discussed.

Two-dimensional programmable optical neural network

A 2-D hybrid optical neural network using a high resolution video monitor as a programmable associative memory is proposed. Experiments and computer simulations of the system have been conducted. The high resolution and large dynamic range of the video monitor enable us to implement a hybrid neural network with more neurons and more accurate operation. The system operates in a high speed asynchronous mode due to the parallel feedback loop. The programmability of the system permits the use of orthogonal projection and multilevel recognition algorithms to increase the robustness and storage capacity of the network.

Multitarget tracking with cubic energy optical neural nets

A neural net processor and its optical realization are described for a multitarget tracking application. A cubic energy function results and a new optical neural processor is required. Initial simulation data are presented.

Comparison of artificial neural networks with pattern recognition and image processing

This paper presents a comparison between the field of artificial neural network and the field of image processing and pattern recognition. It shows that some of the adaptive processing algorithms for pattern recognition and image processing, in terms of neural networks, can be seen as adaptive heteroassociative and autoassociative memories, respectively. The similarities and differences between these two fields are addressed.

Optoelectronic analogs of self-programming neural nets: architecture and methodologies for implementing fast stochastic learning by simulated annealing

Self-organization and learning is a distinctive feature of neural nets and processors that sets them apart from conventional approaches to signal processing. It leads to self-programmability which alleviates the problem of programming complexity in artificial neural nets. In this paper architectures for partitioning an optoelectronic analog of a neural net into distinct layers with prescribed interconnectivity pattern to enable stochastic learning by simulated annealing in the context of a Boltzmann machine are presented. Stochastic learning is of interest because of its relevance to the role of noise in biological neural nets. Practical considerations and methodologies for appreciably accelerating stochastic learning in such a multilayered net are described. These include the use of parallel optical computing of the global energy of the net, the use of fast nonvolatile programmable spatial light modulators to realize fast plasticity, optical generation of random number arrays, and an adaptive noisy thresholding scheme that also makes stochastic learning more biologically plausible. The findings reported predict optoelectronic chips that can be used in the realization of optical learning machines.

Synchronous vs asynchronous behavior of Hopfield's CAM neural net

The performance of Hopfield's neural net operating in synchronous and asynchronous modes is contrasted. Two interconnect matrices are considered: (1) the original Hopfield interconnect matrix; (2) the original Hopfield interconnect matrix with self-neural feedback. Specific attention is focused on techniques to maximize convergence rates and avoid steady-state oscillation. We identify two oscillation modes. Vertical oscillation occurs when the net's energy changes during each iteration. A neural net operated asynchronously cannot oscillate vertically. Synchronous operation, on the other hand, can change a net's energy either positively or negatively and vertical oscillation can occur. Horizontal oscillation occurs when the net alternates between two or more states of the same energy. Certain horizontal oscillations can be avoided by adopting appropriate thresholding rules. We demonstrate, for example, that when (1) the states of neurons with an input sum of zero are assigned the complement of their previous state, (2) the net is operated asynchronously, and (3) nonzero neural autoconnects are allowed, the net will not oscillate either vertically or horizontally.

Nonlinear electroabsorption cell for artificial neural networks

A nonlinear electroabsorption cell that has possible practical applications in the development of artificial neural networks is described. The properties of the cell are based on cross modulation of light in semiinsulating GaAs in the presence of an electric field. The electroabsorption cell has memory capability and similar nonlinear input-output characteristics to the neuron. The cell can be incorporated in an artificial neural network consisting of parallel interconnects and feedback.