Asynchronous self-organizing maps

A recently defined energy function which leads to a self-organizing map is used as a foundation for an asynchronous neural-network algorithm. We generalize the existing stochastic gradient approach to an asynchronous parallel stochastic gradient method for generating a topological map on a distributed computer system (MIMD). A convergence proof is presented and simulation results on a set of problems are included. A practical problem using the energy function approach is that a summation over the entire network is required during the computation of updates. Using simulations we demonstrate effective algorithms that use efficient sampling for the approximation of these sums.

Kernel-based topographic map formation achieved with an information-theoretic approach

A new information-theoretic learning algorithm is introduced for kernel-based topographic map formation. The kernels are allowed to overlap and move freely in the input space, and to have differing kernel ranges. We start with Linsker's infomax principle and observe that it cannot be readily extended to our case, exactly due to the presence of kernels. We then consider Bell and Sejnowski's generalization of Linsker's infomax principle, which suggests differential entropy maximization, and add a second component to be optimized, namely, mutual information minimization between the kernel outputs, in order to take into account the kernel overlap, and thus the topographic map's output redundancy. The result is joint entropy maximization of the kernel outputs, which we adopt as our learning criterion. We derive a learning algorithm and verify its performance both for a synthetic example, for which the optimal result can be derived analytically, and for a classic real-world example.

Joint entropy maximization in kernel-based topographic maps

A new learning algorithm for kernel-based topographic map formation is introduced. The kernel parameters are adjusted individually so as to maximize the joint entropy of the kernel outputs. This is done by maximizing the differential entropies of the individual kernel outputs, given that the map's output redundancy, due to the kernel overlap, needs to be minimized. The latter is achieved by minimizing the mutual information between the kernel outputs. As a kernel, the (radial) incomplete gamma distribution is taken since, for a gaussian input density, the differential entropy of the kernel output will be maximal. Since the theoretically optimal joint entropy performance can be derived for the case of nonoverlapping gaussian mixture densities, a new clustering algorithm is suggested that uses this optimum as its "null" distribution. Finally, it is shown that the learning algorithm is similar to one that performs stochastic gradient descent on the Kullback-Leibler divergence for a heteroskedastic gaussian mixture density model.

Kernel-based topographic map formation by local density modeling

We introduce a new learning algorithm for kernel-based topographic map formation. The algorithm generates a gaussian mixture density model by individually adapting the gaussian kernels' centers and radii to the assumed gaussian local input densities.