Cooling schedules for learning in neural networks

Temperature control for simulated annealing

The optimal cooling schedule for simulated annealing is formulated to derive a differential equation for the time-dependent temperature T(t). Based on this equation, the long-term behavior of T(t), entropy production, and the Kullback-Leibler entropy are studied. For some simple examples, such as a many-level system and the small scale traveling salesman problem, the explicit time dependence of the temperature is obtained. Some comments are given on simulated annealing based on Tsallis statistics.

Generalized radial basis function networks for classification and novelty detection: self-organization of optimal Bayesian decision

By adding reverse connections from the output layer to the central layer it is shown how a generalized radial basis functions (GRBF) network can self-organize to form a Bayesian classifier, which is also capable of novelty detection. For this purpose, three stochastic sequential learning rules are introduced from biological considerations which pertain to the centers, the shapes, and the widths of the receptive fields of the neurons and allow ajoint optimization of all network parameters. The rules are shown to generate maximum-likelihood estimates of the class-conditional probability density functions of labeled data in terms of multivariate normal mixtures. Upon combination with a hierarchy of deterministic annealing procedures, which implement a multiple-scale approach, the learning process can avoid the convergence problems hampering conventional expectation-maximization algorithms. Using an example from the field of speech recognition, the stages of the learning process and the capabilities of the self-organizing GRBF classifier are illustrated.

On Langevin Updating in Multilayer Perceptrons

The Langevin updating rule, in which noise is added to the weights during learning, is presented and shown to improve learning on problems with initially ill-conditioned Hessians. This is particularly important for multilayer perceptrons with many hidden layers, that often have ill-conditioned Hessians. In addition, Manhattan updating is shown to have a similar effect.