Training neural networks to be insensitive to weight random variations

ADMM-Based Algorithm for Training Fault Tolerant RBF Networks and Selecting Centers

In the training stage of radial basis function (RBF) networks, we need to select some suitable RBF centers first. However, many existing center selection algorithms were designed for the fault-free situation. This brief develops a fault tolerant algorithm that trains an RBF network and selects the RBF centers simultaneously. We first select all the input vectors from the training set as the RBF centers. Afterward, we define the corresponding fault tolerant objective function. We then add an -norm term into the objective function. As the -norm term is able to force some unimportant weights to zero, center selection can be achieved at the training stage. Since the -norm term is nondifferentiable, we formulate the original problem as a constrained optimization problem. Based on the alternating direction method of multipliers framework, we then develop an algorithm to solve the constrained optimization problem. The convergence proof of the proposed algorithm is provided. Simulation results show that the proposed algorithm is superior to many existing center selection algorithms.

A Regularizer Approach for RBF Networks Under the Concurrent Weight Failure Situation

Many existing results on fault-tolerant algorithms focus on the single fault source situation, where a trained network is affected by one kind of weight failure. In fact, a trained network may be affected by multiple kinds of weight failure. This paper first studies how the open weight fault and the multiplicative weight noise degrade the performance of radial basis function (RBF) networks. Afterward, we define the objective function for training fault-tolerant RBF networks. Based on the objective function, we then develop two learning algorithms, one batch mode and one online mode. Besides, the convergent conditions of our online algorithm are investigated. Finally, we develop a formula to estimate the test set error of faulty networks trained from our approach. This formula helps us to optimize some tuning parameters, such as RBF width.

SpikeCell: a deterministic spiking neuron

We present a model of spiking neuron that emulates the output of the usual static neurons with sigmoidal activation functions. It allows for hardware implementations of standard feedforward networks, trained off-line with any classical learning algorithm (i.e. back-propagation and its variants). The model is validated on hand-written digits recognition, and image classification tasks. A digital architecture is proposed and evaluated. The area needed for implementing the spiking neuron on a chip is 10 times smaller than that for the corresponding static neuron. The accuracy of the network's output increases with time, and reaches that of the emulated static neural network after an adequate integration period. Single errors in the spike trains, or interruption of the relaxation process, due for example to irradiation in harsh environments, are harmless.