Streaming parallel GPU acceleration of large-scale filter-based spiking neural networks

Robustness to Training Disturbances in SpikeProp Learning

Stability is a key issue during spiking neural network training using SpikeProp. The inherent nonlinearity of Spiking Neuron means that the learning manifold changes abruptly; therefore, we need to carefully choose the learning steps at every instance. Other sources of instability are the external disturbances that come along with training sample as well as the internal disturbances that arise due to modeling imperfection. The unstable learning scenario can be indirectly observed in the form of surges, which are sudden increases in the learning cost and are a common occurrence during SpikeProp training. Research in the past has shown that proper learning step size is crucial to minimize surges during training process. To determine proper learning step in order to avoid steep learning manifolds, we perform weight convergence analysis of SpikeProp learning in the presence of disturbance signals. The weight convergence analysis is further extended to robust stability analysis linked with overall system error. This ensures boundedness of the total learning error with minimal assumption of bounded disturbance signals. These analyses result in the learning rate normalization scheme, which are the key results of this paper. The performance of learning using this scheme has been compared with the prevailing methods for different benchmark data sets and the results show that this method has stable learning reflected by minimal surges during learning, higher success in training instances, and faster learning as well.

Dynamic parallelism for synaptic updating in GPU-accelerated spiking neural network simulations

Graphical processing units (GPUs) can significantly accelerate spiking neural network (SNN) simulations by exploiting parallelism for independent computations. Both the changes in membrane potential at each time-step, and checking for spiking threshold crossings for each neuron, can be calculated independently. However, because synaptic transmission requires communication between many different neurons, efficient parallel processing may be hindered, either by data transfers between GPU and CPU at each time-step or, alternatively, by running many parallel computations for neurons that do not elicit any spikes. This, in turn, would lower the effective throughput of the simulations. Traditionally, a central processing unit (CPU, host) administers the execution of parallel processes on the GPU (device), such as memory initialization on the device, data transfer between host and device, and starting and synchronizing parallel processes. The parallel computing platform CUDA 5.0 introduced dynamic parallelism, which allows the initiation of new parallel applications within an ongoing parallel kernel. Here, we apply dynamic parallelism for synaptic updating in SNN simulations on a GPU. Our algorithm eliminates the need to start many parallel applications at each time-step, and the associated lags of data transfer between CPU and GPU memories. We report a significant speed-up of SNN simulations, when compared to former accelerated parallelization strategies for SNNs on a GPU.

Robust learning in SpikeProp

Training a Spiking Neural Network using SpikeProp and its derivatives faces stability issues. Surges, marked by a sudden rise in learning cost, are a common occurrence during the learning process. They disrupt the learning process and often destabilize the process resulting in failure. A proper learning rate, which is neither too small nor too big, is important to minimize surges. Furthermore, external disturbances due to imperfection in sample data as well as internal disturbances are additional destabilizing source during the learning process. In this paper, we perform error system analysis incorporating external disturbance, followed by weight convergence analysis along with detailed robust stability analysis of SpikeProp learning process to ensure error bound of the learning process. Based on these results, we propose a robust adaptive learning rate scheme that aligns with the results of theoretical analysis. The performance of the proposed method has been compared with other prevalent methods based on different benchmark datasets and the results demonstrate that our method indeed has better performance in terms of convergence and learning speed as well.

Adaptive learning rate of SpikeProp based on weight convergence analysis

A Spiking Neural Network (SNN) training using SpikeProp and its variants is usually affected by sudden rise in learning cost called surges. These surges cause diversion in the learning process and often cause it to fail as well. Researches have shown that proper learning rate is crucial to avoid these surges. In this paper, we perform weight convergence analysis to determine the proper step size in each iteration of weight update and derive an adaptive learning rate extension to SpikeProp that assures convergence of the learning process. We have analyzed the performance of this learning rate adaptation with existing methods via simulations on different benchmarks. The results show that using adaptive learning rate significantly improves the weight convergence and speeds up learning as well.