Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors

Neuron Circuit Failure and Pattern Learning in Electronic Spiking Neural Networks

Biological neural networks demonstrate remarkable resilience and the ability to compensate for neuron losses over time. Thus, the effects of neural/synaptic losses in the brain go mostly unnoticed until the loss becomes profound. This study analyses the capacity of electronic spiking networks to compensate for the sudden, random neuron failure (“death”) due to reliability degradation or other external factors such as exposure to ionizing radiation. Electronic spiking neural networks with memristive synapses are designed to learn spatio-temporal patterns representing 25 or 100-pixel characters. The change in the pattern learning ability of the neural networks is observed as the afferents (input layer neurons) in the network fail/die during network training. Spike-timing-dependent plasticity (STDP) learning behavior is implemented using shaped action potentials with a realistic, non-linear memristor model. This work focuses on three cases: (1) when only neurons participating in the pattern are affected, (2) when non-participating neurons (those that never present spatio-temporal patterns) are disabled, and (3) when random/non-selective neuron death occurs in the network (the most realistic scenario). Case 3 is further analyzed to compare what happens when neuron death occurs over time versus when multiple afferents fail simultaneously. Simulation results emphasize the importance of non-participating neurons during the learning process, concluding that non-participating afferents contribute to improving the learning ability and stability of the neural network. Instantaneous neuron death proves to be more detrimental for the network compared to when afferents fail over time. To a surprising degree, the electronic spiking neural networks can sometimes retain their pattern recognition capability even in the case of significant neuron death.

Surrogate gradients for analog neuromorphic computing

Neuromorphic systems aim to accomplish efficient computation in electronics by mirroring neurobiological principles. Taking advantage of neuromorphic technologies requires effective learning algorithms capable of instantiating high-performing neural networks, while also dealing with inevitable manufacturing variations of individual components, such as memristors or analog neurons. We present a learning framework resulting in bioinspired spiking neural networks with high performance, low inference latency, and sparse spike-coding schemes, which also self-corrects for device mismatch. We validate our approach on the BrainScaleS-2 analog spiking neuromorphic system, demonstrating state-of-the-art accuracy, low latency, and energy efficiency. Our work sketches a path for building powerful neuromorphic processors that take advantage of emerging analog technologies.

To rapidly process temporal information at a low metabolic cost, biological neurons integrate inputs as an analog sum, but communicate with spikes, binary events in time. Analog neuromorphic hardware uses the same principles to emulate spiking neural networks with exceptional energy efficiency. However, instantiating high-performing spiking networks on such hardware remains a significant challenge due to device mismatch and the lack of efficient training algorithms. Surrogate gradient learning has emerged as a promising training strategy for spiking networks, but its applicability for analog neuromorphic systems has not been demonstrated. Here, we demonstrate surrogate gradient learning on the BrainScaleS-2 analog neuromorphic system using an in-the-loop approach. We show that learning self-corrects for device mismatch, resulting in competitive spiking network performance on both vision and speech benchmarks. Our networks display sparse spiking activity with, on average, less than one spike per hidden neuron and input, perform inference at rates of up to 85,000 frames per second, and consume less than 200 mW. In summary, our work sets several benchmarks for low-energy spiking network processing on analog neuromorphic hardware and paves the way for future on-chip learning algorithms.