SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence

EPOC: A 28-nm 5.3 pJ/SOP Event-Driven Parallel Neuromorphic Hardware With Neuromodulation-Based Online Learning

Bio-inspired neuromorphic hardware with learning ability is highly promising to achieve human-like intelligence, particularly in terms of high energy efficiency and strong environmental adaptability. Though many customized prototypes have demonstrated learning ability, learning on neuromorphic hardware still lacks a bio-plausible and unified learning framework, and inherent spike-based sparsity and parallelism have not been fully exploited, which fundamentally limits their computational efficiency and scale. Therefore, we develop a unified, event-driven, and massively parallel multi-core neuromorphic online learning processor, namely EPOC. We present a neuromodulation-based neuromorphic online learning framework to unify various learning algorithms, and EPOC supports high-accuracy local/global supervised Spike Neural Network (SNN) learning with a low-memory-demand streaming single-sample learning strategy through different neuromodulator formulations. EPOC leverages a novel event-driven computation method that fully exploits spike-based sparsity throughout the forward-backward learning phases, and parallel multi-channel and multi-core computing architecture, bringing 9.9$\times$ time efficiency improvement compared with the baseline architecture. We synthesize EPOC in a 28-nm CMOS process and perform extensive benchmarking. EPOC achieves state-of-the-art learning accuracy of 99.2%, 98.2%, and 94.3% on the MNIST, NMNIST, and DVS-Gesture benchmarks, respectively. Local-learning EPOC achieves 2.9$\times$ time efficiency improvement compared with the global learning counterpart. EPOC operates at a typical clock frequency of 100 MHz, providing a peak 328 GOPS/51 GSOPS throughput and a 5.3 pJ/SOP energy efficiency.

Reconfigurable Artificial Synapses Based on Ambipolar Environmentally Stable Tellurium for Neuromorphic Computing

Neuromorphic computing, a promising solution to the von Neumann bottleneck, is paving the way for next-generation computing and sensing systems. However, most studies of artificial synapses mimic only static plasticity, which is far from achieving the complex behaviors of the human brain. Here, we report a reliable neuromorphic computing system that integrates a top floating gate memory architecture and uses peculiar ambipolar tellurium (Te) as a channel material to fabricate reliable nonvolatile memory cells. The memory device clearly exhibits exceptional retention (∼104 s) and endurance (∼104 cycles) properties for ambipolar memory with on/off ratios of 108 (electrons) and 106 (holes). Furthermore, we have also achieved reconfigurable excitatory and inhibitory synapse functions based on a Te ambipolarity device and explored its application in neuromorphic computing for recognition of different levels of complexity images with high accuracy generally above 90%, demonstrating its potential in neuromorphic computing. These findings highlight the prospects of ambipolar Te memory for advancing the future in memory computing hardware.

SMVSNN: An Intelligent Framework for Anticancer Drug-Drug Interaction Prediction Utilizing Spiking Multi-view Siamese Neural Networks

The study of synergistic drug combinations is vital in cancer treatment, enhancing efficacy, reducing resistance, and minimizing side effects through complementary drug actions. Drug-drug interaction (DDI) analysis offers essential theoretical support, and with the rise of data science, intelligent algorithms are increasingly replacing traditional in vitro screening for predicting potential DDIs. Considering the limitations of previous computational methods, such as the application of a single view, overly direct concatenation of drug pair features, and existing data encoding that is difficult to handle, this paper proposes a novel DDI analysis and prediction framework, called the Spiking Multi-View Siamese Neural Network-based (SMVSNN) framework. First, the data of two drugs in each view are processed into fused features using a Siamese spiking convolutional network and a spiking neural perceptron. Second, the processed features from multiple views are integrated into a unified representation through a self-learning attention weight module. Finally, this unified representation is fed into a spiking multilayer perceptron network to obtain the prediction results. Compared to traditional intelligent algorithms, the spiking neurons and the siamese network in SMVSNN can more effectively extract and integrate latent information from drug pair data. Real anticancer drug data, including 904 drugs, 7730 DDI records, and 19 drug interactions, were extracted from authoritative public databases to assess the effectiveness of our framework. The 5-fold cross-validation indicates that SMVSNN outperforms previous models on the majority of metrics. SMVSNN is poised to be an effective method for inferring potential synergistic drug combinations in anticancer therapy.

Reconfigurable neuromorphic functions in antiferroelectric transistors through coupled polarization switching and charge trapping dynamics

The growing demand for energy- and area-efficient emulation of biological nervous systems has fueled significant interest in neuromorphic computing. A promising strategy to achieve compact and efficient neuromorphic functionalities lies in the integration of volatile and non-volatile memory functions. However, implementing these functions is challenging due to the fundamentally distinct physical mechanisms. Traditional ferroelectric materials, with their stable polarization, are ideal for emulating biological synaptic functions but their non-volatile nature conflicts with the short-term memory necessary for neuron-like behavior. Here, we report the design for antiferroelectric gating in two-dimensional channel transistors, incorporating antiferroelectricity with charge trapping dynamics. By tuning the area ratio of the Metal-(Anti-)Ferroelectric-Metal-Insulator-Semiconductor (MFMIS) gate stacks, we enable selective reconfiguration of intrinsic volatile antiferroelectric switching and non-volatile switching-assisted charge trapping/de-trapping, thereby achieving both short- and long-term plasticity. This allows the integration of complementary functionalities of artificial neurons and synapses within a single device platform. Additionally, we further demonstrate synaptic and neuronal functions for implementing unsupervised learning rules and spiking behavior in spiking neural networks. This approach holds great potential for advancing both foundational materials design and technology for neuromorphic hardware applications.

Spiking-PhysFormer: Camera-based remote photoplethysmography with parallel spike-driven transformer

Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 10.1% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.

Topology optimization of random memristors for input-aware dynamic SNN

Machine learning has advanced unprecedentedly, exemplified by GPT-4 and SORA. However, they cannot parallel human brains in efficiency and adaptability due to differences in signal representation, optimization, runtime reconfigurability, and hardware architecture. To address these challenges, we introduce pruning optimization for input-aware dynamic memristive spiking neural network (PRIME). PRIME uses spiking neurons to emulate brain's spiking mechanisms and optimizes the topology of random memristive SNNs inspired by structural plasticity, effectively mitigating memristor programming stochasticity. It also uses the input-aware early-stop policy to reduce latency and leverages memristive in-memory computing to mitigate von Neumann bottleneck. Validated on a 40-nm, 256-K memristor-based macro, PRIME achieves comparable classification accuracy and inception score to software baselines, with energy efficiency improvements of 37.8× and 62.5×. In addition, it reduces computational loads by 77 and 12.5% with minimal performance degradation and demonstrates robustness to stochastic memristor noise. PRIME paves the way for brain-inspired neuromorphic computing.

Enhanced accuracy in first-spike coding using current-based adaptive LIF neuron

First spike timings are crucial for decision-making in spiking neural networks (SNNs). A recently introduced first-spike (FS) coding method demonstrates comparable accuracy to firing-rate (FR) coding in processing complex temporal information through supervised learning. However, its performance still falls behind advanced approaches. In order to explore the potential of FS coding, we enhance the capability of SNNs in classifying auditory datasets by improving neural dynamics. We propose a current-based adaptive LIF neuron (CuAdLIF) with delayed responses and membrane potential adaptation to enhance temporal correlations and preserve long short-memory. Furthermore, we introduce strategies to minimize delays in decision-making and enable adaptive training for FS coding. Results show that the CuAdLIF neuron enhances the extraction of temporal features and significantly improves FS coding accuracy. In addition, our strategies effectively reduce output time delays.

Efficient Deep Spiking Multilayer Perceptrons With Multiplication-Free Inference

Advancements in adapting deep convolution architectures for spiking neural networks (SNNs) have significantly enhanced image classification performance and reduced computational burdens. However, the inability of multiplication-free inference (MFI) to align with attention and transformer mechanisms, which are critical to superior performance on high-resolution vision tasks, imposes limitations on these gains. To address this, our research explores a new pathway, drawing inspiration from the progress made in multilayer perceptrons (MLPs). We propose an innovative spiking MLP architecture that uses batch normalization (BN) to retain MFI compatibility and introduce a spiking patch encoding (SPE) layer to enhance local feature extraction capabilities. As a result, we establish an efficient multistage spiking MLP network that blends effectively global receptive fields with local feature extraction for comprehensive spike-based computation. Without relying on pretraining or sophisticated SNN training techniques, our network secures a top-one accuracy of 66.39% on the ImageNet-1K dataset, surpassing the directly trained spiking ResNet-34 by 2.67%. Furthermore, we curtail computational costs, model parameters, and simulation steps. An expanded version of our network compares with the performance of the spiking VGG-16 network with a 71.64% top-one accuracy, all while operating with a model capacity 2.1 times smaller. Our findings highlight the potential of our deep SNN architecture in effectively integrating global and local learning abilities. Interestingly, the trained receptive field in our network mirrors the activity patterns of cortical cells.

Scaling Spike-Driven Transformer With Efficient Spike Firing Approximation Training

The ambition of brain-inspired Spiking Neural Networks (SNNs) is to become a low-power alternative to traditional Artificial Neural Networks (ANNs). This work addresses two major challenges in realizing this vision: the performance gap between SNNs and ANNs, and the high training costs of SNNs. We identify intrinsic flaws in spiking neurons caused by binary firing mechanisms and propose a Spike Firing Approximation (SFA) method using integer training and spike-driven inference. This optimizes the spike firing pattern of spiking neurons, enhancing efficient training, reducing power consumption, improving performance, enabling easier scaling, and better utilizing neuromorphic chips. We also develop an efficient spike-driven Transformer architecture and a spike-masked autoencoder to prevent performance degradation during SNN scaling. On ImageNet-1k, we achieve state-of-the-art top-1 accuracy of 78.5%, 79.8%, 84.0%, and 86.2% with models containing 10 M, 19 M, 83 M, and 173 M parameters, respectively. For instance, the 10 M model outperforms the best existing SNN by 7.2% on ImageNet, with training time acceleration and inference energy efficiency improved by 4.5× and 3.9×, respectively. We validate the effectiveness and efficiency of the proposed method across various tasks, including object detection, semantic segmentation, and neuromorphic vision tasks. This work enables SNNs to match ANN performance while maintaining the low-power advantage, marking a significant step towards SNNs as a general visual backbone.

Dynamic spatio-temporal pruning for efficient spiking neural networks

Spiking neural networks (SNNs), which draw from biological neuron models, have the potential to improve the computational efficiency of artificial neural networks (ANNs) due to their event-driven nature and sparse data flow. SNNs rely on dynamical sparsity, in that neurons are trained to activate sparsely to minimize data communication. This is critical when accounting for hardware given the bandwidth limitations between memory and processor. Given that neurons are sparsely activated, weights are less frequently accessed, and potentially can be pruned to less performance degradation in a SNN compared to an equivalent ANN counterpart. Reducing the number of synaptic connections between neurons also relaxes memory demands for neuromorphic processors. In this paper, we propose a spatio-temporal pruning algorithm that dynamically adapts to reduce the temporal redundancy that often exists in SNNs when processing Dynamic Vision Sensor (DVS) datasets. Spatial pruning is executed based on both global parameter statistics and inter-layer parameter count and is shown to reduce model degradation under extreme sparsity. We provide an ablation study that isolates the various components of spatio-temporal pruning, and find that our approach achieves excellent performance across all datasets, with especially high performance on datasets with time-varying features. We achieved a 0.69% improvement on the DVS128 Gesture dataset, despite the common expectation that pruning typically degrades performance. Notably, this enhancement comes with an impressive 98.18% reduction in parameter space and a 50% reduction in time redundancy.

Encrypted Spiking Neural Networks Based on Adaptive Differential Privacy Mechanism

Spike neural networks (SNNs) perform excellently in various domains. However, SNNs based on differential privacy (DP) protocols introduce uniform noise to the gradient parameters, which may affect the trade-off between model efficiency and personal privacy. Therefore, the adaptive differential private SNN (ADPSNN) is proposed in this work. It dynamically adjusts the privacy budget based on the correlations between the output spikes and labels. In addition, the noise is added to the gradient parameters according to the privacy budget. The ADPSNN is tested on four datasets with different spiking neurons including leaky integrated-and-firing (LIF) and integrate-and-fire (IF) models. Experimental results show that the LIF neuron model provides superior utility on the MNIST (accuracy 99.56%) and Fashion-MNIST (accuracy 92.26%) datasets, while the IF neuron model performs well on the CIFAR10 (accuracy 90.67%) and CIFAR100 (accuracy 66.10%) datasets. Compared to existing methods, the accuracy of ADPSNN is improved by 0.09% to 3.1%. The ADPSNN has many potential applications, such as image classification, health care, and intelligent driving.

Research on Anti-Interference Performance of Spiking Neural Network Under Network Connection Damage

BACKGROUND

With the development of artificial intelligence, memristors have become an ideal choice to optimize new neural network architectures and improve computing efficiency and energy efficiency due to their combination of storage and computing power. In this context, spiking neural networks show the ability to resist Gaussian noise, spike interference, and AC electric field interference by adjusting synaptic plasticity. The anti-interference ability to spike neural networks has become an important direction of electromagnetic protection bionics research.

METHODS

Therefore, this research constructs two types of spiking neural network models with LIF model as nodes: VGG-SNN and FCNN-SNN, and combines pruning algorithm to simulate network connection damage during the training process. By comparing and analyzing the millimeter wave radar human motion dataset and MNIST dataset with traditional artificial neural networks, the anti-interference performance of spiking neural networks and traditional artificial neural networks under the same probability of edge loss was deeply explored.

RESULTS

The experimental results show that on the millimeter wave radar human motion dataset, the accuracy of the spiking neural network decreased by 5.83% at a sparsity of 30%, while the accuracy of the artificial neural network decreased by 18.71%. On the MNIST dataset, the accuracy of the spiking neural network decreased by 3.91% at a sparsity of 30%, while the artificial neural network decreased by 10.13%.

CONCLUSIONS

Therefore, under the same network connection damage conditions, spiking neural networks exhibit unique anti-interference performance advantages. The performance of spiking neural networks in information processing and pattern recognition is relatively more stable and outstanding. Further analysis reveals that factors such as network structure, encoding method, and learning algorithm have a significant impact on the anti-interference performance of both.

Bio-plausible reconfigurable spiking neuron for neuromorphic computing

Biological neurons use diverse temporal expressions of spikes to achieve efficient communication and modulation of neural activities. Nonetheless, existing neuromorphic computing systems mainly use simplified neuron models with limited spiking behaviors due to high cost of emulating these biological spike patterns. Here, we propose a compact reconfigurable neuron design using the intrinsic dynamics of a NbO2-based spiking unit and excellent tunability in an electrochemical memory (ECRAM) to emulate the fast-slow dynamics in a bio-plausible neuron. The resistance of the ECRAM was effective in tuning the temporal dynamics of the membrane potential, contributing to flexible reconfiguration of various bio-plausible firing modes, such as phasic and burst spiking, and exhibiting adaptive spiking behaviors in changing environment. We used the bio-plausible neuron model to build spiking neural networks with bursting neurons and demonstrated improved classification accuracies over simplified models, showing great promises for use in more bio-plausible neuromorphic computing systems.

Spiking Neural Networks with Adaptive Membrane Time Constant for Event-Based Tracking

The brain-inspired Spiking Neural Networks (SNNs) work in an event-driven manner and have an implicit recurrence in neuronal membrane potential to memorize information over time, which are inherently suitable to handle temporal event-based streams. Despite their temporal nature and recent approaches advancements, these methods have predominantly been assessed on event-based classification tasks. In this paper, we explore the utility of SNNs for event-based tracking tasks. Specifically, we propose a brain-inspired adaptive Leaky Integrate-and-Fire neuron (BA-LIF) that can adaptively adjust the membrane time constant according to the inputs, thereby accelerating the leakage of meaningless noise features and reducing the decay of valuable information. SNNs composed of our proposed BA-LIF neurons can achieve high performance without a careful and time-consuming trial-by-error initialization on the membrane time constant. The adaptive capability of our network is further improved by introducing an extra temporal feature aggregator (TFA) that assigns attention weights over the temporal dimension. Extensive experiments on various event-based tracking datasets validate the effectiveness of our proposed method. We further validate the generalization capability of our method by applying it to other event-classification tasks.

Brain-inspired wiring economics for artificial neural networks

Wiring patterns of brain networks embody a trade-off between information transmission, geometric constraints, and metabolic cost, all of which must be balanced to meet functional needs. Geometry and wiring economy are crucial in the development of brains, but their impact on artificial neural networks (ANNs) remains little understood. Here, we adopt a wiring cost-controlled training framework that simultaneously optimizes wiring efficiency and task performance during structural evolution of sparse ANNs whose nodes are located at arbitrary but fixed positions. We show that wiring cost control improves performance across a wide range of tasks, ANN architectures and training methods, and can promote task-specific structural modules. An optimal wiring cost range provides both enhanced predictive performance and high values of topological properties, such as modularity and clustering, which are observed in real brain networks and known to improve robustness, interpretability, and performance of ANNs. In addition, ANNs trained using wiring cost can emulate the connection distance distribution observed in the brains of real organisms (such as Ciona intestinalis and Caenorhabditis elegans), especially when achieving high task performance, offering insights into biological organizing principles. Our results shed light on the relationship between topology and task specialization of ANNs trained within biophysical constraints, and their geometric resemblance to real neuronal-level brain maps.

Spiking Variational Policy Gradient for Brain Inspired Reinforcement Learning

Recent studies in reinforcement learning have explored brain-inspired function approximators and learning algorithms to simulate brain intelligence and adapt to neuromorphic hardware. Among these approaches, reward-modulated spike-timing-dependent plasticity (R-STDP) is biologically plausible and energy-efficient, but suffers from a gap between its local learning rules and the global learning objectives, which limits its performance and applicability. In this paper, we design a recurrent winner-take-all network and propose the spiking variational policy gradient (SVPG), a new R-STDP learning method derived theoretically from the global policy gradient. Specifically, the policy inference is derived from an energy-based policy function using mean-field inference, and the policy optimization is based on a last-step approximation of the global policy gradient. These fill the gap between the local learning rules and the global target. In experiments including a challenging ViZDoom vision-based navigation task and two realistic robot control tasks, SVPG successfully solves all the tasks. In addition, SVPG exhibits better inherent robustness to various kinds of input, network parameters, and environmental perturbations than compared methods.

Enhancing SNN-based spatio-temporal learning: A benchmark dataset and Cross-Modality Attention model

Spiking Neural Networks (SNNs), renowned for their low power consumption, brain-inspired architecture, and spatio-temporal representation capabilities, have garnered considerable attention in recent years. Similar to Artificial Neural Networks (ANNs), high-quality benchmark datasets are of great importance to the advances of SNNs. However, our analysis indicates that many prevalent neuromorphic datasets lack strong temporal correlation, preventing SNNs from fully exploiting their spatio-temporal representation capabilities. Meanwhile, the integration of event and frame modalities offers more comprehensive visual spatio-temporal information. Yet, the SNN-based cross-modality fusion remains underexplored. In this work, we present a neuromorphic dataset called DVS-SLR that can better exploit the inherent spatio-temporal properties of SNNs. Compared to existing datasets, it offers advantages in terms of higher temporal correlation, larger scale, and more varied scenarios. In addition, our neuromorphic dataset contains corresponding frame data, which can be used for developing SNN-based fusion methods. By virtue of the dual-modal feature of the dataset, we propose a Cross-Modality Attention (CMA) based fusion method. The CMA model efficiently utilizes the unique advantages of each modality, allowing for SNNs to learn both temporal and spatial attention scores from the spatio-temporal features of event and frame modalities, subsequently allocating these scores across modalities to enhance their synergy. Experimental results demonstrate that our method not only improves recognition accuracy but also ensures robustness across diverse scenarios.

SNN-BERT: Training-efficient Spiking Neural Networks for energy-efficient BERT

Spiking Neural Networks (SNNs) are naturally suited to process sequence tasks such as NLP with low power, due to its brain-inspired spatio-temporal dynamics and spike-driven nature. Current SNNs employ "repeat coding" that re-enter all input tokens at each timestep, which fails to fully exploit temporal relationships between the tokens and introduces memory overhead. In this work, we align the number of input tokens with the timestep and refer to this input coding as "individual coding". To cope with the increase in training time for individual encoded SNNs due to the dramatic increase in timesteps, we design a Bidirectional Parallel Spiking Neuron (BPSN) with following features: First, BPSN supports spike parallel computing and effectively avoids the issue of uninterrupted firing; Second, BPSN excels in handling adaptive sequence length tasks, which is a capability that existing work does not have; Third, the fusion of bidirectional information enhances the temporal information modeling capabilities of SNNs; To validate the effectiveness of our BPSN, we present the SNN-BERT, a deep direct training SNN architecture based on the BERT model in NLP. Compared to prior repeat 4-timestep coding baseline, our method achieves a 6.46× reduction in energy consumption and a significant 16.1% improvement, raising the performance upper bound of the SNN domain on the GLUE dataset to 74.4%. Additionally, our method achieves 3.5× training acceleration and 3.8× training memory optimization. Compared with artificial neural networks of similar architecture, we obtain comparable performance but up to 22.5× energy efficiency. We would provide the codes.

Spike-HAR++: an energy-efficient and lightweight parallel spiking transformer for event-based human action recognition

Event-based cameras are suitable for human action recognition (HAR) by providing movement perception with highly dynamic range, high temporal resolution, high power efficiency and low latency. Spike Neural Networks (SNNs) are naturally suited to deal with the asynchronous and sparse data from the event cameras due to their spike-based event-driven paradigm, with less power consumption compared to artificial neural networks. In this paper, we propose two end-to-end SNNs, namely Spike-HAR and Spike-HAR++, to introduce spiking transformer into event-based HAR. Spike-HAR includes two novel blocks: a spike attention branch, which enables model to focus on regions with high spike rates, reducing the impact of noise to improve the accuracy, and a parallel spike transformer block with simplified spiking self-attention mechanism, increasing computational efficiency. To better extract crucial information from high-level features, we modify the architecture of the spike attention branch and extend it in Spike-HAR to a higher dimension, proposing Spike-HAR++ to further enhance classification performance. Comprehensive experiments were conducted on four HAR datasets: SL-Animals-DVS, N-LSA64, DVS128 Gesture and DailyAction-DVS, to demonstrate the superior performance of our proposed model. Additionally, the proposed Spike-HAR and Spike-HAR++ require only 0.03 and 0.06 mJ, respectively, to process a sequence of event frames, with model sizes of only 0.7 and 1.8 M. This efficiency positions it as a promising new SNN baseline for the HAR community. Code is available at Spike-HAR++.

Graph spiking neural network for advanced urban flood risk assessment

Urban flooding significantly impacts city planning and resident safety. Traditional flood risk models, divided into physical and data-driven types, face challenges like data requirements and limited scalability. To overcome these, this study developed a model combining graph convolutional network (GCN) and spiking neural network (SNN), enabling the extraction of both spatial and temporal features from diverse data sources. We built a comprehensive flood risk dataset by integrating social media reports with weather and geographical data from six Chinese cities. The proposed Graph SNN model demonstrated superior performance compared to GCN and LSTM models, achieving high accuracy (85.3%), precision (0.811), recall (0.832), and F1 score (0.821). It also exhibited higher energy efficiency, making it scalable for real-time flood prediction in various urban environments. This research advances flood risk assessment by efficiently processing heterogeneous data while reducing energy consumption, offering a sustainable solution for urban flood management.

Self-architectural knowledge distillation for spiking neural networks

Spiking neural networks (SNNs) have attracted attention due to their biological plausibility and the potential for low-energy applications on neuromorphic hardware. Two mainstream approaches are commonly used to obtain SNNs, i.e., ANN-to-SNN conversion methods, and Directly-trained-SNN methods. However, the former achieve excellent performance at the cost of a large number of time steps (i.e., latency), while the latter exhibit lower latency but suffers from suboptimal performance. To tackle the performance-latency trade-off, we propose Self-Architectural Knowledge Distillation (SAKD), an intuitive and effective method for SNNs leveraging Knowledge Distillation (KD). We adopt a bilevel teacher-student training strategy in SAKD, i.e., level-1 involves directly transferring same-architectural pre-trained ANN weights to SNNs, and level-2 encourages the SNNs to mimic ANN's behavior, considering both final responses and intermediate features aspects. Learning with informative supervision signals fostered by labels and ANNs, our SAKD achieves new state-of-the-art (SOTA) performance with a few time steps on widely-used classification benchmark datasets. On ImageNet-1K, with only 4 time steps, our Spiking-ResNet34 model attains a Top-1 accuracy of 70.04%, outperforming the previous same-architectural SOTA methods. Notably, our SEW-ResNet152 model reaches a Top-1 accuracy of 77.30% on ImageNet-1K, setting a new SOTA benchmark for SNNs. Furthermore, we apply our SAKD to various dense prediction downstream tasks, such as object detection and semantic segmentation, demonstrating strong generalization ability and superior performance. In conclusion, our proposed SAKD framework presents a promising approach for achieving both high performance and low latency in SNNs, potentially paving the way for future advancements in the field.

SiamEFT: adaptive-time feature extraction hybrid network for RGBE multi-domain object tracking

Integrating RGB and Event (RGBE) multi-domain information obtained by high-dynamic-range and temporal-resolution event cameras has been considered an effective scheme for robust object tracking. However, existing RGBE tracking methods have overlooked the unique spatio-temporal features over different domains, leading to object tracking failure and inefficiency, especally for objects against complex backgrounds. To address this problem, we propose a novel tracker based on adaptive-time feature extraction hybrid networks, namely Siamese Event Frame Tracker (SiamEFT), which focuses on the effective representation and utilization of the diverse spatio-temporal features of RGBE. We first design an adaptive-time attention module to aggregate event data into frames based on adaptive-time weights to enhance information representation. Subsequently, the SiamEF module and cross-network fusion module combining artificial neural networks and spiking neural networks hybrid network are designed to effectively extract and fuse the spatio-temporal features of RGBE. Extensive experiments on two RGBE datasets (VisEvent and COESOT) show that the SiamEFT achieves a success rate of 0.456 and 0.574, outperforming the state-of-the-art competing methods and exhibiting a 2.3-fold enhancement in efficiency. These results validate the superior accuracy and efficiency of SiamEFT in diverse and challenging scenes.

Direct training high-performance deep spiking neural networks: a review of theories and methods

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs), in virtue of their high biological plausibility, rich spatial-temporal dynamics, and event-driven computation. The direct training algorithms based on the surrogate gradient method provide sufficient flexibility to design novel SNN architectures and explore the spatial-temporal dynamics of SNNs. According to previous studies, the performance of models is highly dependent on their sizes. Recently, direct training deep SNNs have achieved great progress on both neuromorphic datasets and large-scale static datasets. Notably, transformer-based SNNs show comparable performance with their ANN counterparts. In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance in a systematic and comprehensive way, including theory fundamentals, spiking neuron models, advanced SNN models and residual architectures, software frameworks and neuromorphic hardware, applications, and future trends.

Emergence and reconfiguration of modular structure for artificial neural networks during continual familiarity detection

Advances in artificial intelligence enable neural networks to learn a wide variety of tasks, yet our understanding of the learning dynamics of these networks remains limited. Here, we study the temporal dynamics during learning of Hebbian feedforward neural networks in tasks of continual familiarity detection. Drawing inspiration from network neuroscience, we examine the network's dynamic reconfiguration, focusing on how network modules evolve throughout learning. Through a comprehensive assessment involving metrics like network accuracy, modular flexibility, and distribution entropy across diverse learning modes, our approach reveals various previously unknown patterns of network reconfiguration. We find that the emergence of network modularity is a salient predictor of performance and that modularization strengthens with increasing flexibility throughout learning. These insights not only elucidate the nuanced interplay of network modularity, accuracy, and learning dynamics but also bridge our understanding of learning in artificial and biological agents.

Electrocardiography Classification with Leaky Integrate-and-Fire Neurons in an Artificial Neural Network-Inspired Spiking Neural Network Framework

Monitoring heart conditions through electrocardiography (ECG) has been the cornerstone of identifying cardiac irregularities. Cardiologists often rely on a detailed analysis of ECG recordings to pinpoint deviations that are indicative of heart anomalies. This traditional method, while effective, demands significant expertise and is susceptible to inaccuracies due to its manual nature. In the realm of computational analysis, Artificial Neural Networks (ANNs) have gained prominence across various domains, which can be attributed to their superior analytical capabilities. Conversely, Spiking Neural Networks (SNNs), which mimic the neural activity of the brain more closely through impulse-based processing, have not seen widespread adoption. The challenge lies primarily in the complexity of their training methodologies. Despite this, SNNs offer a promising avenue for energy-efficient computational models capable of displaying a high-level performance. This paper introduces an innovative approach employing SNNs augmented with an attention mechanism to enhance feature recognition in ECG signals. By leveraging the inherent efficiency of SNNs, coupled with the precision of attention modules, this model aims to refine the analysis of cardiac signals. The novel aspect of our methodology involves adapting the learned parameters from ANNs to SNNs using leaky integrate-and-fire (LIF) neurons. This transfer learning strategy not only capitalizes on the strengths of both neural network models but also addresses the training challenges associated with SNNs. The proposed method is evaluated through extensive experiments on two publicly available benchmark ECG datasets. The results show that our model achieves an overall accuracy of 93.8% on the MIT-BIH Arrhythmia dataset and 85.8% on the 2017 PhysioNet Challenge dataset. This advancement underscores the potential of SNNs in the field of medical diagnostics, offering a path towards more accurate, efficient, and less resource-intensive analyses of heart diseases.

Methodology based on spiking neural networks for univariate time-series forecasting

Spiking Neural Networks (SNN) are recognised as well-suited for processing spatiotemporal information with ultra-low energy consumption. However, proposals based on SNN for classification tasks are more common than for forecasting problems. In this sense, this paper presents a new general training methodology for univariate time-series forecasting based on SNN. The methodology is focused on one-step ahead forecasting problems and combines a PulseWidth Modulation based encoding-decoding algorithm with a Surrogate Gradient method as supervised training algorithm. In order to validate the generality of the presented methodology sine-wave, 3 UCI and 1 available real-world datasets are used. The results show very satisfactory forecasting results (MAE∈[0.0094,0.2891]) regardless of the characteristics of the dataset or the application field. In addition, weights can be initialised just once to achieve robust results, boosting the advantages of computational and energy cost of SNN.

Accelerating spiking neural network simulations with PymoNNto and PymoNNtorch

Spiking neural network simulations are a central tool in Computational Neuroscience, Artificial Intelligence, and Neuromorphic Engineering research. A broad range of simulators and software frameworks for such simulations exist with different target application areas. Among these, PymoNNto is a recent Python-based toolbox for spiking neural network simulations that emphasizes the embedding of custom code in a modular and flexible way. While PymoNNto already supports GPU implementations, its backend relies on NumPy operations. Here we introduce PymoNNtorch, which is natively implemented with PyTorch while retaining PymoNNto's modular design. Furthermore, we demonstrate how changes to the implementations of common network operations in combination with PymoNNtorch's native GPU support can offer speed-up over conventional simulators like NEST, ANNarchy, and Brian 2 in certain situations. Overall, we show how PymoNNto's modular and flexible design in combination with PymoNNtorch's GPU acceleration and optimized indexing operations facilitate research and development of spiking neural networks in the Python programming language.

Adapted MLP-Mixer network based on crossbar arrays of fast and multilevel switching (Co-Fe-B)x(LiNbO3)100-x nanocomposite memristors

MLP-Mixer based on multilayer perceptrons (MLPs) is a novel architecture of a neuromorphic computing system (NCS) introduced for image classification tasks without convolutional layers. Its software realization demonstrates high classification accuracy, although the number of trainable weights is relatively low. One more promising way of improving the NCS performance, especially in terms of power consumption, is its hardware realization using memristors. Therefore, in this work, we proposed an NCS with an adapted MLP-Mixer architecture and memristive weights. For this purpose, we used a passive crossbar array of (Co-Fe-B)x(LiNbO3)100-x memristors. Firstly, we studied the characteristics of such memristors, including their minimal resistive switching time, which was extrapolated to be in the picosecond range. Secondly, we created a fully hardware NCS with memristive weights that are capable of classification of simple 4-bit vectors. The system was shown to be robust to noise introduction in the input patterns. Finally, we used experimental memristive characteristics to simulate an adapted MLP-Mixer architecture that demonstrated a classification accuracy of (94.7 ± 0.3)% on the Modified National Institute of Standards and Technology (MNIST) dataset. The obtained results are the first steps toward the realization of memristive NCS with a promising MLP-Mixer architecture.

Learnable Leakage and Onset-Spiking Self-Attention in SNNs with Local Error Signals

Spiking neural networks (SNNs) have garnered significant attention due to their computational patterns resembling biological neural networks. However, when it comes to deep SNNs, how to focus on critical information effectively and achieve a balanced feature transformation both temporally and spatially becomes a critical challenge. To address these challenges, our research is centered around two aspects: structure and strategy. Structurally, we optimize the leaky integrate-and-fire (LIF) neuron to enable the leakage coefficient to be learnable, thus making it better suited for contemporary applications. Furthermore, the self-attention mechanism is introduced at the initial time step to ensure improved focus and processing. Strategically, we propose a new normalization method anchored on the learnable leakage coefficient (LLC) and introduce a local loss signal strategy to enhance the SNN's training efficiency and adaptability. The effectiveness and performance of our proposed methods are validated on the MNIST, FashionMNIST, and CIFAR-10 datasets. Experimental results show that our model presents a superior, high-accuracy performance in just eight time steps. In summary, our research provides fresh insights into the structure and strategy of SNNs, paving the way for their efficient and robust application in practical scenarios.