Spiking Neural Network Regularization With Fixed and Adaptive Drop-Keep Probabilities

Reconstruction of Adaptive Leaky Integrate-and-Fire Neuron to Enhance the Spiking Neural Networks Performance by Establishing Complex Dynamics

Since digital spiking signals can carry rich information and propagate with low computational consumption, spiking neural networks (SNNs) have received great attention from neuroscientists and are regarded as the future development object of neural networks. However, generating the appropriate spiking signals remains challenging, which is related to the dynamics property of neurons. Most existing studies imitate the biological neurons based on the correlation of synaptic input and output, but these models have only one time constant, thus ignoring the structural differentiation and versatility in biological neurons. In this article, we propose the reconstruction of adaptive leaky integrate-and-fire (R-ALIF) neuron to perform complex behaviors similar to real neurons. First, a synaptic cleft time constant is introduced into the membrane voltage charging equation to distinguish the leakage degree between the neuron membrane and the synaptic cleft, which can expand the representation space of spiking neurons to facilitate SNNs to obtain better information expression way. Second, R-ALIF constructs a voltage threshold adjustment equation to balance the firing rate of output signals. Third, three time constants are transformed into learnable parameters, enabling the adaptive adjustment of dynamics equation and enhancing the information expression ability of SNNs. Fourth, the computational graph of R-ALIF is optimized to improve the performance of SNNs. Moreover, we adopt a temporal dropout (TemDrop) method to solve the overfitting problem in SNNs and propose a data augmentation method for neuromorphic datasets. Finally, we evaluate our method on CIFAR10-DVS, ASL-DVS, and CIFAR-100, and achieve top1 accuracy of 81.0%, 99.8%, and 67.83%, respectively, with few time steps. We believe that our method will further promote the development of SNNs trained by spatiotemporal backpropagation (STBP).

A Fusion Network With Stacked Denoise Autoencoder and Meta Learning for Lateral Walking Gait Phase Recognition and Multi-Step-Ahead Prediction

Lateral walking gait phase recognition and prediction are the premise of hip exoskeleton application in lateral resistance walk exercise. We presented a fusion network with stacked denoise autoencoder and meta learning (SDA-NN-ML) to recognize gait phase and predict gait percentage from IMU signals. Experiments were conducted to detect the four lateral walking gait phases and predict their percentage across different speeds. The performance of SDA-NN-ML and Support Vector Machine (SVM), Adaptive Boosting (AdaBoost) and Long Short Term Memory (LSTM) were evaluated. The cross-subject recognition accuracy of SDA-NN-ML (89.94%) decreased by 4.62% compared to the training accuracy, which outperformed SVM (8.60%), AdaBoost (5.61%), and LSTM (7.12%). For real-time and cross-subject prediction of gait phase percentage, the RMSE of SDA-NN-ML (0.2043) outperformed that of a single regression network (0.2426). With a signal noise ratio of 100:30, the cross-subject recognition accuracy decreased by a mere 5.70%, while the prediction result (RMSE) of SDA-NN-ML increased by 0.0167 when compared to the noise-free results. SDA-NN-ML demonstrates a stable multi-step-ahead prediction ability with an accuracy higher than 82.50% and an RMSE of less than 0.23 when the ahead time is less than 200 ms. The results demonstrated that the proposed method has high accuracy and robust performance in lateral walking gait recognition and prediction.

Capacitated Clustering via Majorization-Minimization and Collaborative Neurodynamic Optimization

This paper addresses capacitated clustering based on majorization-minimization and collaborative neurodynamic optimization (CNO). Capacitated clustering is formulated as a combinatorial optimization problem. Its objective function consists of fractional terms with intra-cluster similarities in their numerators and cluster cardinalities in their denominators as normalized cluster compactness measures. To obviate the difficulty in optimizing the objective function with factional terms, the combinatorial optimization problem is reformulated as an iteratively reweighted quadratic unconstrained binary optimization problem with a surrogate function and a penalty function in a majorization-minimization framework. A clustering algorithm is developed based on CNO for solving the reformulated problem. It employs multiple Boltzmann machines operating concurrently for local searches and a particle swarm optimization rule for repositioning neuronal states upon their local convergence. Experimental results on ten benchmark datasets are elaborated to demonstrate the superior clustering performance of the proposed approaches against seven baseline algorithms in terms of 21 internal cluster validity criteria.

Supervised Feature Selection via Collaborative Neurodynamic Optimization

As a crucial part of machine learning and pattern recognition, feature selection aims at selecting a subset of the most informative features from the set of all available features. In this article, supervised feature selection is at first formulated as a mixed-integer optimization problem with an objective function of weighted feature redundancy and relevancy subject to a cardinality constraint on the number of selected features. It is equivalently reformulated as a bound-constrained mixed-integer optimization problem by augmenting the objective function with a penalty function for realizing the cardinality constraint. With additional bilinear and linear equality constraints for realizing the integrality constraints, it is further reformulated as a bound-constrained biconvex optimization problem with two more penalty terms. Two collaborative neurodynamic optimization (CNO) approaches are proposed for solving the formulated and reformulated feature selection problems. One of the proposed CNO approaches uses a population of discrete-time recurrent neural networks (RNNs), and the other use a pair of continuous-time projection networks operating concurrently on two timescales. Experimental results on 13 benchmark datasets are elaborated to substantiate the superiority of the CNO approaches to several mainstream methods in terms of average classification accuracy with three commonly used classifiers.

STNet: A novel spiking neural network combining its own time signal with the spatial signal of an artificial neural network

Introduction

This article proposes a novel hybrid network that combines the temporal signal of a spiking neural network (SNN) with the spatial signal of an artificial neural network (ANN), namely the Spatio-Temporal Combined Network (STNet).

Methods

Inspired by the way the visual cortex in the human brain processes visual information, two versions of STNet are designed: a concatenated one (C-STNet) and a parallel one (P-STNet). In the C-STNet, the ANN, simulating the primary visual cortex, extracts the simple spatial information of objects first, and then the obtained spatial information is encoded as spiking time signals for transmission to the rear SNN which simulates the extrastriate visual cortex to process and classify the spikes. With the view that information from the primary visual cortex reaches the extrastriate visual cortex via ventral and dorsal streams, in P-STNet, the parallel combination of the ANN and the SNN is employed to extract the original spatio-temporal information from samples, and the extracted information is transferred to a posterior SNN for classification.

Results

The experimental results of the two STNets obtained on six small and two large benchmark datasets were compared with eight commonly used approaches, demonstrating that the two STNets can achieve improved performance in terms of accuracy, generalization, stability, and convergence.

Discussion

These prove that the idea of combining ANN and SNN is feasible and can greatly improve the performance of SNN.