Weight Noise Injection-Based MLPs With Group Lasso Penalty: Asymptotic Convergence and Application to Node Pruning

Structured Pruning for Deep Convolutional Neural Networks: A Survey

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at: https://github.com/he-y/Awesome-Pruning. A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey.

Regularization Effect of Random Node Fault/Noise on Gradient Descent Learning Algorithm

For decades, adding fault/noise during training by gradient descent has been a technique for getting a neural network (NN) tolerant to persistent fault/noise or getting an NN with better generalization. In recent years, this technique has been readvocated in deep learning to avoid overfitting. Yet, the objective function of such fault/noise injection learning has been misinterpreted as the desired measure (i.e., the expected mean squared error (mse) of the training samples) of the NN with the same fault/noise. The aims of this article are: 1) to clarify the above misconception and 2) investigate the actual regularization effect of adding node fault/noise when training by gradient descent. Based on the previous works on adding fault/noise during training, we speculate the reason why the misconception appears. In the sequel, it is shown that the learning objective of adding random node fault during gradient descent learning (GDL) for a multilayer perceptron (MLP) is identical to the desired measure of the MLP with the same fault. If additive (resp. multiplicative) node noise is added during GDL for an MLP, the learning objective is not identical to the desired measure of the MLP with such noise. For radial basis function (RBF) networks, it is shown that the learning objective is identical to the corresponding desired measure for all three fault/noise conditions. Empirical evidence is presented to support the theoretical results and, hence, clarify the misconception that the objective function of a fault/noise injection learning might not be interpreted as the desired measure of the NN with the same fault/noise. Afterward, the regularization effect of adding node fault/noise during training is revealed for the case of RBF networks. Notably, it is shown that the regularization effect of adding additive or multiplicative node noise (MNN) during training an RBF is reducing network complexity. Applying dropout regularization in RBF networks, its effect is the same as adding MNN during training.

Spiking Neural Network Regularization With Fixed and Adaptive Drop-Keep Probabilities

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures.

Learning Optimized Structure of Neural Networks by Hidden Node Pruning With L1 Regularization

We propose three different methods to determine the optimal number of hidden nodes based on L₁ regularization for a multilayer perceptron network. The first two methods, respectively, use a set of multiplier functions and multipliers for the hidden-layer nodes and implement the L₁ regularization on those, while the third method equipped with the same multipliers uses a smoothing approximation of the L₁ regularization. Each of these methods begins with a given number of hidden nodes, then the network is trained to obtain an optimal architecture discarding redundant hidden nodes using the multiplier functions or multipliers. A simple and generic method, namely, the matrix-based convergence proving method (MCPM), is introduced to prove the weak and strong convergence of the presented smoothing algorithms. The performance of the three pruning methods has been tested on 11 different classification datasets. The results demonstrate the efficient pruning abilities and competitive generalization by the proposed methods. The theoretical results are also validated by the results.

ℓ1-gain filter design of discrete-time positive neural networks with mixed delays

This paper mainly focuses on the filter design with ℓ₁-gain disturbance attenuation performance for a class of discrete-time positive neural networks. Discrete and distributed time-varying delays occurring in neuron transmission are taken into account. Especially, the probabilistic distribution of distributed delays is described by a Bernoulli random process in the system model. First, criteria on the positiveness and the unique equilibrium of discrete-time neural networks are presented. Second, through linear Lyapunov method, sufficient conditions for globally asymptotic stability with ℓ₁-gain disturbance attenuation performance of positive neural networks are proposed. Third, using the results obtained above, criteria on ℓ₁-gain stability of the established filtering error system are presented, based on which a linear programming (LP) approach is put forward to design the desired positive filter. Finally, two examples of applications to water distribution network and genetic regulatory network are given to demonstrate the effectiveness and applicability of the derived results.

Landslide Susceptibility Prediction Using Particle-Swarm-Optimized Multilayer Perceptron: Comparisons with Multilayer-Perceptron-Only, BP Neural Network, and Information Value Models

Landslides are one type of serious geological hazard which cause immense losses of local life and property. Landslide susceptibility prediction (LSP) can be used to determine the spatial probability of landslide occurrence in a certain area. It is important to implement LSP for landslide hazard prevention and reduction. This study developed a particle-swarm-optimized multilayer perceptron (PSO-MLP) model for LSP implementation to overcome the drawbacks of the conventional gradient descent algorithm and to determine the optimal structural parameters of MLP. Shicheng County in Jiangxi Province of China was used as the study area. In total, 369 landslides, randomly selected non-landslides, and 14 landslide-related predisposing factors were used to train and test the present PSO-MLP model and three other comparative models (an MLP-only model with the gradient descent algorithm, a back-propagation neural network (BPNN), and an information value (IV) model). The results showed that the PSO-MLP model had the most accurate prediction performance (area under the receiver operating characteristic curve (AUC) of 0.822 and frequency ratio (FR) accuracy of 0.856) compared with the MLP-only (0.791 and 0.829), BPNN (0.800 and 0.840), and IV (0.788 and 0.824) models. It can be concluded that the proposed PSO-MLP model addresses the drawbacks of the MLP-only model well and performs better than conventional artificial neural networks (ANNs) and statistical models. The spatial probability distribution law of landslide occurrence in Shicheng County was well revealed by the landslide susceptibility map produced using the PSO-MLP model. Furthermore, the present PSO-MLP model may have higher prediction and classification performances in some other fields compared with conventional ANNs and statistical models.