Constructive Feedforward Neural Networks Using Hermite Polynomial Activation Functions

GEPAF: A non-monotonic generalized activation function in neural network for improving prediction with diverse data distributions characteristics

The world today has made prescriptive analytics that uses data-driven insights to guide future actions. The distribution of data, however, differs depending on the scenario, making it difficult to interpret and comprehend the data efficiently. Different neural network models are used to solve this, taking inspiration from the complex network architecture in the human brain. The activation function is crucial in introducing non-linearity to process data gradients effectively. Although popular activation functions such as ReLU, Sigmoid, Swish, and Tanh have advantages and disadvantages, they may struggle to adapt to diverse data characteristics. A generalized activation function named the Generalized Exponential Parametric Activation Function (GEPAF) is proposed to address this issue. This function consists of three parameters expressed: α, which stands for a differencing factor similar to the mean; σ, which stands for a variance to control distribution spread; and p, which is a power factor that improves flexibility; all these parameters are present in the exponent. When p=2, the activation function resembles a Gaussian function. Initially, this paper describes the mathematical derivation and validation of the properties of this function mathematically and graphically. After this, the GEPAF function is practically implemented in real-world supply chain datasets. One dataset features a small sample size but exhibits high variance, while the other shows significant variance with a moderate amount of data. An LSTM network processes the dataset for sales and profit prediction. The suggested function performs better than popular activation functions when a comparative analysis of the activation function is performed, showing at least 30% improvement in regression evaluation metrics and better loss decay characteristics.

Dendrite Net: A White-Box Module for Classification, Regression, and System Identification

The simulation of biological dendrite computations is vital for the development of artificial intelligence (AI). This article presents a basic machine-learning (ML) algorithm, called Dendrite Net or DD, just like the support vector machine (SVM) or multilayer perceptron (MLP). DD's main concept is that the algorithm can recognize this class after learning, if the output's logical expression contains the corresponding class's logical relationship among inputs (and \ or \ not). Experiments and main results: DD, a white-box ML algorithm, showed excellent system identification performance for the black-box system. Second, it was verified by nine real-world applications that DD brought better generalization capability relative to the MLP architecture that imitated neurons' cell body (Cell body Net) for regression. Third, by MNIST and FASHION-MNIST datasets, it was verified that DD showed higher testing accuracy under greater training loss than the cell body net for classification. The number of modules can effectively adjust DD's logical expression capacity, which avoids overfitting and makes it easy to get a model with outstanding generalization capability. Finally, repeated experiments in MATLAB and PyTorch (Python) demonstrated that DD was faster than Cell body Net both in epoch and forwardpropagation. The main contribution of this article is the basic ML algorithm (DD) with a white-box attribute, controllable precision for better generalization capability, and lower computational complexity. Not only can DD be used for generalized engineering, but DD has vast development potential as a module for deep learning. DD code is available at https://github.com/liugang1234567/Gang-neuron.

An Adaptive Deep Belief Network With Sparse Restricted Boltzmann Machines

Deep belief network (DBN) is an efficient learning model for unknown data representation, especially nonlinear systems. However, it is extremely hard to design a satisfactory DBN with a robust structure because of traditional dense representation. In addition, backpropagation algorithm-based fine-tuning tends to yield poor performance since its ease of being trapped into local optima. In this article, we propose a novel DBN model based on adaptive sparse restricted Boltzmann machines (AS-RBM) and partial least square (PLS) regression fine-tuning, abbreviated as ARP-DBN, to obtain a more robust and accurate model than the existing ones. First, the adaptive learning step size is designed to accelerate an RBM training process, and two regularization terms are introduced into such a process to realize sparse representation. Second, initial weight derived from AS-RBM is further optimized via layer-by-layer PLS modeling starting from the output layer to input one. Third, we present the convergence and stability analysis of the proposed method. Finally, our approach is tested on Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification. Simulation results demonstrate that it has higher learning accuracy and faster learning speed. It can be used to build a more robust model than the existing ones.

IR-UWB Pulse Generation Using FPGA Scheme for through Obstacle Human Detection

This research proposes a scheme of field programmable gate array (FPGA) to generate an impulse-radio ultra-wideband (IR-UWB) pulse. The FPGA scheme consists of three parts: digital clock manager, four-delay-paths stratagem, and edge combiner. The IR-UWB radar system is designed to detect human subjects from their respiration underneath the rubble in the aftermath of an earthquake and to locate the human subjects based on range estimation. The proposed IR-UWB radar system is experimented with human subjects lying underneath layers of stacked clay bricks in supine and prone position. The results reveal that the IR-UWB radar system achieves a pulse duration of 540 ps with a bandwidth of 2.073 GHz (fractional bandwidth of 1.797). In addition, the IR-UWB technology can detect human subjects underneath the rubble from respiration and identify the location of human subjects by range estimation. The novelty of this research lies in the use of the FPGA scheme to achieve an IR-UWB pulse with a 2.073 GHz (117 MHz–2.19 GHz) bandwidth, thereby rendering the technology suitable for a wide range of applications, in addition to through-obstacle detection.

Universal Approximation by Using the Correntropy Objective Function

Several objective functions have been proposed in the literature to adjust the input parameters of a node in constructive networks. Furthermore, many researchers have focused on the universal approximation capability of the network based on the existing objective functions. In this brief, we use a correntropy measure based on the sigmoid kernel in the objective function to adjust the input parameters of a newly added node in a cascade network. The proposed network is shown to be capable of approximating any continuous nonlinear mapping with probability one in a compact input sample space. Thus, the convergence is guaranteed. The performance of our method was compared with that of eight different objective functions, as well as with an existing one hidden layer feedforward network on several real regression data sets with and without impulsive noise. The experimental results indicate the benefits of using a correntropy measure in reducing the root mean square error and increasing the robustness to noise.

On Training Efficiency and Computational Costs of a Feed Forward Neural Network: A Review

A comprehensive review on the problem of choosing a suitable activation function for the hidden layer of a feed forward neural network has been widely investigated. Since the nonlinear component of a neural network is the main contributor to the network mapping capabilities, the different choices that may lead to enhanced performances, in terms of training, generalization, or computational costs, are analyzed, both in general-purpose and in embedded computing environments. Finally, a strategy to convert a network configuration between different activation functions without altering the network mapping capabilities will be presented.

Generalized single-hidden layer feedforward networks for regression problems

In this paper, traditional single-hidden layer feedforward network (SLFN) is extended to novel generalized SLFN (GSLFN) by employing polynomial functions of inputs as output weights connecting randomly generated hidden units with corresponding output nodes. The significant contributions of this paper are as follows: 1) a primal GSLFN (P-GSLFN) is implemented using randomly generated hidden nodes and polynomial output weights whereby the regression matrix is augmented by full or partial input variables and only polynomial coefficients are to be estimated; 2) a simplified GSLFN (S-GSLFN) is realized by decomposing the polynomial output weights of the P-GSLFN into randomly generated polynomial nodes and tunable output weights; 3) both P- and S-GSLFN are able to achieve universal approximation if the output weights are tuned by ridge regression estimators; and 4) by virtue of the developed batch and online sequential ridge ELM (BR-ELM and OSR-ELM) learning algorithms, high performance of the proposed GSLFNs in terms of generalization and learning speed is guaranteed. Comprehensive simulation studies and comparisons with standard SLFNs are carried out on real-world regression benchmark data sets. Simulation results demonstrate that the innovative GSLFNs using BR-ELM and OSR-ELM are superior to standard SLFNs in terms of accuracy, training speed, and structure compactness.

Quantum-based algorithm for optimizing artificial neural networks

This paper presents a quantum-based algorithm for evolving artificial neural networks (ANNs). The aim is to design an ANN with few connections and high classification performance by simultaneously optimizing the network structure and the connection weights. Unlike most previous studies, the proposed algorithm uses quantum bit representation to codify the network. As a result, the connectivity bits do not indicate the actual links but the probability of the existence of the connections, thus alleviating mapping problems and reducing the risk of throwing away a potential candidate. In addition, in the proposed model, each weight space is decomposed into subspaces in terms of quantum bits. Thus, the algorithm performs a region by region exploration, and evolves gradually to find promising subspaces for further exploitation. This is helpful to provide a set of appropriate weights when evolving the network structure and to alleviate the noisy fitness evaluation problem. The proposed model is tested on four benchmark problems, namely breast cancer and iris, heart, and diabetes problems. The experimental results show that the proposed algorithm can produce compact ANN structures with good generalization ability compared to other algorithms.

AN INCREMENTAL FRAMEWORK BASED ON CROSS-VALIDATION FOR ESTIMATING THE ARCHITECTURE OF A MULTILAYER PERCEPTRON

We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using cross-validation. We consider five variants that implement forward/backward search, using single/multiple operators and searching depth-first/breadth-first. On 44 classification and 30 regression datasets, we exhaustively search for the optimal and evaluate the goodness based on: (1) Order, the accuracy with respect to the optimal and (2) Rank, the computational complexity. We check for the effect of two resampling methods (5 × 2, ten-fold cv), four statistical tests (5 × 2 cv t, ten-fold cv t, Wilcoxon, sign) and two corrections for multiple comparisons (Bonferroni, Holm). We also compare with Dynamic Node Creation (DNC) and Cascade Correlation (CC). Our results show that: (1) On most datasets, networks with few hidden units are optimal, (2) forward searching finds simpler architectures, (3) variants using single node additions (deletions) generally stop early and get stuck in simple (complex) networks, (4) choosing the best of multiple operators finds networks closer to the optimal, (5) MOST variants generally find simpler networks having lower or comparable error rates than DNC and CC.

Intelligent complementary sliding-mode control for LUSMS-based X-Y-theta motion control stage

An intelligent complementary sliding-mode control (ICSMC) system using a recurrent wavelet-based Elman neural network (RWENN) estimator is proposed in this study to control the mover position of a linear ultrasonic motors (LUSMs)-based X-Y-theta motion control stage for the tracking of various contours. By the addition of a complementary generalized error transformation, the complementary sliding-mode control (CSMC) can efficiently reduce the guaranteed ultimate bound of the tracking error by half compared with the slidingmode control (SMC) while using the saturation function. To estimate a lumped uncertainty on-line and replace the hitting control of the CSMC directly, the RWENN estimator is adopted in the proposed ICSMC system. In the RWENN, each hidden neuron employs a different wavelet function as an activation function to improve both the convergent precision and the convergent time compared with the conventional Elman neural network (ENN). The estimation laws of the RWENN are derived using the Lyapunov stability theorem to train the network parameters on-line. A robust compensator is also proposed to confront the uncertainties including approximation error, optimal parameter vectors, and higher-order terms in Taylor series. Finally, some experimental results of various contours tracking show that the tracking performance of the ICSMC system is significantly improved compared with the SMC and CSMC systems.

Bagging and boosting negatively correlated neural networks

In this paper, we propose two cooperative ensemble learning algorithms, i.e., NegBagg and NegBoost, for designing neural network (NN) ensembles. The proposed algorithms incrementally train different individual NNs in an ensemble using the negative correlation learning algorithm. Bagging and boosting algorithms are used in NegBagg and NegBoost, respectively, to create different training sets for different NNs in the ensemble. The idea behind using negative correlation learning in conjunction with the bagging/boosting algorithm is to facilitate interaction and cooperation among NNs during their training. Both NegBagg and NegBoost use a constructive approach to automatically determine the number of hidden neurons for NNs. NegBoost also uses the constructive approach to automatically determine the number of NNs for the ensemble. The two algorithms have been tested on a number of benchmark problems in machine learning and NNs, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, satellite, soybean, and waveform problems. The experimental results show that NegBagg and NegBoost require a small number of training epochs to produce compact NN ensembles with good generalization.

Feed-Forward Neural Networks Based on the Eigenstates of the Quantum Harmonic Oscillator

The paper introduces feed-forward neural networks where the hidden units employ orthogonal Hermite polynomials for their activation functions. The proposed neural networks have some interesting properties: (i) the basis functions are invariant under the Fourier transform, subject only to a change of scale, (ii) the basis functions are the eigenstates of the quantum harmonic oscillator, and stem from the solution of Schrödinger’s diffusion equation. The proposed feed-forward neural networks belong to the general category of nonparametric estimators and can be used for function approximation, system modelling and image processing.