Geometrical interpretation and architecture selection of MLP

Gradient boosting DD-MLP Net: An ensemble learning model using near-infrared spectroscopy to classify after-stroke dyskinesia degree during exercise

This study aims to develop an automatic assessment of after-stroke dyskinesias degree by combining machine learning and near-infrared spectroscopy (NIRS). Thirty-five subjects were divided into five stages (healthy, patient: Brunnstrom stages 3, 4, 5, 6). NIRS was used to record the muscular hemodynamic responses from bilateral femoris (biceps brachii) muscles during passive and active upper (lower) limbs circular exercise. We used the D-S evidence theory to conduct feature information fusion and established a Gradient Boosting DD-MLP Net model, combining the dendrite network and multilayer perceptron, to realize automatic dyskinesias degree evaluation. Our model classified the upper limb dyskinesias with high accuracy: 98.91% under the passive mode and 98.69% under the active mode, and classified the lower limb dyskinesias with high accuracy: 99.45% and 99.63% under the passive and active modes, respectively. Our model combined with NIRS has great potential in monitoring the after-stroke dyskinesias degree and guiding rehabilitation training.

The Estimation Life Cycle of Lithium-Ion Battery Based on Deep Learning Network and Genetic Algorithm

This study uses deep learning to model the discharge characteristic curve of the lithium-ion battery. The battery measurement instrument was used to charge and discharge the battery to establish the discharge characteristic curve. The parameter method tries to find the discharge characteristic curve and was improved by MLP (multilayer perceptron), RNN (recurrent neural network), LSTM (long short-term memory), and GRU (gated recurrent unit). The results obtained by these methods were graphs. We used genetic algorithm (GA) to obtain the parameters of the discharge characteristic curve equation.

Predicting Outflow Hydrographs of Potential Dike Breaches in a Bifurcating River System Using NARX Neural Networks

Early flood forecasting systems can mitigate flood damage during extreme events. Typically, the effects of flood events in terms of inundation depths and extents are computed using detailed hydraulic models. However, a major drawback of these models is the computational time, which is generally in the order of hours to days for large river basins. Gaining insight in the outflow hydrographs in case of dike breaches is especially important to estimate inundation extents. In this study, NARX neural networks that were capable of predicting outflow hydrographs of multiple dike breaches accurately were developed. The timing of the dike failures and the cumulative outflow volumes were accurately predicted. These findings show that neural networks—specifically, NARX networks that are capable of predicting flood time series—have the potential to be used within a flood early warning system in the future.

Energy Management through Cost Forecasting for Residential Buildings in New Zealand

Over the last two decades, the residential building sector has been one of the largest energy consumption sectors in New Zealand. The relationship between that sector and household energy consumption should be carefully studied in order to optimize the energy consumption structure and satisfy energy demands. Researchers have made efforts in this field; however, few have concentrated on the association between household energy use and the cost of residential buildings. This study examined the correlation between household energy use and residential building cost. Analysis of the data indicates that they are significantly correlated. Hence, this study proposes time series methods, including the exponential smoothing method and the autoregressive integrated moving average (ARIMA) model for forecasting residential building costs of five categories of residential buildings (one-storey house, two-storey house, townhouse, residential apartment and retirement village building) in New Zealand. Moreover, the artificial neutral networks (ANNs) model was used to forecast the future usage of three types of household energy (electricity, gas and petrol) using the residential building costs. The t-test was used to validate the effectiveness of the obtained ANN models. The results indicate that the ANN models can generate acceptable forecasts. The primary contributions of this paper are twofold: (1) Identify the close correlation between household energy use and residential building costs; (2) provide a new clue for optimizing energy management.

Developing a Library of Shear Walls Database and the Neural Network Based Predictive Meta-Model

There is a large amount of useful information from past experimental tests, which are usually ignored in test-setup for the new ones. Variation of assumptions, materials, test procedures, and test objectives make it difficult to choose the right model for validation of the numerical models. Results from different experiments are sometimes in conflict with each other, or have minimum correlation. Furthermore, not all these information are easily accessible for researchers and engineers. Therefore, this paper presents the results of a comprehensive study on different experimental models for steel plate and reinforced concrete shear walls. A unique library of up to 13 parameters (mechanical properties and geometric characteristics) affecting the strength, stiffness and drift ratio of the shear walls are gathered including their sensitivity analysis. Next, a predictive meta-model is developed based on artificial neural network. It is capable of forecasting the responses for any desired shear wall with good accuracy. The proposed network can be used to as an alternative to the nonlinear numerical simulations or expensive experimental test.

Classifying Parkinson's Disease Based on Acoustic Measures Using Artificial Neural Networks

In recent years, neural networks have become very popular in all kinds of prediction problems. In this paper, multiple feed-forward artificial neural networks (ANNs) with various configurations are used in the prediction of Parkinson’s disease (PD) of tested individuals, based on extracted features from 26 different voice samples per individual. Results are validated via the leave-one-subject-out (LOSO) scheme. Few feature selection procedures based on Pearson’s correlation coefficient, Kendall’s correlation coefficient, principal component analysis, and self-organizing maps, have been used for boosting the performance of algorithms and for data reduction. The best test accuracy result has been achieved with Kendall’s correlation coefficient-based feature selection, and the most relevant voice samples are recognized. Multiple ANNs have proven to be the best classification technique for diagnosis of PD without usage of the feature selection procedure (on raw data). Finally, a neural network is fine-tuned, and a test accuracy of 86.47% was achieved.

AN INCREMENTAL FRAMEWORK BASED ON CROSS-VALIDATION FOR ESTIMATING THE ARCHITECTURE OF A MULTILAYER PERCEPTRON

We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using cross-validation. We consider five variants that implement forward/backward search, using single/multiple operators and searching depth-first/breadth-first. On 44 classification and 30 regression datasets, we exhaustively search for the optimal and evaluate the goodness based on: (1) Order, the accuracy with respect to the optimal and (2) Rank, the computational complexity. We check for the effect of two resampling methods (5 × 2, ten-fold cv), four statistical tests (5 × 2 cv t, ten-fold cv t, Wilcoxon, sign) and two corrections for multiple comparisons (Bonferroni, Holm). We also compare with Dynamic Node Creation (DNC) and Cascade Correlation (CC). Our results show that: (1) On most datasets, networks with few hidden units are optimal, (2) forward searching finds simpler architectures, (3) variants using single node additions (deletions) generally stop early and get stuck in simple (complex) networks, (4) choosing the best of multiple operators finds networks closer to the optimal, (5) MOST variants generally find simpler networks having lower or comparable error rates than DNC and CC.

A new formulation for feedforward neural networks

Feedforward neural network is one of the most commonly used function approximation techniques and has been applied to a wide variety of problems arising from various disciplines. However, neural networks are black-box models having multiple challenges/difficulties associated with training and generalization. This paper initially looks into the internal behavior of neural networks and develops a detailed interpretation of the neural network functional geometry. Based on this geometrical interpretation, a new set of variables describing neural networks is proposed as a more effective and geometrically interpretable alternative to the traditional set of network weights and biases. Then, this paper develops a new formulation for neural networks with respect to the newly defined variables; this reformulated neural network (ReNN) is equivalent to the common feedforward neural network but has a less complex error response surface. To demonstrate the learning ability of ReNN, in this paper, two training methods involving a derivative-based (a variation of backpropagation) and a derivative-free optimization algorithms are employed. Moreover, a new measure of regularization on the basis of the developed geometrical interpretation is proposed to evaluate and improve the generalization ability of neural networks. The value of the proposed geometrical interpretation, the ReNN approach, and the new regularization measure are demonstrated across multiple test problems. Results show that ReNN can be trained more effectively and efficiently compared to the common neural networks and the proposed regularization measure is an effective indicator of how a network would perform in terms of generalization.

An experimental study on nonlinear function computation for neural/fuzzy hardware design

An experimental study on the influence of the computation of basic nodal nonlinear functions on the performance of (NFSs) is described in this paper. Systems' architecture size, their approximation capability, and the smoothness of provided mappings are used as performance indexes for this comparative paper. Two widely used kernel functions, the sigmoid-logistic function and the Gaussian function, are analyzed by their computation through an accuracy-controllable approximation algorithm designed for hardware implementation. Two artificial neural network (ANN) paradigms are selected for the analysis: backpropagation neural networks (BPNNs) with one hidden layer and radial basis function (RBF) networks. Extensive simulation of simple benchmark approximation problems is used in order to achieve generalizable conclusions. For the performance analysis of fuzzy systems, a functional equivalence theorem is used to extend obtained results to fuzzy inference systems (FISs). Finally, the adaptive neurofuzzy inference system (ANFIS) paradigm is used to observe the behavior of neurofuzzy systems with learning capabilities.

One-class-at-a-time removal sequence planning method for multiclass classification problems

Using dynamic programming, this work develops a one-class-at-a-time removal sequence planning method to decompose a multiclass classification problem into a series of two-class problems. Compared with previous decomposition methods, the approach has the following distinct features. First, under the one-class-at-a-time framework, the approach guarantees the optimality of the decomposition. Second, for a K-class problem, the number of binary classifiers required by the method is only K-1. Third, to achieve higher classification accuracy, the approach can easily be adapted to form a committee machine. A drawback of the approach is that its computational burden increases rapidly with the number of classes. To resolve this difficulty, a partial decomposition technique is introduced that reduces the computational cost by generating a suboptimal solution. Experimental results demonstrate that the proposed approach consistently outperforms two conventional decomposition methods.

A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks

In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang et al. developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance.

Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition

In this letter, we attempt to quantify the significance of increasing the number of neurons in the hidden layer of a feedforward neural network architecture using the singular value decomposition (SVD). Through this, we extend some well-known properties of the SVD in evaluating the generalizability of single hidden layer feedforward networks (SLFNs) with respect to the number of hidden layer neurons. The generalization capability of the SLFN is measured by the degree of linear independency of the patterns in hidden layer space, which can be indirectly quantified from the singular values obtained from the SVD, in a postlearning step. A pruning/growing technique based on these singular values is then used to estimate the necessary number of neurons in the hidden layer. More importantly, we describe in detail properties of the SVD in determining the structure of a neural network particularly with respect to the robustness of the selected model.

Real-time learning capability of neural networks

In some practical applications of neural networks, fast response to external events within an extremely short time is highly demanded and expected. However, the extensively used gradient-descent-based learning algorithms obviously cannot satisfy the real-time learning needs in many applications, especially for large-scale applications and/or when higher generalization performance is required. Based on Huang's constructive network model, this paper proposes a simple learning algorithm capable of real-time learning which can automatically select appropriate values of neural quantizers and analytically determine the parameters (weights and bias) of the network at one time only. The performance of the proposed algorithm has been systematically investigated on a large batch of benchmark real-world regression and classification problems. The experimental results demonstrate that our algorithm can not only produce good generalization performance but also have real-time learning and prediction capability. Thus, it may provide an alternative approach for the practical applications of neural networks where real-time learning and prediction implementation is required.

Generalized Haar DWT and transformations between decision trees and neural networks

The core contribution of this paper is a three-fold improvement of the Haar discrete wavelet transform (DWT). It is modified to efficiently transform a multiclass- (rather than numerical-) valued function over a multidimensional (rather than low dimensional) domain, or transform a multiclass-valued decision tree into another useful representation. We prove that this multidimensional, multiclass DWT uses dynamic programming to minimize (within its framework) the number of nontrivial wavelet coefficients needed to summarize a training set or decision tree. It is a spatially localized algorithm that takes linear time in the number of training samples, after a sort. Convergence of the DWT to benchmark training sets seems to degrade with rising dimension in this test of high dimensional wavelets, which have been seen as difficult to implement. This multiclass multidimensional DWT has tightly coupled applications from learning "dyadic" decision trees directly from training data, rebalancing or converting preexisting decision trees to fixed depth boolean or threshold neural networks (in effect parallelizing the evaluation of the trees), or learning rule/exception sets represented as a new form of tree called an "E-tree", which could greatly help interpretation/visualization of a dataset.