Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm

In this paper, a hybrid Taguchi-genetic algorithm (HTGA) is applied to solve the problem of tuning both network structure and parameters of a feedforward neural network. The HTGA approach is a method of combining the traditional genetic algorithm (TGA), which has a powerful global exploration capability, with the Taguchi method, which can exploit the optimum offspring. The Taguchi method is inserted between crossover and mutation operations of a TGA. Then, the systematic reasoning ability of the Taguchi method is incorporated in the crossover operations to select the better genes to achieve crossover, and consequently enhance the genetic algorithms. Therefore, the HTGA approach can be more robust, statistically sound, and quickly convergent. First, the authors evaluate the performance of the presented HTGA approach by studying some global numerical optimization problems. Then, the presented HTGA approach is effectively applied to solve three examples on forecasting the sunspot numbers, tuning the associative memory, and solving the XOR problem. The numbers of hidden nodes and the links of the feedforward neural network are chosen by increasing them from small numbers until the learning performance is good enough. As a result, a partially connected feedforward neural network can be obtained after tuning. This implies that the cost of implementation of the neural network can be reduced. In these studied problems of tuning both network structure and parameters of a feedforward neural network, there are many parameters and numerous local optima so that these studied problems are challenging enough for evaluating the performances of any proposed GA-based approaches. The computational experiments show that the presented HTGA approach can obtain better results than the existing method reported recently in the literature.

Generalized RLS Approach to the Training of Neural Networks

Recursive least square (RLS) is an efficient approach to neural network training. However, in the classical RLS algorithm, there is no explicit decay in the energy function. This will lead to an unsatisfactory generalization ability for the trained networks. In this paper, we propose a generalized RLS (GRLS) model which includes a general decay term in the energy function for the training of feedforward neural networks. In particular, four different weight decay functions, namely, the quadratic weight decay, the constant weight decay and the newly proposed multimodal and quartic weight decay are discussed. By using the GRLS approach, not only the generalization ability of the trained networks is significantly improved but more unnecessary weights are pruned to obtain a compact network. Furthermore, the computational complexity of the GRLS remains the same as that of the standard RLS algorithm. The advantages and tradeoffs of using different decay functions are analyzed and then demonstrated with examples. Simulation results show that our approach is able to meet the design goals: improving the generalization ability of the trained network while getting a compact network.

Modelling the growth of Leuconostoc mesenteroides by Artificial Neural Networks

The combined effect of temperature (10.5 to 24.5 degrees C), pH level (5.5 to 7.5), sodium chloride level (0.25% to 6.25%) and sodium nitrite level (0 to 200 ppm) on the predicted specific growth rate (Gr), lag-time (Lag) and maximum population density (yEnd) of Leuconostoc mesenteroides under aerobic and anaerobic conditions, was studied using an Artificial Neural Network-based model (ANN) in comparison with Response Surface Methodology (RS). For both aerobic and anaerobic conditions, two types of ANN model were elaborated, unidimensional for each of the growth parameters, and multidimensional in which the three parameters Gr, Lag, and yEnd are combined. Although in general no significant statistical differences were observed between both types of model, we opted for the unidimensional model, because it obtained the lowest mean value for the standard error of prediction for generalisation. The ANN models developed provided reliable estimates for the three kinetic parameters studied; the SEP values in aerobic conditions ranged from between 2.82% for Gr, 6.05% for Lag and 10% for yEnd, a higher degree accuracy than those of the RS model (Gr: 9.54%; Lag: 8.89%; yEnd: 10.27%). Similar results were observed for anaerobic conditions. During external validation, a higher degree of accuracy (Af) and bias (Bf) were observed for the ANN model compared with the RS model. ANN predictive growth models are a valuable tool, enabling swift determination of L. mesenteroides growth parameters.

Biological engineering applications of feedforward neural networks designed and parameterized by genetic algorithms

Two neural network (NN) applications in the field of biological engineering are developed, designed and parameterized by an evolutionary method based on the evolutionary process of genetic algorithms. The developed systems are a fault detection NN model and a predictive modeling NN system. An indirect or 'weak specification' representation was used for the encoding of NN topologies and training parameters into genes of the genetic algorithm (GA). Some a priori knowledge of the demands in network topology for specific application cases is required by this approach, so that the infinite search space of the problem is limited to some reasonable degree. Both one-hidden-layer and two-hidden-layer network architectures were explored by the GA. Except for the network architecture, each gene of the GA also encoded the type of activation functions in both hidden and output nodes of the NN and the type of minimization algorithm that was used by the backpropagation algorithm for the training of the NN. Both models achieved satisfactory performance, while the GA system proved to be a powerful tool that can successfully replace the problematic trial-and-error approach that is usually used for these tasks.

Mutation-Based Genetic Neural Network

Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA) is a popular approach to address the local optima and design problems of ANN. The typical approach is to combine the strength of backpropagation (BP) in weight learning and EA's capability of searching the architecture space. However, the BP's "gradient descent" approach requires a highly computer-intensive operation that relatively restricts the search coverage of EA by compelling it to use a small population size. To address this problem, we utilized mutation-based genetic neural network (MGNN) to replace BP by using the mutation strategy of local adaptation of evolutionary programming (EP) to effect weight learning. The MGNN's mutation enables the network to dynamically evolve its structure and adapt its weights at the same time. Moreover, MGNN's EP-based encoding scheme allows for a flexible and less restricted formulation of the fitness function and makes fitness computation fast and efficient. This makes it feasible to use larger population sizes and allows MGNN to have a relatively wide search coverage of the architecture space. MGNN implements a stopping criterion where overfitness occurrences are monitored through "sliding-windows" to avoid premature learning and overlearning. Statistical analysis of its performance to some well-known classification problems demonstrate its good generalization capability. It also reveals that locally adapting or scheduling the strategy parameters embedded in each individual network may provide a proper balance between the local and global searching capabilities of MGNN.

Geometrical interpretation and architecture selection of MLP

A geometrical interpretation of the multilayer perceptron (MLP) is suggested in this paper. Some general guidelines for selecting the architecture of the MLP, i.e., the number of the hidden neurons and the hidden layers, are proposed based upon this interpretation and the controversial issue of whether four-layered MLP is superior to the three-layered MLP is also carefully examined.

EVOLUTIONARY COMPUTATION FOR ON-LINE AND OFF-LINE PARAMETER TUNING OF EVOLVING FUZZY NEURAL NETWORKS

This work applies Evolutionary Computation to achieve completely self-adapting Evolving Fuzzy Neural Networks (EFuNNs) for operating in both incremental (on-line) and batch (off-line) modes. EFuNNs belong to a class of Evolving Connectionist Systems (ECOS), capable of performing clustering-based, on-line, local area learning and rule extraction. Through Evolutionary Computation, its parameters such as learning rates and membership functions are continuously adjusted to reflect the changes in the dynamics of incoming data. The proposed methods are tested on the Mackey–Glass series and the results demonstrate a substantial improvement in EFuNN's performance.

Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication

We present a method for learning nonlinear systems, echo state networks (ESNs). ESNs employ artificial recurrent neural networks in a way that has recently been proposed independently as a learning mechanism in biological brains. The learning method is computationally efficient and easy to use. On a benchmark task of predicting a chaotic time series, accuracy is improved by a factor of 2400 over previous techniques. The potential for engineering applications is illustrated by equalizing a communication channel, where the signal error rate is improved by two orders of magnitude.

An improved automated ultrasonic NDE system by wavelet and neuron networks

Despite of the widespread and increasing use of digitized signals, the ultrasonic testing community has not realized yet the full potential of the electronic processing. The performance of an ultrasonic flaw detection method is evaluated by the success of distinguishing the flaw echoes from those scattered by microstructures. So, de-noising of ultrasonic signals is extremely important as to correctly identify smaller defects, because the probability of detection usually decreases as the defect size decreases, while the probability of false call does increase. In this paper, the wavelet transform has been successfully experimented to suppress noise and to enhance flaw location from ultrasonic signal, with a good defect localization. The obtained result is then directed to an automatic Artificial Neuronal Networks classification and learning algorithm of defects from A-scan data. Since there is some uncertainty connected with the testing technique, the system needs a numerical modelling. So, knowing the technical characteristics of the transducer, we can preview which are the defects that experimental inspection should find. Indeed, the system performs simulation of the ultrasonic wave propagation in the material, and gives a very helpful tool to get information and physical phenomena understanding, which can help to a suitable prediction of the service life of the component.

NEURAL NETWORKS AND GENETIC ALGORITHMS FOR DOMAIN INDEPENDENT MULTICLASS OBJECT DETECTION

This paper describes a domain independent approach to multiple class rotation invariant 2D object detection problems. The approach avoids preprocessing, segmentation and specific feature extraction. Instead, raw image pixel values are used as inputs to the learning systems. Five object detection methods have been developed and tested, the basic method and four variations which are expected to improve the accuracy of the basic method. In the basic method cutouts of the objects of interest are used to train multilayer feed forward networks using back propagation. The trained network is then used as a template to sweep the full image and find the objects of interest. The variations are (1) Use of a centred weight initialization method in network training, (2) Use of a genetic algorithm to train the network, (3) Use of a genetic algorithm, with fitness based on detection rate and false alarm rate, to refine the weights found in basic approach, and (4) Use of the same genetic algorithm to refine the weights found by method 2. These methods have been tested on three detection problems of increasing difficulty: an easy database of circles and squares, a medium difficulty database of coins and a very difficult database of retinal pathologies. For detecting the objects in all classes of interest in the easy and the medium difficulty problems, a 100% detection rate with no false alarms was achieved. However the results on the retinal pathologies were unsatisfactory. The centred weight initialization algorithm improved the detection performance over the basic approach on all three databases. In addition, refinement of weights with a genetic algorithm significantly improved detection performance on the three databases.

The goal of domain independent object recognition was achieved for the detection of relatively small regular objects in larger images with relatively uncluttered backgrounds. Detection performance on irregular objects in complex, highly cluttered backgrounds such as the retina pictures, however, has not been achieved to an acceptable level.

Genetically-designed Neural Networks for Error Reduction in an Optimized Biomechanical Model of the Human Elbow Joint Complex

A real time dynamic biomechanical model of the human elbow joint has been used as the first step in the process of calculating time varying joint position from the electromyograms (EMGs) of eight muscles crossing the joint. Since calculation of position has a high sensitivity to errors in the model torque calculation, a genetic algorithm (GA) neural network (NN) has been developed for automatic error reduction in the dynamic model. Genetic algorithms are used to design many neural network structures during a preliminary trial effort, and then each network's performance is ranked to choose a trained network that represents the most accurate result. Experimental results from three subjects have shown model error reduction in 84.2% of the data sets from a subject on which the model had been trained, and 52.6% of the data sets from the subjects on which the model had not been trained. Furthermore, the GA networks reduced the error standard deviation across all subjects, showing that progress in error reduction was made evenly across all data sets.

Incremental training of first order recurrent neural networks to predict a context-sensitive language

In recent years it has been shown that first order recurrent neural networks trained by gradient-descent can learn not only regular but also simple context-free and context-sensitive languages. However, the success rate was generally low and severe instability issues were encountered. The present study examines the hypothesis that a combination of evolutionary hill climbing with incremental learning and a well-balanced training set enables first order recurrent networks to reliably learn context-free and mildly context-sensitive languages. In particular, we trained the networks to predict symbols in string sequences of the context-sensitive language [a(n)b(n)c(n); n > or = 1. Comparative experiments with and without incremental learning indicated that incremental learning can accelerate and facilitate training. Furthermore, incrementally trained networks generally resulted in monotonic trajectories in hidden unit activation space, while the trajectories of non-incrementally trained networks were oscillating. The non-incrementally trained networks were more likely to generalise.

Neural network training with global optimization techniques

This paper presents an approach of using Simulated Annealing and Tabu Search for the simultaneous optimization of neural network architectures and weights. The problem considered is the odor recognition in an artificial nose. Both methods have produced networks with high classification performance and low complexity. Generalization has been improved by using the backpropagation algorithm for fine tuning. The combination of simple and traditional search methods has shown to be very suitable for generating compact and efficient networks.

Evolutionary computing for knowledge discovery in medical diagnosis

One of the major challenges in medical domain is the extraction of comprehensible knowledge from medical diagnosis data. In this paper, a two-phase hybrid evolutionary classification technique is proposed to extract classification rules that can be used in clinical practice for better understanding and prevention of unwanted medical events. In the first phase, a hybrid evolutionary algorithm (EA) is utilized to confine the search space by evolving a pool of good candidate rules, e.g. genetic programming (GP) is applied to evolve nominal attributes for free structured rules and genetic algorithm (GA) is used to optimize the numeric attributes for concise classification rules without the need of discretization. These candidate rules are then used in the second phase to optimize the order and number of rules in the evolution for forming accurate and comprehensible rule sets. The proposed evolutionary classifier (EvoC) is validated upon hepatitis and breast cancer datasets obtained from the UCI machine-learning repository. Simulation results show that the evolutionary classifier produces comprehensible rules and good classification accuracy for the medical datasets. Results obtained from t-tests further justify its robustness and invariance to random partition of datasets.

Incremental learning in biological and machine learning systems

Incremental learning concepts are reviewed in machine learning and neurobiology. They are identified in evolution, neurodevelopment and learning. A timeline of qualitative axon, neuron and synapse development summarizes the review on neurodevelopment. A discussion of experimental results on data incremental learning with recurrent artificial neural networks reveals that incremental learning often seems to be more efficient or powerful than standard learning but can produce unexpected side effects. A characterization of incremental learning is proposed which takes the elaborated biological and machine learning concepts into account.

Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks)

In this paper we present a cooperative coevolutive model for the evolution of neural network topology and weights, called MOBNET. MOBNET evolves subcomponents that must be combined in order to form a network, instead of whole networks. The problem of assigning credit to the subcomponents is approached as a multi-objective optimization task. The subcomponents in a cooperative coevolutive model must fulfill different criteria to be useful, these criteria usually conflict with each other. The problem of evaluating the fitness on an individual based on many criteria that must be optimized together can be approached as a multi-criteria optimization problems, so the methods from multi-objective optimization offer the most natural way to solve the problem. In this work we show how using several objectives for every subcomponent and evaluating its fitness as a multi-objective optimization problem, the performance of the model is highly competitive. MOBNET is compared with several standard methods of classification and with other neural network models in solving four real-world problems, and it shows the best overall performance of all classification methods applied. It also produces smaller networks when compared to other models. The basic idea underlying MOBNET is extensible to a more general model of coevolutionary computation, as none of its features are exclusive of neural networks design. There are many applications of cooperative coevolution that could benefit from the multi-objective optimization approach proposed in this paper.

An evolutionary artificial neural networks approach for breast cancer diagnosis

This paper presents an evolutionary artificial neural network (EANN) approach based on the pareto-differential evolution (PDE) algorithm augmented with local search for the prediction of breast cancer. The approach is named memetic pareto artificial neural network (MPANN). Artificial neural networks (ANNs) could be used to improve the work of medical practitioners in the diagnosis of breast cancer. Their abilities to approximate nonlinear functions and capture complex relationships in the data are instrumental abilities which could support the medical domain. We compare our results against an evolutionary programming approach and standard backpropagation (BP), and we show experimentally that MPANN has better generalization and much lower computational cost.

Hierarchical genetic algorithm for near optimal feedforward neural network design

In this paper, we propose a genetic algorithm based design procedure for a multi layer feed forward neural network. A hierarchical genetic algorithm is used to evolve both the neural networks topology and weighting parameters. Compared with traditional genetic algorithm based designs for neural networks, the hierarchical approach addresses several deficiencies, including a feasibility check highlighted in literature. A multi objective cost function is used herein to optimize the performance and topology of the evolved neural network simultaneously. In the prediction of Mackey Glass chaotic time series, the networks designed by the proposed approach prove to be competitive, or even superior, to traditional learning algorithms for the multi layer Perceptron networks and radial basis function networks. Based upon the chosen cost function, a linear weight combination decision making approach has been applied to derive an approximated Pareto optimal solution set. Therefore, designing a set of neural networks can be considered as solving a two objective optimization problem.

Improving artificial neural networks with a pruning methodology and genetic algorithms for their application in microbial growth prediction in food

The application of Artificial Neural Networks (ANN) in predictive microbiology is presented in this paper. This technique was used to build up a predictive model of the joint effect of NaCl concentration, pH level and storage temperature on kinetic parameters of the growth curve of Lactobacillus plantarum using ANN and Response Surface Model (RSM). Sigmoid functions were fitted to the data and kinetic parameters were estimated and used to build the models in which the independent variables were the factors mentioned above (NaCl, pH, temperature), and in some models, the values of the optical densities (OD) vs. time of the growth curve were also included in order to improve the error of estimation. The determination of the proper size of an ANN was the first step of the estimation. This study shows the usefulness of an ANN pruning methodology. The pruning of the network is a process consisting of removing unnecessary parameters (weights) and nodes during the training process of the network without losing its generalization capacity. The best architecture has been sought using genetic algorithms (GA) in conjunction with pruning algorithms and regularization methods in which the initial distribution of the parameters (weights) of the network is not uniform. The ANN model has been compared with the response surface model by means of the Standard Error of Prediction (SEP). The best values obtained were 14.04% of SEP for the growth rate and 14.84% for the lag estimation by the best ANN model, which were much better than those obtained by the RSM, 35.63% and 39.30%, respectively. These were very promising results that, in our opinion, open up an extremely important field of research.

A new algorithm to design compact two-hidden-layer artificial neural networks

This paper describes the cascade neural network design algorithm (CNNDA), a new algorithm for designing compact, two-hidden-layer artificial neural networks (ANNs). This algorithm determines an ANN's architecture with connection weights automatically. The design strategy used in the CNNDA was intended to optimize both the generalization ability and the training time of ANNs. In order to improve the generalization ability, the CNDDA uses a combination of constructive and pruning algorithms and bounded fan-ins of the hidden nodes. A new training approach, by which the input weights of a hidden node are temporarily frozen when its output does not change much after a few successive training cycles, was used in the CNNDA for reducing the computational cost and the training time. The CNNDA was tested on several benchmarks including the cancer, diabetes and character-recognition problems in ANNs. The experimental results show that the CNNDA can produce compact ANNs with good generalization ability and short training time in comparison with other algorithms.