Variable surrogate model-based particle swarm optimization for high-dimensional expensive problems

Many industrial applications require time-consuming and resource-intensive evaluations of suitable solutions within very limited time frames. Therefore, many surrogate-assisted evaluation algorithms (SAEAs) have been widely used to optimize expensive problems. However, due to the curse of dimensionality and its implications, scaling SAEAs to high-dimensional expensive problems is still challenging. This paper proposes a variable surrogate model-based particle swarm optimization (called VSMPSO) to meet this challenge and extends it to solve 200-dimensional problems. Specifically, a single surrogate model constructed by simple random sampling is taken to explore different promising areas in different iterations. Moreover, a variable model management strategy is used to better utilize the current global model and accelerate the convergence rate of the optimizer. In addition, the strategy can be applied to any SAEA irrespective of the surrogate model used. To control the trade-off between optimization results and optimization time consumption of SAEAs, we consider fitness value and running time as a bi-objective problem. Applying the proposed approach to a benchmark test suite of dimensions ranging from 30 to 200 and comparisons with four state-of-the-art algorithms show that the proposed VSMPSO achieves high-quality solutions and computational efficiency for high-dimensional problems.

A bi-stage surrogate-assisted hybrid algorithm for expensive optimization problems

Surrogate-assisted meta-heuristic algorithms have shown good performance to solve the computationally expensive problems within a limited computational resource. Compared to the method that only one surrogate model is utilized, the surrogate ensembles have shown more efficiency to get a good optimal solution. In this paper, we propose a bi-stage surrogate-assisted hybrid algorithm to solve the expensive optimization problems. The framework of the proposed method is composed of two stages. In the first stage, a number of global searches will be conducted in sequence to explore different sub-spaces of the decision space, and the solution with the maximum uncertainty in the final generation of each global search will be evaluated using the exact expensive problems to improve the accuracy of the approximation on corresponding sub-space. In the second stage, the local search is added to exploit the sub-space, where the best position found so far locates, to find a better solution for real expensive evaluation. Furthermore, the local and global searches in the second stage take turns to be conducted to balance the trade-off of the exploration and exploitation. Two different meta-heuristic algorithms are, respectively, utilized for the global and local search. To evaluate the performance of our proposed method, we conduct the experiments on seven benchmark problems, the Lennard–Jones potential problem and a constrained test problem, respectively, and compare with five state-of-the-art methods proposed for solving expensive problems. The experimental results show that our proposed method can obtain better results, especially on high-dimensional problems.

Absolute Positioning Accuracy Improvement in an Industrial Robot

The absolute positioning accuracy of a robot is an important specification that determines its performance, but it is affected by several error sources. Typical calibration methods only consider kinematic errors and neglect complex non-kinematic errors, thus limiting the absolute positioning accuracy. To further improve the absolute positioning accuracy, we propose an artificial neural network optimized by the differential evolution algorithm. Specifically, the structure and parameters of the network are iteratively updated by differential evolution to improve both accuracy and efficiency. Then, the absolute positioning deviation caused by kinematic and non-kinematic errors is compensated using the trained network. To verify the performance of the proposed network, the simulations and experiments are conducted using a six-degree-of-freedom robot and a laser tracker. The robot average positioning accuracy improved from 0.8497 mm before calibration to 0.0490 mm. The results demonstrate the substantial improvement in the absolute positioning accuracy achieved by the proposed network on an industrial robot.

A Critical Review of the Modeling and Optimization of Combined Heat and Power Dispatch

Combined heat and power (CHP) systems are attracting increasing attention for their ability to improve the economics and sustainability of the electricity system. Determining how to best operate these systems is difficult because they can consist of many generating units whose operation is governed by complex nonlinear physics. Mathematical programming is a useful tool to support the operation of CHP systems, and has been the subject of substantial research attention since the early 1990s. This paper critically reviews the modeling and optimization work that has been done on the CHP economic dispatch problem, and the CHP economic and emission dispatch problem. A summary of the common models used for these problems is provided, along with comments on future modeling work that would beneficial to the field. The majority of optimization approaches studied for CHP system operation are metaheuristic algorithms. A discussion of the limitations and benefits of metaheuristic algorithms is given. Finally, a case study optimizing five classic CHP system test instances demonstrates the advantages of the using deterministic global search algorithms over metaheuristic search algorithms.

Evolutionary Computation for Parameter Extraction of Organic Thin-Film Transistors Using Newly Synthesized Liquid Crystalline Nickel Phthalocyanine

In this work, the topic of the detrimental contact effects in organic thin-film transistors (OTFTs) is revisited. In this case, contact effects are considered as a tool to enhance the characterization procedures of OTFTs, achieving more accurate values for the fundamental parameters of the transistor threshold voltage, carrier mobility and on-off current ratio. The contact region is also seen as a fundamental part of the device which is sensitive to physical, chemical and fabrication variables. A compact model for OTFTs, which includes the effects of the contacts, and a recent proposal of an associated evolutionary parameter extraction procedure are reviewed. Both the model and the procedure are used to assess the effect of the annealing temperature on a nickel-1,4,8,11,15,18,22,25-octakis(hexyl)phthalocyanine (NiPc6)-based OTFT. A review of the importance of phthalocyanines in organic electronics is also provided. The characterization of the contact region in NiPc6 OTFTs complements the results extracted from other physical-chemical techniques such as differential scanning calorimetry or atomic force microscopy, in which the transition from crystal to columnar mesophase imposes a limit for the optimum performance of the annealed OTFTs.

Self-Organizing RBF Neural Network Using an Adaptive Gradient Multiobjective Particle Swarm Optimization

One of the major obstacles in using radial basis function (RBF) neural networks is the convergence toward local minima instead of the global minima. For this reason, an adaptive gradient multiobjective particle swarm optimization (AGMOPSO) algorithm is designed to optimize both the structure and parameters of RBF neural networks in this paper. First, the AGMOPSO algorithm, based on a multiobjective gradient method and a self-adaptive flight parameters mechanism, is developed to improve the computation performance. Second, the AGMOPSO-based self-organizing RBF neural network (AGMOPSO-SORBF) can optimize the parameters (centers, widths, and weights), as well as determine the network size. The goal of AGMOPSO-SORBF is to find a tradeoff between the accuracy and the complexity of RBF neural networks. Third, the convergence analysis of AGMOPSO-SORBF is detailed to ensure the prerequisite of any successful applications. Finally, the merits of our proposed approach are verified on multiple numerical examples. The results indicate that the proposed AGMOPSO-SORBF achieves much better generalization capability and compact network structure than some other existing methods.

Training Radial Basis Function Neural Networks for Classification via Class-Specific Clustering

In training radial basis function neural networks (RBFNNs), the locations of Gaussian neurons are commonly determined by clustering. Training inputs can be clustered on a fully unsupervised manner (input clustering), or some supervision can be introduced, for example, by concatenating the input vectors with weighted output vectors (input-output clustering). In this paper, we propose to apply clustering separately for each class (class-specific clustering). The idea has been used in some previous works, but without evaluating the benefits of the approach. We compare the class-specific, input, and input-output clustering approaches in terms of classification performance and computational efficiency when training RBFNNs. To accomplish this objective, we apply three different clustering algorithms and conduct experiments on 25 benchmark data sets. We show that the class-specific approach significantly reduces the overall complexity of the clustering, and our experimental results demonstrate that it can also lead to a significant gain in the classification performance, especially for the networks with a relatively few Gaussian neurons. Among other applied clustering algorithms, we combine, for the first time, a dynamic evolutionary optimization method, multidimensional particle swarm optimization, and the class-specific clustering to optimize the number of cluster centroids and their locations.

A growing and pruning sequential learning algorithm of hyper basis function neural network for function approximation

Radial basis function (RBF) neural network is constructed of certain number of RBF neurons, and these networks are among the most used neural networks for modeling of various nonlinear problems in engineering. Conventional RBF neuron is usually based on Gaussian type of activation function with single width for each activation function. This feature restricts neuron performance for modeling the complex nonlinear problems. To accommodate limitation of a single scale, this paper presents neural network with similar but yet different activation function-hyper basis function (HBF). The HBF allows different scaling of input dimensions to provide better generalization property when dealing with complex nonlinear problems in engineering practice. The HBF is based on generalization of Gaussian type of neuron that applies Mahalanobis-like distance as a distance metrics between input training sample and prototype vector. Compared to the RBF, the HBF neuron has more parameters to optimize, but HBF neural network needs less number of HBF neurons to memorize relationship between input and output sets in order to achieve good generalization property. However, recent research results of HBF neural network performance have shown that optimal way of constructing this type of neural network is needed; this paper addresses this issue and modifies sequential learning algorithm for HBF neural network that exploits the concept of neuron's significance and allows growing and pruning of HBF neuron during learning process. Extensive experimental study shows that HBF neural network, trained with developed learning algorithm, achieves lower prediction error and more compact neural network.

Radial basis function network training using a nonsymmetric partition of the input space and particle swarm optimization

This paper presents a novel algorithm for training radial basis function (RBF) networks, in order to produce models with increased accuracy and parsimony. The proposed methodology is based on a nonsymmetric variant of the fuzzy means (FM) algorithm, which has the ability to determine the number and locations of the hidden-node RBF centers, whereas the synaptic weights are calculated using linear regression. Taking advantage of the short computational times required by the FM algorithm, we wrap a particle swarm optimization (PSO) based engine around it, designed to optimize the fuzzy partition. The result is an integrated framework for fully determining all the parameters of an RBF network. The proposed approach is evaluated through its application on 12 real-world and synthetic benchmark datasets and is also compared with other neural network training techniques. The results show that the RBF network models produced by the PSO-based nonsymmetric FM algorithm outperform the models produced by the other techniques, exhibiting higher prediction accuracies in shorter computational times, accompanied by simpler network structures.

An efficient method for ectopic beats cancellation based on radial basis function

The analysis of the surface Electrocardiogram (ECG) is the most extended noninvasive technique in cardiological diagnosis. In order to properly use the ECG, we need to cancel out ectopic beats. These beats may occur in both normal subjects and patients with heart disease, and their presence represents an important source of error which must be handled before any other analysis. This paper presents a method for electrocardiogram ectopic beat cancellation based on Radial Basis Function Neural Network (RBFNN). A train-able neural network ensemble approach to develop customized electrocardiogram beat classifier in an effort to further improve the performance of ECG processing and to offer individualized health care is presented. Six types of beats including: Normal Beats (NB); Premature Ventricular Contractions (PVC); Left Bundle Branch Blocks (LBBB); Right Bundle Branch Blocks (RBBB); Paced Beats (PB) and Ectopic Beats (EB) are obtained from the MIT-BIH arrhythmia database. Four morphological features are extracted from each beat after the preprocessing of the selected records. Average Results for the RBFNN based method provided an ectopic beat reduction (EBR) of (mean ± std) EBR = 7, 23 ± 2.18 in contrast to traditional compared methods that, for the best case, yielded EBR = 4.05 ± 2.13. The results prove that RBFNN based methods are able to obtain a very accurate reduction of ectopic beats together with low distortion of the QRST complex.

A REPAIR ALGORITHM FOR RADIAL BASIS FUNCTION NEURAL NETWORK AND ITS APPLICATION TO CHEMICAL OXYGEN DEMAND MODELING

This paper presents a repair algorithm for the design of a Radial Basis Function (RBF) neural network. The proposed repair RBF (RRBF) algorithm starts from a single prototype randomly initialized in the feature space. The algorithm has two main phases: an architecture learning phase and a parameter adjustment phase. The architecture learning phase uses a repair strategy based on a sensitivity analysis (SA) of the network's output to judge when and where hidden nodes should be added to the network. New nodes are added to repair the architecture when the prototype does not meet the requirements. The parameter adjustment phase uses an adjustment strategy where the capabilities of the network are improved by modifying all the weights. The algorithm is applied to two application areas: approximating a non-linear function, and modeling the key parameter, chemical oxygen demand (COD) used in the waste water treatment process. The results of simulation show that the algorithm provides an efficient solution to both problems.

Concurrent subspace width optimization method for RBF neural network modeling

Radial basis function neural networks (RBFNNs) are widely used in nonlinear function approximation. One of the challenges in RBFNN modeling is determining how to effectively optimize width parameters to improve approximation accuracy. To solve this problem, a width optimization method, concurrent subspace width optimization (CSWO), is proposed based on a decomposition and coordination strategy. This method decomposes the large-scale width optimization problem into several subspace optimization (SSO) problems, each of which has a single optimization variable and smaller training and validation data sets so as to greatly simplify optimization complexity. These SSOs can be solved concurrently, thus computational time can be effectively reduced. With top-level system coordination, the optimization of SSOs can converge to a consistent optimum, which is equivalent to the optimum of the original width optimization problem. The proposed method is tested with four mathematical examples and one practical engineering approximation problem. The results demonstrate the efficiency and robustness of CSWO in optimizing width parameters over the traditional width optimization methods.

Generating balanced learning and test sets for function approximation problems

In function approximation problems, one of the most common ways to evaluate a learning algorithm consists in partitioning the original data set (input/output data) into two sets: learning, used for building models, and test, applied for genuine out-of-sample evaluation. When the partition into learning and test sets does not take into account the variability and geometry of the original data, it might lead to non-balanced and unrepresentative learning and test sets and, thus, to wrong conclusions in the accuracy of the learning algorithm. How the partitioning is made is therefore a key issue and becomes more important when the data set is small due to the need of reducing the pessimistic effects caused by the removal of instances from the original data set. Thus, in this work, we propose a deterministic data mining approach for a distribution of a data set (input/output data) into two representative and balanced sets of roughly equal size taking the variability of the data set into consideration with the purpose of allowing both a fair evaluation of learning's accuracy and to make reproducible machine learning experiments usually based on random distributions. The sets are generated using a combination of a clustering procedure, especially suited for function approximation problems, and a distribution algorithm which distributes the data set into two sets within each cluster based on a nearest-neighbor approach. In the experiments section, the performance of the proposed methodology is reported in a variety of situations through an ANOVA-based statistical study of the results.

An efficient self-organizing RBF neural network for water quality prediction

This paper presents a flexible structure Radial Basis Function (RBF) neural network (FS-RBFNN) and its application to water quality prediction. The FS-RBFNN can vary its structure dynamically in order to maintain the prediction accuracy. The hidden neurons in the RBF neural network can be added or removed online based on the neuron activity and mutual information (MI), to achieve the appropriate network complexity and maintain overall computational efficiency. The convergence of the algorithm is analyzed in both the dynamic process phase and the phase following the modification of the structure. The proposed FS-RBFNN has been tested and compared to other algorithms by applying it to the problem of identifying a nonlinear dynamic system. Experimental results show that the FS-RBFNN can be used to design an RBF structure which has fewer hidden neurons; the training time is also much faster. The algorithm is applied for predicting water quality in the wastewater treatment process. The results demonstrate its effectiveness.

A neural network of smooth hinge functions

Smooth hinging hyperplane (SHH) has been proposed as an improvement over the well-known hinging hyperplane (HH) by the fact that it retains the useful features of HH while overcoming HH's drawback of nondifferentiability. This paper introduces a formal characterization of smooth hinge function (SHF), which can be used to generate SHH as a neural network. A method for the general construction of SHF is also given. Furthermore, the work proves that SHH is better than HH in functional approximation, i.e., the optimal error of SHH approximating a general function is always smaller or equal to that of HH. Particularly, in the case that the SHF is generated via the integration of a class of sigmoidal functions, it is further proven that the corresponding SHH of the 2m SHFs would outperform a neural network with m of the sigmoidal function from which the SHF is derived. Any upper bound established on the approximation error of a neural network of m sigmoidal activation functions can hence be translated to the SHH of m SHFs by replacing m with [m/2]. The work also includes an algorithm for the identification of SHH making use of its differentiability property. Simulation experiments are presented to validate the theoretical conclusions to possible extent.

Construction of tunable radial basis function networks using orthogonal forward selection

An orthogonal forward selection (OFS) algorithm based on leave-one-out (LOO) criteria is proposed for the construction of radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines an RBF node, namely, its center vector and diagonal covariance matrix, by minimizing the LOO statistics. For regression application, the LOO criterion is chosen to be the LOO mean-square error, while the LOO misclassification rate is adopted in two-class classification application. This OFS-LOO algorithm is computationally efficient, and it is capable of constructing parsimonious RBF networks that generalize well. Moreover, the proposed algorithm is fully automatic, and the user does not need to specify a termination criterion for the construction process. The effectiveness of the proposed RBF network construction procedure is demonstrated using examples taken from both regression and classification applications.

Multiobjective hybrid optimization and training of recurrent neural networks

The application of neural networks to solve a problem involves tasks with a high computational cost until a suitable network is found, and these tasks mainly involve the selection of the network topology and the training step. We usually select the network structure by means of a trial-and-error procedure, and we then train the network. In the case of recurrent neural networks (RNNs), the lack of suitable training algorithms sometimes hampers these procedures due to vanishing gradient problems. This paper addresses the simultaneous training and topology optimization of RNNs using multiobjective hybrid procedures. The proposal is based on the SPEA2 and NSGA2 algorithms for making hybrid methods using the Baldwinian hybridization strategy. We also study the effects of the selection of the objectives, crossover, and mutation in the diversity during evolution. The proposals are tested in the experimental section to train and optimize the networks in the competition on artificial time-series (CATS) benchmark.

Direction-dependent learning approach for radial basis function networks

Direction-dependent scaling, shaping, and rotation of Gaussian basis functions are introduced for maximal trend sensing with minimal parameter representations for input output approximation. It is shown that shaping and rotation of the radial basis functions helps in reducing the total number of function units required to approximate any given input-output data, while improving accuracy. Several alternate formulations that enforce minimal parameterization of the most general radial basis functions are presented. A novel "directed graph" based algorithm is introduced to facilitate intelligent direction based learning and adaptation of the parameters appearing in the radial basis function network. Further, a parameter estimation algorithm is incorporated to establish starting estimates for the model parameters using multiple windows of the input-output data. The efficacy of direction-dependent shaping and rotation in function approximation is evaluated by modifying the minimal resource allocating network and considering different test examples. The examples are drawn from recent literature to benchmark the new algorithm versus existing methods.

A Hybrid Forward Algorithm for RBF Neural Network Construction

This paper proposes a novel hybrid forward algorithm (HFA) for the construction of radial basis function (RBF) neural networks with tunable nodes. The main objective is to efficiently and effectively produce a parsimonious RBF neural network that generalizes well. In this study, it is achieved through simultaneous network structure determination and parameter optimization on the continuous parameter space. This is a mixed integer hard problem and the proposed HFA tackles this problem using an integrated analytic framework, leading to significantly improved network performance and reduced memory usage for the network construction. The computational complexity analysis confirms the efficiency of the proposed algorithm, and the simulation results demonstrate its effectiveness.