An optimization methodology for neural network weights and architectures

Geometrical Interpretation and Design of Multilayer Perceptrons

The multilayer perceptron (MLP) neural network is interpreted from the geometrical viewpoint in this work, that is, an MLP partition an input feature space into multiple nonoverlapping subspaces using a set of hyperplanes, where the great majority of samples in a subspace belongs to one object class. Based on this high-level idea, we propose a three-layer feedforward MLP (FF-MLP) architecture for its implementation. In the first layer, the input feature space is split into multiple subspaces by a set of partitioning hyperplanes and rectified linear unit (ReLU) activation, which is implemented by the classical two-class linear discriminant analysis (LDA). In the second layer, each neuron activates one of the subspaces formed by the partitioning hyperplanes with specially designed weights. In the third layer, all subspaces of the same class are connected to an output node that represents the object class. The proposed design determines all MLP parameters in a feedforward one-pass fashion analytically without backpropagation. Experiments are conducted to compare the performance of the traditional backpropagation-based MLP (BP-MLP) and the new FF-MLP. It is observed that the FF-MLP outperforms the BP-MLP in terms of design time, training time, and classification performance in several benchmarking datasets. Our source code is available at https://colab.research.google.com/drive/1Gz0L8AnT4ijrUchrhEXXsnaacrFdenn?usp = sharing.

BatTS: a hybrid method for optimizing deep feedforward neural network

Deep feedforward neural networks (DFNNs) have attained remarkable success in almost every computational task. However, the selection of DFNN architecture is still based on handcraft or hit-and-trial methods. Therefore, an essential factor regarding DFNN is about designing its architecture. Unfortunately, creating architecture for DFNN is a very laborious and time-consuming task for performing state-of-art work. This article proposes a new hybrid methodology (BatTS) to optimize the DFNN architecture based on its performance. BatTS is a result of integrating the Bat algorithm, Tabu search (TS), and Gradient descent with a momentum backpropagation training algorithm (GDM). The main features of the BatTS are the following: a dynamic process of finding new architecture based on Bat, the skill to escape from local minima, and fast convergence in evaluating new architectures based on the Tabu search feature. The performance of BatTS is compared with the Tabu search based approach and random trials. The process goes through an empirical evaluation of four different benchmark datasets and shows that the proposed hybrid methodology has improved performance over existing techniques which are mainly random trials.

A Novel Genetic Neural Network Algorithm with Link Switches and Its Application in University Professional Course Evaluation

This study exploits a novel enhanced genetic neural network algorithm with link switches (EGA-NNLS) to model the professional university course evaluating system. Various indices should be employed to evaluate the learning effect of a professional course comprehensively and objectively, and the traditional artificial evaluation methods cannot achieve this goal. The presented data-driven modeling method, EGA-NNLS, combines a neural network with link switches (NN-LS) with an enhanced genetic algorithm (EGA) and the Levenberg-Marquardt (LM) algorithm. It employs an optimized network structure combined with EGA and NN-LS to learn the relationships between the system's input and output from historical data and uses the network's gradient information via the LM algorithm. Compared with the traditional backpropagation neural network (BPNN), EGA-NNLS achieves a faster convergence speed and higher evaluation precision. In order to verify the efficiency of EGA-NNLS, it is applied to a collection of experimental data for modeling the professional university course evaluating system.

Fast Seismic Assessment of Built Urban Areas with the Accuracy of Mechanical Methods Using a Feedforward Neural Network

Capacity curves obtained from nonlinear static analyses are widely used to perform seismic assessments of structures as an alternative to dynamic analysis. This paper presents a novel ‘en masse’ method to assess the seismic vulnerability of urban areas swiftly and with the accuracy of mechanical methods. At the core of this methodology is the calculation of the capacity curves of low-rise reinforced concrete buildings using neural networks, where no modeling of the building is required. The curves are predicted with minimal error, needing only basic geometric and material parameters of the structures to be specified. As a first implementation, a typology of prismatic buildings is defined and a training set of more than 7000 structures generated. The capacity curves are calculated through push-over analysis using SAP2000. The results feature the prediction of 100-point curves in a single run of the network while maintaining a very low mean absolute error. This paper proposes a method that improves current seismic assessment tools by providing a fast and accurate calculation of the vulnerability of large sets of buildings in urban environments.

Artificial Neural Networks for Modelling and Predicting Urban Air Pollutants: Case of Lithuania

This study focuses on the Vilnius (capital of Lithuania) agglomeration, which is facing the issue of air pollution resulting from the city’s physical expansion. The increased number of industries and vehicles caused an increase in the rate of fuel consumption and pollution in Vilnius, which has rendered air pollution control policies and air pollution management more significant. In this study, the differences in the pollutants’ means were tested using two-sided t-tests. Additionally, a 2-layer artificial neural network and a pollution data were both used as tools for predicting and warning air pollution after loop traffic has taken effect in Vilnius Old Town from July of 2020. Highly accurate data analysis methods provide reliable data for predicting air pollution. According to the validation, the multilayer perceptron network (MLPN1), with a hyperbolic tangent activation function with a 4-4-2 partition, produced valuable results and identified the main pollutants affecting and predicting air quality in the Old Town: maximum concentration of sulphur dioxide per 1 hour (SO2_1 h, normalized importance = 100%); carbon monoxide (CO) was the second pollutant with the highest indication of normalized importance, equalling 59.0%.

Using Artificial Neural Networks in Predicting the Level of Stress among Military Conscripts

The present study aims to elucidate the main variables that increase the level of stress at the beginning of military conscription service using an artificial neural network (ANN)-based prediction model. Random sample data were obtained from one battalion of the Lithuanian Armed Forces, and a survey was conducted to generate data for the training and testing of the ANN models. Using nonlinearity in stress research, numerous ANN structures were constructed and verified to limit the optimal number of neurons, hidden layers, and transfer functions. The highest accuracy was obtained by the multilayer perceptron neural network (MLPNN) with a 6-2-2 partition. A standardized rescaling method was used for covariates. For the activation function, the hyperbolic tangent was used with 20 units in one hidden layer as well as the back-propagation algorithm. The best ANN model was determined as the model that showed the smallest cross-entropy error, the correct classification rate, and the area under the ROC curve. These findings show, with high precision, that cohesion in a team and adaptation to military routines are two critical elements that have the greatest impact on the stress level of conscripts.

Multilayer Perceptron New Method for Selecting the Architecture Based on the Choice of Different Activation Functions

Multilayer perceptron has a large amount of classifications and regression applications in many fields: pattern recognition, voice, and classification problems. But the architecture choice in particular, the activation function type used for each neuron has a great impact on the convergence and performance. In the present article, the authors introduce a new approach to optimize the selection of network architecture, weights, and activation functions. To solve the obtained model the authors use a genetic algorithm and train the network with a back-propagation method. The numerical results show the effectiveness of the approach shown in this article, and the advantages of the new model compared to the existing previous model in the literature.

Sleep apnea detection from a single-lead ECG signal with automatic feature-extraction through a modified LeNet-5 convolutional neural network

Sleep apnea (SA) is the most common respiratory sleep disorder, leading to some serious neurological and cardiovascular diseases if left untreated. The diagnosis of SA is traditionally made using Polysomnography (PSG). However, this method requires many electrodes and wires, as well as an expert to monitor the test. Several researchers have proposed instead using a single channel signal for SA diagnosis. Among these options, the ECG signal is one of the most physiologically relevant signals of SA occurrence, and one that can be easily recorded using a wearable device. However, existing ECG signal-based methods mainly use features (i.e. frequency domain, time domain, and other nonlinear features) acquired from ECG and its derived signals in order to construct the model. This requires researchers to have rich experience in ECG, which is not common. A convolutional neural network (CNN) is a kind of deep neural network that can automatically learn effective feature representation from training data and has been successfully applied in many fields. Meanwhile, most studies have not considered the impact of adjacent segments on SA detection. Therefore, in this study, we propose a modified LeNet-5 convolutional neural network with adjacent segments for SA detection. Our experimental results show that our proposed method is useful for SA detection, and achieves better or comparable results when compared with traditional machine learning methods.

Dew Point Temperature Estimation: Application of Artificial Intelligence Model Integrated with Nature-Inspired Optimization Algorithms

Dew point temperature (DPT) is known to fluctuate in space and time regardless of the climatic zone considered. The accurate estimation of the DPT is highly significant for various applications of hydro and agro–climatological researches. The current research investigated the hybridization of a multilayer perceptron (MLP) neural network with nature-inspired optimization algorithms (i.e., gravitational search (GSA) and firefly (FFA)) to model the DPT of two climatically contrasted (humid and semi-arid) regions in India. Daily time scale measured weather information, such as wet bulb temperature (WBT), vapor pressure (VP), relative humidity (RH), and dew point temperature, was used to build the proposed predictive models. The efficiencies of the proposed hybrid MLP networks (MLP–FFA and MLP–GSA) were authenticated against standard MLP tuned by a Levenberg–Marquardt back-propagation algorithm, extreme learning machine (ELM), and support vector machine (SVM) models. Statistical evaluation metrics such as Nash Sutcliffe efficiency (NSE), root mean square error (RMSE), and mean absolute error (MAE) were used to validate the model efficiency. The proposed hybrid MLP models exhibited excellent estimation accuracy. The hybridization of MLP with nature-inspired optimization algorithms boosted the estimation accuracy that is clearly owing to the tuning robustness. In general, the applied methodology showed very convincing results for both inspected climate zones.

Artificial Intelligence-Based Grading Quality of Bovine Blastocyst Digital Images: Direct Capture with Juxtaposed Lenses of Smartphone Camera and Stereomicroscope Ocular Lens

In this study, we developed an online graphical and intuitive interface connected to a server aiming to facilitate professional access worldwide to those facing problems with bovine blastocysts classification. The interface Blasto3Q, where 3Q refers to the three qualities of the blastocyst grading, contains a description of 24 variables that were extracted from the image of the blastocyst and analyzed by three Artificial Neural Networks (ANNs) that classify the same loaded image. The same embryo (i.e., the biological specimen) was submitted to digital image capture by the control group (inverted microscope with 40× magnification) and the experimental group (stereomicroscope with maximum of magnification plus 4× zoom from the cell phone camera). The images obtained from the control and experimental groups were uploaded on Blasto3Q. Each image from both sources was evaluated for segmentation and submitted (only if it could be properly or partially segmented) for automatic quality grade classification by the three ANNs of the Blasto3Q program. Adjustments on the software program through the use of scaling algorithm software were performed to ensure the proper search and segmentation of the embryo in the raw images when they were captured by the smartphone, since this source produced small embryo images compared with those from the inverted microscope. With this new program, 77.8% of the images from smartphones were successfully segmented and from those, 85.7% were evaluated by the Blasto3Q in agreement with the control group.

State-of-the-art in artificial neural network applications: A survey

This is a survey of neural network applications in the real-world scenario. It provides a taxonomy of artificial neural networks (ANNs) and furnish the reader with knowledge of current and emerging trends in ANN applications research and area of focus for researchers. Additionally, the study presents ANN application challenges, contributions, compare performances and critiques methods. The study covers many applications of ANN techniques in various disciplines which include computing, science, engineering, medicine, environmental, agriculture, mining, technology, climate, business, arts, and nanotechnology, etc. The study assesses ANN contributions, compare performances and critiques methods. The study found that neural-network models such as feedforward and feedback propagation artificial neural networks are performing better in its application to human problems. Therefore, we proposed feedforward and feedback propagation ANN models for research focus based on data analysis factors like accuracy, processing speed, latency, fault tolerance, volume, scalability, convergence, and performance. Moreover, we recommend that instead of applying a single method, future research can focus on combining ANN models into one network-wide application.

Optimization of Deep Neural Networks Using SoCs with OpenCL

In the optimization of deep neural networks (DNNs) via evolutionary algorithms (EAs) and the implementation of the training necessary for the creation of the objective function, there is often a trade-off between efficiency and flexibility. Pure software solutions implemented on general-purpose processors tend to be slow because they do not take advantage of the inherent parallelism of these devices, whereas hardware realizations based on heterogeneous platforms (combining central processing units (CPUs), graphics processing units (GPUs) and/or field-programmable gate arrays (FPGAs)) are designed based on different solutions using methodologies supported by different languages and using very different implementation criteria. This paper first presents a study that demonstrates the need for a heterogeneous (CPU-GPU-FPGA) platform to accelerate the optimization of artificial neural networks (ANNs) using genetic algorithms. Second, the paper presents implementations of the calculations related to the individuals evaluated in such an algorithm on different (CPU- and FPGA-based) platforms, but with the same source files written in OpenCL. The implementation of individuals on remote, low-cost FPGA systems on a chip (SoCs) is found to enable the achievement of good efficiency in terms of performance per watt.

A Method Based on Artificial Intelligence To Fully Automatize The Evaluation of Bovine Blastocyst Images

Morphological analysis is the standard method of assessing embryo quality; however, its inherent subjectivity tends to generate discrepancies among evaluators. Using genetic algorithms and artificial neural networks (ANNs), we developed a new method for embryo analysis that is more robust and reliable than standard methods. Bovine blastocysts produced in vitro were classified as grade 1 (excellent or good), 2 (fair), or 3 (poor) by three experienced embryologists according to the International Embryo Technology Society (IETS) standard. The images (n = 482) were subjected to automatic feature extraction, and the results were used as input for a supervised learning process. One part of the dataset (15%) was used for a blind test posterior to the fitting, for which the system had an accuracy of 76.4%. Interestingly, when the same embryologists evaluated a sub-sample (10%) of the dataset, there was only 54.0% agreement with the standard (mode for grades). However, when using the ANN to assess this sub-sample, there was 87.5% agreement with the modal values obtained by the evaluators. The presented methodology is covered by National Institute of Industrial Property (INPI) and World Intellectual Property Organization (WIPO) patents and is currently undergoing a commercial evaluation of its feasibility.

Master-Leader-Slave Cuckoo Search with Parameter Control for ANN Optimization and Its Real-World Application to Water Quality Prediction

Artificial neural networks (ANNs) have been employed to solve a broad variety of tasks. The selection of an ANN model with appropriate weights is important in achieving accurate results. This paper presents an optimization strategy for ANN model selection based on the cuckoo search (CS) algorithm, which is rooted in the obligate brood parasitic actions of some cuckoo species. In order to enhance the convergence ability of basic CS, some modifications are proposed. The fraction P_a of the n nests replaced by new nests is a fixed parameter in basic CS. As the selection of P_a is a challenging issue and has a direct effect on exploration and therefore on convergence ability, in this work the P_a is set to a maximum value at initialization to achieve more exploration in early iterations and it is decreased during the search to achieve more exploitation in later iterations until it reaches the minimum value in the final iteration. In addition, a novel master-leader-slave multi-population strategy is used where the slaves employ the best fitness function among all slaves, which is selected by the leader under a certain condition. This fitness function is used for subsequent Lévy flights. In each iteration a copy of the best solution of each slave is migrated to the master and then the best solution is found by the master. The method is tested on benchmark classification and time series prediction problems and the statistical analysis proves the ability of the method. This method is also applied to a real-world water quality prediction problem with promising results.

Data-Driven Modeling for UGI Gasification Processes via an Enhanced Genetic BP Neural Network With Link Switches

In this brief, an enhanced genetic back-propagation neural network with link switches (EGA-BPNN-LS) is proposed to address a data-driven modeling problem for gasification processes inside United Gas Improvement (UGI) gasifiers. The online-measured temperature of crude gas produced during the gasification processes plays a dominant role in the syngas industry; however, it is difficult to model temperature dynamics via first principles due to the practical complexity of the gasification process, especially as reflected by severe changes in the gas temperature resulting from infrequent manipulations of the gasifier in practice. The proposed data-driven modeling approach, EGA-BPNN-LS, incorporates an NN-LS, an EGA, and the Levenberg-Marquardt (LM) algorithm. The approach cannot only learn the relationships between the control input and the system output from historical data using an optimized network structure through a combination of EGA and NN-LS but also makes use of the networks gradient information via the LM algorithm. EGA-BPNN-LS is applied to a set of data collected from the field to model the UGI gasification processes, and the effectiveness of EGA-BPNN-LS is verified.

MLPNN Training via a Multiobjective Optimization of Training Error and Stochastic Sensitivity

The training of a multilayer perceptron neural network (MLPNN) concerns the selection of its architecture and the connection weights via the minimization of both the training error and a penalty term. Different penalty terms have been proposed to control the smoothness of the MLPNN for better generalization capability. However, controlling its smoothness using, for instance, the norm of weights or the Vapnik-Chervonenkis dimension cannot distinguish individual MLPNNs with the same number of free parameters or the same norm. In this paper, to enhance generalization capabilities, we propose a stochastic sensitivity measure (ST-SM) to realize a new penalty term for MLPNN training. The ST-SM determines the expectation of the squared output differences between the training samples and the unseen samples located within their Q -neighborhoods for a given MLPNN. It provides a direct measurement of the MLPNNs output fluctuations, i.e., smoothness. We adopt a two-phase Pareto-based multiobjective training algorithm for minimizing both the training error and the ST-SM as biobjective functions. Experiments on 20 UCI data sets show that the MLPNNs trained by the proposed algorithm yield better accuracies on testing data than several recent and classical MLPNN training methods.

Quantum-based algorithm for optimizing artificial neural networks

This paper presents a quantum-based algorithm for evolving artificial neural networks (ANNs). The aim is to design an ANN with few connections and high classification performance by simultaneously optimizing the network structure and the connection weights. Unlike most previous studies, the proposed algorithm uses quantum bit representation to codify the network. As a result, the connectivity bits do not indicate the actual links but the probability of the existence of the connections, thus alleviating mapping problems and reducing the risk of throwing away a potential candidate. In addition, in the proposed model, each weight space is decomposed into subspaces in terms of quantum bits. Thus, the algorithm performs a region by region exploration, and evolves gradually to find promising subspaces for further exploitation. This is helpful to provide a set of appropriate weights when evolving the network structure and to alleviate the noisy fitness evaluation problem. The proposed model is tested on four benchmark problems, namely breast cancer and iris, heart, and diabetes problems. The experimental results show that the proposed algorithm can produce compact ANN structures with good generalization ability compared to other algorithms.

Performance evaluation of multilayer perceptrons for discriminating and quantifying multiple kinds of odors with an electronic nose

This paper studies several types and arrangements of perceptron modules to discriminate and quantify multiple odors with an electronic nose. We evaluate the following types of multilayer perceptron. (A) A single multi-output (SMO) perceptron both for discrimination and for quantification. (B) An SMO perceptron for discrimination followed by multiple multi-output (MMO) perceptrons for quantification. (C) An SMO perceptron for discrimination followed by multiple single-output (MSO) perceptrons for quantification. (D) MSO perceptrons for discrimination followed by MSO perceptrons for quantification, called the MSO-MSO perceptron model, under the following conditions: (D1) using a simple one-against-all (OAA) decomposition method; (D2) adopting a simple OAA decomposition method and virtual balance step; and (D3) employing a local OAA decomposition method, virtual balance step and local generalization strategy all together. The experimental results for 12 kinds of volatile organic compounds at 85 concentration levels in the training set and 155 concentration levels in the test set show that the MSO-MSO perceptron model with the D3 learning procedure is the most effective of those tested for discrimination and quantification of many kinds of odors.

Hybrid Training Method for MLP: Optimization of Architecture and Training

The performance of an artificial neural network (ANN) depends upon the selection of proper connection weights, network architecture, and cost function during network training. This paper presents a hybrid approach (GaTSa) to optimize the performance of the ANN in terms of architecture and weights. GaTSa is an extension of a previous method (TSa) proposed by the authors. GaTSa is based on the integration of the heuristic simulated annealing (SA), tabu search (TS), genetic algorithms (GA), and backpropagation, whereas TSa does not use GA. The main advantages of GaTSa are the following: a constructive process to add new nodes in the architecture based on GA, the ability to escape from local minima with uphill moves (SA feature), and faster convergence by the evaluation of a set of solutions (TS feature). The performance of GaTSa is investigated through an empirical evaluation of 11 public-domain data sets using different cost functions in the simultaneous optimization of the multilayer perceptron ANN architecture and weights. Experiments demonstrated that GaTSa can also be used for relevant feature selection. GaTSa presented statistically relevant results in comparison with other global and local optimization techniques.

Configuration of continuous piecewise-linear neural networks

The problem of constructing a general continuous piecewise-linear neural network is considered in this paper. It is shown that every projection domain of an arbitrary continuous piecewise-linear function can be partitioned into convex polyhedra by using difference functions of its local linear functions. Based on these convex polyhedra, a group of continuous piecewise-linear basis functions are formulated. It is proven that a linear combination of these basis functions plus a constant, which we call a standard continuous piecewise-linear neural network, can represent all continuous piecewise-linear functions. In addition, the proposed standard continuous piecewise-linear neural network is applied to solve some function approximation problems. A number of numerical experiments are presented to illustrate that the standard continuous piecewise-linear neural network can be a promising tool for function approximation.

Global convergence and limit cycle behavior of weights of perceptron

In this paper, it is found that the weights of a perceptron are bounded for all initial weights if there exists a nonempty set of initial weights that the weights of the perceptron are bounded. Hence, the boundedness condition of the weights of the perceptron is independent of the initial weights. Also, a necessary and sufficient condition for the weights of the perceptron exhibiting a limit cycle behavior is derived. The range of the number of updates for the weights of the perceptron required to reach the limit cycle is estimated. Finally, it is suggested that the perceptron exhibiting the limit cycle behavior can be employed for solving a recognition problem when downsampled sets of bounded training feature vectors are linearly separable. Numerical computer simulation results show that the perceptron exhibiting the limit cycle behavior can achieve a better recognition performance compared to a multilayer perceptron.