Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm

Image response regression via deep neural networks

Delineating associations between images and covariates is a central aim of imaging studies. To tackle this problem, we propose a novel non-parametric approach in the framework of spatially varying coefficient models, where the spatially varying functions are estimated through deep neural networks. Our method incorporates spatial smoothness, handles subject heterogeneity, and provides straightforward interpretations. It is also highly flexible and accurate, making it ideal for capturing complex association patterns. We establish estimation and selection consistency and derive asymptotic error bounds. We demonstrate the method's advantages through intensive simulations and analyses of two functional magnetic resonance imaging data sets.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

A Novel Genetic Neural Network Algorithm with Link Switches and Its Application in University Professional Course Evaluation

This study exploits a novel enhanced genetic neural network algorithm with link switches (EGA-NNLS) to model the professional university course evaluating system. Various indices should be employed to evaluate the learning effect of a professional course comprehensively and objectively, and the traditional artificial evaluation methods cannot achieve this goal. The presented data-driven modeling method, EGA-NNLS, combines a neural network with link switches (NN-LS) with an enhanced genetic algorithm (EGA) and the Levenberg-Marquardt (LM) algorithm. It employs an optimized network structure combined with EGA and NN-LS to learn the relationships between the system's input and output from historical data and uses the network's gradient information via the LM algorithm. Compared with the traditional backpropagation neural network (BPNN), EGA-NNLS achieves a faster convergence speed and higher evaluation precision. In order to verify the efficiency of EGA-NNLS, it is applied to a collection of experimental data for modeling the professional university course evaluating system.

Two-Phase Switching Optimization Strategy in Deep Neural Networks

Optimization in a deep neural network is always challenging due to the vanishing gradient problem and intensive fine-tuning of network hyperparameters. Inspired by multistage decision control systems, the stochastic diagonal approximate greatest descent (SDAGD) algorithm is proposed in this article to seek for optimal learning weights using a two-phase switching optimization strategy. The proposed optimizer controls the relative step length derived based on the long-term optimal trajectory and adopts the diagonal approximated Hessian for efficient weight update. In Phase-I, it computes the greatest step length at the boundary of each local spherical search region and, subsequently, descends rapidly toward the direction of an optimal solution. In Phase-II, it switches to an approximate Newton method automatically once it is closer to the optimal solution to achieve fast convergence. The experiments show that SDAGD produces steeper learning curves and achieves lower misclassification rates compared with other optimization techniques. Implementation of the proposed optimizer to deeper networks is also investigated in this article to study the vanishing gradient problem.

A Bilevel Learning Model and Algorithm for Self-Organizing Feed-Forward Neural Networks for Pattern Classification

Conventional artificial neural network (ANN) learning algorithms for classification tasks, either derivative-based optimization algorithms or derivative-free optimization algorithms work by training ANN first (or training and validating ANN) and then testing ANN, which are a two-stage and one-pass learning mechanism. Thus, this learning mechanism may not guarantee the generalization ability of a trained ANN. In this article, a novel bilevel learning model is constructed for self-organizing feed-forward neural network (FFNN), in which the training and testing processes are integrated into a unified framework. In this bilevel model, the upper level optimization problem is built for testing error on testing data set and network architecture based on network complexity, whereas the lower level optimization problem is constructed for network weights based on training error on training data set. For the bilevel framework, an interactive learning algorithm is proposed to optimize the architecture and weights of an FFNN with consideration of both training error and testing error. In this interactive learning algorithm, a hybrid binary particle swarm optimization (BPSO) taken as an upper level optimizer is used to self-organize network architecture, whereas the Levenberg-Marquardt (LM) algorithm as a lower level optimizer is utilized to optimize the connection weights of an FFNN. The bilevel learning model and algorithm have been tested on 20 benchmark classification problems. Experimental results demonstrate that the bilevel learning algorithm can significantly produce more compact FFNNs with more excellent generalization ability when compared with conventional learning algorithms.

Optimization of vector convolutional deep neural network using binary real cumulative incarnation for detection of distributed denial of service attacks

In today’s technological world, distributed denial of service (DDoS) attacks threaten Internet users by flooding huge network traffic to make critical Internet services unavailable to genuine users. Therefore, design of DDoS attack detection system is on urge to mitigate these attacks for protecting the critical services. Nowadays, deep learning techniques are extensively used to detect these attacks. The existing deep feature learning approaches face the lacuna of designing an appropriate deep neural network structure for detection of DDoS attacks which leads to poor performance in terms of accuracy and false alarm. In this article, a tuned vector convolutional deep neural network (TVCDNN) is proposed by optimizing the structure and parameters of the deep neural network using binary and real cumulative incarnation (CuI), respectively. The CuI is a genetic-based optimization technique which optimizes the tuning process by providing values generated from best-fit parents. The TVCDNN is tested with publicly available benchmark network traffic datasets and compared with existing classifiers and optimization techniques. It is evident that the proposed optimization approach yields promising results compared to the existing optimization techniques. Further, the proposed approach achieves significant improvement in performance over the state-of-the-art attack detection systems.

The role of foreign technologies and R&D in innovation processes within catching-up CEE countries

Prior research showed that there is a growing consensus among researchers, which point out a key role of external knowledge sources such as external R&D and technologies in enhancing firms´ innovation. However, firms´ from catching-up Central and Eastern European (CEE) countries have already shown in the past that their innovation models differ from those applied, for example, in Western Europe. This study therefore introduces a novel two-staged model combining artificial neural networks and random forests to reveal the importance of internal and external factors influencing firms´ innovation performance in the case of 3,361 firms from six catching-up CEE countries (Czech Republic, Slovakia, Poland, Estonia, Latvia and Lithuania), by using the World Banks´ Enterprise Survey data from 2019. We confirm the hypothesis that innovators in the catching-up CEE countries depend more on internal knowledge sources and, moreover, that participation in the firms groups represents an important factor of firms´ innovation. Surprisingly, we reject the hypothesis that foreign technologies are a crucial source of external knowledge. This study contributes to the theories of open innovation and absorptive capacity in the context of selected CEE countries and provides several practical implications for firms.

Memetic algorithms for training feedforward neural networks: an approach based on gravitational search algorithm

The backpropagation (BP) algorithm is a gradient-based algorithm used for training a feedforward neural network (FNN). Despite the fact that BP is still used today when FNNs are trained, it has some disadvantages, including the following: (i) it fails when non-differentiable functions are addressed, (ii) it can become trapped in local minima, and (iii) it has slow convergence. In order to solve some of these problems, metaheuristic algorithms have been used to train FNN. Although they have good exploration skills, they are not as good as gradient-based algorithms at exploitation tasks. The main contribution of this article lies in its application of novel memetic approaches based on the Gravitational Search Algorithm (GSA) and Chaotic Gravitational Search Algorithm (CGSA) algorithms, called respectively Memetic Gravitational Search Algorithm (MGSA) and Memetic Chaotic Gravitational Search Algorithm (MCGSA), to train FNNs in three classical benchmark problems: the XOR problem, the approximation of a continuous function, and classification tasks. The results show that both approaches constitute suitable alternatives for training FNNs, even improving on the performance of other state-of-the-art metaheuristic algorithms such as ParticleSwarm Optimization (PSO), the Genetic Algorithm (GA), the Adaptive Differential Evolution algorithm with Repaired crossover rate (Rcr-JADE), and the Covariance matrix learning and Bimodal distribution parameter setting Differential Evolution (COBIDE) algorithm. Swarm optimization, the genetic algorithm, the adaptive differential evolution algorithm with repaired crossover rate, and the covariance matrix learning and bimodal distribution parameter setting differential evolution algorithm.

Parameter Setting for Deep Neural Networks Using Swarm Intelligence on Phishing Websites Classification

Over the past years, the application of deep neural networks in a wide range of areas is noticeably increasing. While many state-of-the-art deep neural networks are providing the performance comparable or in some cases even superior to humans, major challenges such as parameter settings for learning deep neural networks and construction of deep learning architectures still exist. The implications of those challenges have a significant impact on how a deep neural network is going to perform on a specific task. With the proposed method, presented in this paper, we are addressing the problem of parameter setting for a deep neural network utilizing swarm intelligence algorithms. In our experiments, we applied the proposed method variants to the classification task for distinguishing between phishing and legitimate websites. The performance of the proposed method is evaluated and compared against four different phishing datasets, two of which we prepared on our own. The results, obtained from the conducted empirical experiments, have proven the proposed approach to be very promising. By utilizing the proposed swarm intelligence based methods, we were able to statistically significantly improve the predictive performance when compared to the manually tuned deep neural network. In general, the improvement of classification accuracy ranges from 2.5% to 3.8%, while the improvement of F1-score reached even 24% on one of the datasets.

Predicting Surgical Complications in Patients Undergoing Elective Adult Spinal Deformity Procedures Using Machine Learning

STUDY DESIGN

Cross-sectional database study.

OBJECTIVE

To train and validate machine learning models to identify risk factors for complications following surgery for adult spinal deformity (ASD).

SUMMARY OF BACKGROUND DATA

Machine learning models such as logistic regression (LR) and artificial neural networks (ANNs) are valuable tools for analyzing and interpreting large and complex data sets. ANNs have yet to be used for risk factor analysis in orthopedic surgery.

METHODS

The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) database was queried for patients who underwent surgery for ASD. This query returned 4,073 patients, which data were used to train and evaluate our models. The predictive variables used included sex, age, ethnicity, diabetes, smoking, steroid use, coagulopathy, functional status, American Society of Anesthesiologists (ASA) class >3, body mass index (BMI), pulmonary comorbidities, and cardiac comorbidities. The models were used to predict cardiac complications, wound complications, venous thromboembolism (VTE), and mortality. Using ASA class as a benchmark for prediction, area under receiver operating characteristic curves (AUC) was used to determine the accuracy of our machine learning models.

RESULTS

The mean age of patients was 59.5 years. Forty-one percent of patients were male whereas 59.0% of patients were female. ANN and LR outperformed ASA scoring in predicting every complication (p<.05). The ANN outperformed LR in predicting cardiac complication, wound complication, and mortality (p<.05).

CONCLUSIONS

Machine learning algorithms outperform ASA scoring for predicting individual risk prognosis. These algorithms also outperform LR in predicting individual risk for all complications except VTE. With the growing size of medical data, the training of machine learning on these large data sets promises to improve risk prognostication, with the ability of continuously learning making them excellent tools in complex clinical scenarios.

LEVEL OF EVIDENCE

Level III.

Predicting Surgical Complications in Adult Patients Undergoing Anterior Cervical Discectomy and Fusion Using Machine Learning

Objective

Machine learning algorithms excel at leveraging big data to identify complex patterns that can be used to aid in clinical decision-making. The objective of this study is to demonstrate the performance of machine learning models in predicting postoperative complications following anterior cervical discectomy and fusion (ACDF).

Methods

Artificial neural network (ANN), logistic regression (LR), support vector machine (SVM), and random forest decision tree (RF) models were trained on a multicenter data set of patients undergoing ACDF to predict surgical complications based on readily available patient data. Following training, these models were compared to the predictive capability of American Society of Anesthesiologists (ASA) physical status classification.

Results

A total of 20,879 patients were identified as having undergone ACDF. Following exclusion criteria, patients were divided into 14,615 patients for training and 6,264 for testing data sets. ANN and LR consistently outperformed ASA physical status classification in predicting every complication (p < 0.05). The ANN outperformed LR in predicting venous thromboembolism, wound complication, and mortality (p < 0.05). The SVM and RF models were no better than random chance at predicting any of the postoperative complications (p < 0.05).

Conclusion

ANN and LR algorithms outperform ASA physical status classification for predicting individual postoperative complications. Additionally, neural networks have greater sensitivity than LR when predicting mortality and wound complications. With the growing size of medical data, the training of machine learning on these large datasets promises to improve risk prognostication, with the ability of continuously learning making them excellent tools in complex clinical scenarios.

Optimization of Deep Neural Networks Using SoCs with OpenCL

In the optimization of deep neural networks (DNNs) via evolutionary algorithms (EAs) and the implementation of the training necessary for the creation of the objective function, there is often a trade-off between efficiency and flexibility. Pure software solutions implemented on general-purpose processors tend to be slow because they do not take advantage of the inherent parallelism of these devices, whereas hardware realizations based on heterogeneous platforms (combining central processing units (CPUs), graphics processing units (GPUs) and/or field-programmable gate arrays (FPGAs)) are designed based on different solutions using methodologies supported by different languages and using very different implementation criteria. This paper first presents a study that demonstrates the need for a heterogeneous (CPU-GPU-FPGA) platform to accelerate the optimization of artificial neural networks (ANNs) using genetic algorithms. Second, the paper presents implementations of the calculations related to the individuals evaluated in such an algorithm on different (CPU- and FPGA-based) platforms, but with the same source files written in OpenCL. The implementation of individuals on remote, low-cost FPGA systems on a chip (SoCs) is found to enable the achievement of good efficiency in terms of performance per watt.

Neural network for regression problems with reduced training sets

Although they are powerful and successful in many applications, artificial neural networks (ANNs) typically do not perform well with complex problems that have a limited number of training cases. Often, collecting additional training data may not be feasible or may be costly. Thus, this work presents a new radial-basis network (RBN) design that overcomes the limitations of using ANNs to accurately model regression problems with minimal training data. This new design involves a multi-stage training process that couples an orthogonal least squares (OLS) technique with gradient-based optimization. New termination criteria are also introduced to improve accuracy. In addition, the algorithms are designed to require minimal heuristic parameters, thus improving ease of use and consistency in performance. The proposed approach is tested with experimental and practical regression problems, and the results are compared with those from typical network models. The results show that the new design demonstrates improved accuracy with reduced dependence on the amount of training data. As demonstrated, this new ANN provides a platform for approximating potentially slow but high-fidelity computational models, and thus fostering inter-model connectivity and multi-scale modeling.

Disease-free survival assessment by artificial neural networks for hepatocellular carcinoma patients after radiofrequency ablation

BACKGROUND/PURPOSE

Radiofrequency ablation (RFA) provides an effective treatment for patients who exhibit early hepatocellular carcinoma (HCC) stages or are waiting for liver transplantation. It is important to assess patients after RFA. The goal of this study was to build artificial neural network models with HCC-related variables to predict the 1-year and 2-year disease-free survival (DFS) of HCC patients receiving RFA treatments.

METHODS

This study was a retrospective study that tracked HCC patients who received computer tomography-guided percutaneous RFA between January 2009 and April 2012. The numbers of total patients with 1-year and 2-year DFS were 252 and 179, respectively. A total of 15 HCC clinical variables were collected for the construction of artificial neural network models for DFS prediction. Internal validation and validation conducted using simulated prospective data were performed.

RESULTS

The results showed that the model with 15 inputs showed better performance compared with the models including only significant features. Parameters for performance assessment of 1-year DFS prediction were as follows: accuracy 85.0% (70.0%), sensitivity 75.0% (63.3%), specificity 87.5% (71.8%), and area under the curve 0.84 (0.77) for internal validation (simulated prospective validation). For 2-year DFS prediction, the values of accuracy, sensitivity, specificity, and area under the curve were 67.9% (63.9%), 50.0% (56.3%), 85.7% (70.0%), and 0.75 (0.72), respectively, for internal validation (simulated prospective validation).

CONCLUSION

This study revealed that the proposed artificial neural network models constructed with 15 clinical HCC relevant features could achieve an acceptable prediction performance for DFS. Such models can support clinical physicians to deal with clinical decision-making processes on the prognosis of HCC patients receiving RFA treatments.

Data-Driven Modeling for UGI Gasification Processes via an Enhanced Genetic BP Neural Network With Link Switches

In this brief, an enhanced genetic back-propagation neural network with link switches (EGA-BPNN-LS) is proposed to address a data-driven modeling problem for gasification processes inside United Gas Improvement (UGI) gasifiers. The online-measured temperature of crude gas produced during the gasification processes plays a dominant role in the syngas industry; however, it is difficult to model temperature dynamics via first principles due to the practical complexity of the gasification process, especially as reflected by severe changes in the gas temperature resulting from infrequent manipulations of the gasifier in practice. The proposed data-driven modeling approach, EGA-BPNN-LS, incorporates an NN-LS, an EGA, and the Levenberg-Marquardt (LM) algorithm. The approach cannot only learn the relationships between the control input and the system output from historical data using an optimized network structure through a combination of EGA and NN-LS but also makes use of the networks gradient information via the LM algorithm. EGA-BPNN-LS is applied to a set of data collected from the field to model the UGI gasification processes, and the effectiveness of EGA-BPNN-LS is verified.

A stable and optimized neural network model for crash injury severity prediction

The study proposes a convex combination (CC) algorithm to fast and stably train a neural network (NN) model for crash injury severity prediction, and a modified NN pruning for function approximation (N2PFA) algorithm to optimize the network structure. To demonstrate the proposed approaches and to compare them with the NN trained by traditional back-propagation (BP) algorithm and an ordered logit (OL) model, a two-vehicle crash dataset in 2006 provided by the Florida Department of Highway Safety and Motor Vehicles (DHSMV) was employed. According to the results, the CC algorithm outperforms the BP algorithm both in convergence ability and training speed. Compared with a fully connected NN, the optimized NN contains much less network nodes and achieves comparable classification accuracy. Both of them have better fitting and predicting performance than the OL model, which again demonstrates the NN's superiority over statistical models for predicting crash injury severity. The pruned input nodes also justify the ability of the structure optimization method for identifying the factors irrelevant to crash-injury outcomes. A sensitivity analysis of the optimized NN is further conducted to determine the explanatory variables' impact on each injury severity outcome. While most of the results conform to the coefficient estimation in the OL model and previous studies, some variables are found to have non-linear relationships with injury severity, which further verifies the strength of the proposed method.

Quantum-based algorithm for optimizing artificial neural networks

This paper presents a quantum-based algorithm for evolving artificial neural networks (ANNs). The aim is to design an ANN with few connections and high classification performance by simultaneously optimizing the network structure and the connection weights. Unlike most previous studies, the proposed algorithm uses quantum bit representation to codify the network. As a result, the connectivity bits do not indicate the actual links but the probability of the existence of the connections, thus alleviating mapping problems and reducing the risk of throwing away a potential candidate. In addition, in the proposed model, each weight space is decomposed into subspaces in terms of quantum bits. Thus, the algorithm performs a region by region exploration, and evolves gradually to find promising subspaces for further exploitation. This is helpful to provide a set of appropriate weights when evolving the network structure and to alleviate the noisy fitness evaluation problem. The proposed model is tested on four benchmark problems, namely breast cancer and iris, heart, and diabetes problems. The experimental results show that the proposed algorithm can produce compact ANN structures with good generalization ability compared to other algorithms.

Leuconostoc mesenteroides growth in food products: prediction and sensitivity analysis by adaptive-network-based fuzzy inference systems

Background

An adaptive-network-based fuzzy inference system (ANFIS) was compared with an artificial neural network (ANN) in terms of accuracy in predicting the combined effects of temperature (10.5 to 24.5°C), pH level (5.5 to 7.5), sodium chloride level (0.25% to 6.25%) and sodium nitrite level (0 to 200 ppm) on the growth rate of Leuconostoc mesenteroides under aerobic and anaerobic conditions.

Methods

The ANFIS and ANN models were compared in terms of six statistical indices calculated by comparing their prediction results with actual data: mean absolute percentage error (MAPE), root mean square error (RMSE), standard error of prediction percentage (SEP), bias factor (B_f), accuracy factor (A_f), and absolute fraction of variance (R²). Graphical plots were also used for model comparison.

Conclusions

The learning-based systems obtained encouraging prediction results. Sensitivity analyses of the four environmental factors showed that temperature and, to a lesser extent, NaCl had the most influence on accuracy in predicting the growth rate of Leuconostoc mesenteroides under aerobic and anaerobic conditions. The observed effectiveness of ANFIS for modeling microbial kinetic parameters confirms its potential use as a supplemental tool in predictive mycology. Comparisons between growth rates predicted by ANFIS and actual experimental data also confirmed the high accuracy of the Gaussian membership function in ANFIS. Comparisons of the six statistical indices under both aerobic and anaerobic conditions also showed that the ANFIS model was better than all ANN models in predicting the four kinetic parameters. Therefore, the ANFIS model is a valuable tool for quickly predicting the growth rate of Leuconostoc mesenteroides under aerobic and anaerobic conditions.

Mortality predicted accuracy for hepatocellular carcinoma patients with hepatic resection using artificial neural network

The aim of this present study is firstly to compare significant predictors of mortality for hepatocellular carcinoma (HCC) patients undergoing resection between artificial neural network (ANN) and logistic regression (LR) models and secondly to evaluate the predictive accuracy of ANN and LR in different survival year estimation models. We constructed a prognostic model for 434 patients with 21 potential input variables by Cox regression model. Model performance was measured by numbers of significant predictors and predictive accuracy. The results indicated that ANN had double to triple numbers of significant predictors at 1-, 3-, and 5-year survival models as compared with LR models. Scores of accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) of 1-, 3-, and 5-year survival estimation models using ANN were superior to those of LR in all the training sets and most of the validation sets. The study demonstrated that ANN not only had a great number of predictors of mortality variables but also provided accurate prediction, as compared with conventional methods. It is suggested that physicians consider using data mining methods as supplemental tools for clinical decision-making and prognostic evaluation.