Improving the performance of mutation-based evolving artificial neural networks with self-adaptive mutations

Neuroevolution is a promising approach for designing artificial neural networks using an evolutionary algorithm. Unlike recent trending methods that rely on gradient-based algorithms, neuroevolution can simultaneously evolve the topology and weights of neural networks. In neuroevolution with topological evolution, handling crossover is challenging because of the competing conventions problem. Mutation-based evolving artificial neural network is an alternative topology and weights neuroevolution approach that omits crossover and uses only mutations for genetic variation. This study enhances the performance of mutation-based evolving artificial neural network in two ways. First, the mutation step size controlling the magnitude of the parameter perturbation is automatically adjusted by a self-adaptive mutation mechanism, enabling a balance between exploration and exploitation during the evolution process. Second, the structural mutation probabilities are automatically adjusted depending on the network size, preventing excessive expansion of the topology. The proposed methods are compared with conventional neuroevolution algorithms using locomotion tasks provided in the OpenAI Gym benchmarks. The results demonstrate that the proposed methods with the self-adaptive mutation mechanism can achieve better performance. In addition, the adjustment of structural mutation probabilities can mitigate topological bloat while maintaining performance.

Digital Pathology for Better Clinical Practice

(1) Background: Digital pathology (DP) is transforming the landscape of clinical practice, offering a revolutionary approach to traditional pathology analysis and diagnosis. (2) Methods: This innovative technology involves the digitization of traditional glass slides which enables pathologists to access, analyze, and share high-resolution whole-slide images (WSI) of tissue specimens in a digital format. By integrating cutting-edge imaging technology with advanced software, DP promises to enhance clinical practice in numerous ways. DP not only improves quality assurance and standardization but also allows remote collaboration among experts for a more accurate diagnosis. Artificial intelligence (AI) in pathology significantly improves cancer diagnosis, classification, and prognosis by automating various tasks. It also enhances the spatial analysis of tumor microenvironment (TME) and enables the discovery of new biomarkers, advancing their translation for therapeutic applications. (3) Results: The AI-driven immune assays, Immunoscore (IS) and Immunoscore-Immune Checkpoint (IS-IC), have emerged as powerful tools for improving cancer diagnosis, prognosis, and treatment selection by assessing the tumor immune contexture in cancer patients. Digital IS quantitative assessment performed on hematoxylin-eosin (H&E) and CD3+/CD8+ stained slides from colon cancer patients has proven to be more reproducible, concordant, and reliable than expert pathologists' evaluation of immune response. Outperforming traditional staging systems, IS demonstrated robust potential to enhance treatment efficiency in clinical practice, ultimately advancing cancer patient care. Certainly, addressing the challenges DP has encountered is essential to ensure its successful integration into clinical guidelines and its implementation into clinical use. (4) Conclusion: The ongoing progress in DP holds the potential to revolutionize pathology practices, emphasizing the need to incorporate powerful AI technologies, including IS, into clinical settings to enhance personalized cancer therapy.

Emerging Modularity During the Evolution of Neural Networks

Modularity is a feature of most small, medium and large–scale living organisms that has evolved over many years of evolution. A lot of artificial systems are also modular, however, in this case, the modularity is the most frequently a consequence of a handmade design process. Modular systems that emerge automatically, as a result of a learning process, are very rare. What is more, we do not know mechanisms which result in modularity. The main goal of the paper is to continue the work of other researchers on the origins of modularity, which is a form of optimal organization of matter, and the mechanisms that led to the spontaneous formation of modular living forms in the process of evolution in response to limited resources and environmental variability. The paper focuses on artificial neural networks and proposes a number of mechanisms operating at the genetic level, both those borrowed from the natural world and those designed by hand, the use of which may lead to network modularity and hopefully to an increase in their effectiveness. In addition, the influence of external factors on the shape of the networks, such as the variability of tasks and the conditions in which these tasks are performed, is also analyzed. The analysis is performed using the Hill Climb Assembler Encoding constructive neuro-evolutionary algorithm. The algorithm was extended with various module-oriented mechanisms and tested under various conditions. The aim of the tests was to investigate how individual mechanisms involved in the evolutionary process and factors external to this process affect modularity and efficiency of neural networks.

Devising novel performance measures for assessing the behavior of multilayer perceptrons trained on regression tasks

This methodological article is mainly aimed at establishing a bridge between classification and regression tasks, in a frame shaped by performance evaluation. More specifically, a general procedure for calculating performance measures is proposed, which can be applied to both classification and regression models. To this end, a notable change in the policy used to evaluate the confusion matrix is made, with the goal of reporting information about regression performance therein. This policy, called generalized token sharing, allows to a) assess models trained on both classification and regression tasks, b) evaluate the importance of input features, and c) inspect the behavior of multilayer perceptrons by looking at their hidden layers. The occurrence of success and failure patterns at the hidden layers of multilayer perceptrons trained and tested on selected regression problems, together with the effectiveness of layer-wise training, is also discussed.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

An Evolutionary Field Theorem: Evolutionary Field Optimization in Training of Power-Weighted Multiplicative Neurons for Nitrogen Oxides-Sensitive Electronic Nose Applications

Neuroevolutionary machine learning is an emerging topic in the evolutionary computation field and enables practical modeling solutions for data-driven engineering applications. Contributions of this study to the neuroevolutionary machine learning area are twofold: firstly, this study presents an evolutionary field theorem of search agents and suggests an algorithm for Evolutionary Field Optimization with Geometric Strategies (EFO-GS) on the basis of the evolutionary field theorem. The proposed EFO-GS algorithm benefits from a field-adapted differential crossover mechanism, a field-aware metamutation process to improve the evolutionary search quality. Secondly, the multiplicative neuron model is modified to develop Power-Weighted Multiplicative (PWM) neural models. The modified PWM neuron model involves the power-weighted multiplicative units similar to dendritic branches of biological neurons, and this neuron model can better represent polynomial nonlinearity and they can operate in the real-valued neuron mode, complex-valued neuron mode, and the mixed-mode. In this study, the EFO-GS algorithm is used for the training of the PWM neuron models to perform an efficient neuroevolutionary computation. Authors implement the proposed PWM neural processing with the EFO-GS in an electronic nose application to accurately estimate Nitrogen Oxides (NO_x) pollutant concentrations from low-cost multi-sensor array measurements and demonstrate improvements in estimation performance.

Empirical Study of Data-Driven Evolutionary Algorithms in Noisy Environments

For computationally intensive problems, data-driven evolutionary algorithms (DDEAs) are advantageous for low computational budgets because they build surrogate models based on historical data to approximate the expensive evaluation. Real-world optimization problems are highly susceptible to noisy data, but most of the existing DDEAs are developed and tested on ideal and clean environments; hence, their performance is uncertain in practice. In order to discover how DDEAs are affected by noisy data, this paper empirically studied the performance of DDEAs in different noisy environments. To fulfill the research purpose, we implemented four representative DDEAs and tested them on common benchmark problems with noise simulations in a systematic manner. Specifically, the simulation of noisy environments considered different levels of noise intensity and probability. The experimental analysis revealed the association relationships among noisy environments, benchmark problems and the performance of DDEAs. The analysis showed that noise will generally cause deterioration of the DDEA’s performance in most cases, but the effects could vary with different types of problem landscapes and different designs of DDEAs.

Activating Silent Synapses in Sulfurized Indium Selenide for Neuromorphic Computing

The transformation from silent to functional synapses is accompanied by the evolutionary process of human brain development and is essential to hardware implementation of the evolutionary artificial neural network but remains a challenge for mimicking silent to functional synapse activation. Here, we developed a simple approach to successfully realize activation of silent to functional synapses by controlled sulfurization of chemical vapor deposition-grown indium selenide crystals. The underlying mechanism is attributed to the migration of sulfur anions introduced by sulfurization. One of our most important findings is that the functional synaptic behaviors can be modulated by the degree of sulfurization and temperature. In addition, the essential synaptic behaviors including potentiation/depression, paired-pulse facilitation, and spike-rate-dependent plasticity are successfully implemented in the partially sulfurized functional synaptic device. The developed simple approach of introducing sulfur anions in layered selenide opens an effective new avenue to realize activation of silent synapses for application in evolutionary artificial neural networks.

A Sequential Handwriting Recognition Model Based on a Dynamically Configurable CRNN

Handwriting recognition refers to recognizing a handwritten input that includes character(s) or digit(s) based on an image. Because most applications of handwriting recognition in real life contain sequential text in various languages, there is a need to develop a dynamic handwriting recognition system. Inspired by the neuroevolutionary technique, this paper proposes a Dynamically Configurable Convolutional Recurrent Neural Network (DC-CRNN) for the handwriting recognition sequence modeling task. The proposed DC-CRNN is based on the Salp Swarm Optimization Algorithm (SSA), which generates the optimal structure and hyperparameters for Convolutional Recurrent Neural Networks (CRNNs). In addition, we investigate two types of encoding techniques used to translate the output of optimization to a CRNN recognizer. Finally, we proposed a novel hybridized SSA with Late Acceptance Hill-Climbing (LAHC) to improve the exploitation process. We conducted our experiments on two well-known datasets, IAM and IFN/ENIT, which include both the Arabic and English languages. The experimental results have shown that LAHC significantly improves the SSA search process. Therefore, the proposed DC-CRNN outperforms the handcrafted CRNN methods.

A Bilevel Learning Model and Algorithm for Self-Organizing Feed-Forward Neural Networks for Pattern Classification

Conventional artificial neural network (ANN) learning algorithms for classification tasks, either derivative-based optimization algorithms or derivative-free optimization algorithms work by training ANN first (or training and validating ANN) and then testing ANN, which are a two-stage and one-pass learning mechanism. Thus, this learning mechanism may not guarantee the generalization ability of a trained ANN. In this article, a novel bilevel learning model is constructed for self-organizing feed-forward neural network (FFNN), in which the training and testing processes are integrated into a unified framework. In this bilevel model, the upper level optimization problem is built for testing error on testing data set and network architecture based on network complexity, whereas the lower level optimization problem is constructed for network weights based on training error on training data set. For the bilevel framework, an interactive learning algorithm is proposed to optimize the architecture and weights of an FFNN with consideration of both training error and testing error. In this interactive learning algorithm, a hybrid binary particle swarm optimization (BPSO) taken as an upper level optimizer is used to self-organize network architecture, whereas the Levenberg-Marquardt (LM) algorithm as a lower level optimizer is utilized to optimize the connection weights of an FFNN. The bilevel learning model and algorithm have been tested on 20 benchmark classification problems. Experimental results demonstrate that the bilevel learning algorithm can significantly produce more compact FFNNs with more excellent generalization ability when compared with conventional learning algorithms.

Post-Processing of High Formwork Monitoring Data Based on the Back Propagation Neural Networks Model and the Autoregressive—Moving-Average Model

Many high formwork systems are currently equipped with health monitoring systems, and the analysis of the data obtained can determine whether high formwork is a hazard. Therefore, the post-processing of monitoring data has become an issue of widespread concern. In this paper, we discussed the fitting effect of the symmetrical high formwork monitoring data using the autoregressive–moving-average (ARMA) model and the back propagation neural networks (BPNN) combined model to process. In the actual project, the symmetry of the high formwork system allows the analysis of local monitoring results to be well extended to the whole. For the establishment of the ARMA model, the accurate judgment of the model order has a significant impact. In this paper, back propagation neural networks (BPNN) are used to simulate the ARMA process. The order of the ARMA model is estimated by determining the optimal neural network structure, which is suitable for linear or nonlinear sequences. We validated this approach from the ARMA model data simulated in Monte Carlo and compared it with the Akaike information criterion (AIC) and Bayesian information criterion (BIC). The length of the sequence, the coefficients and the order of the ARMA model are considered as factors that influence the judgment effect. Under different conditions, the BPNN always shows an accuracy rate of more than 90%, while the BIC only has a higher accuracy rate when the model order is low and the judgment efficiency of the AIC is below 50%. Finally, the proposed method successfully modeled the stress sequence and obtained the stress change trend. Compared with AIC and BIC, the efficiency of the processing time series is increased by about 50% when an order is obtained by BPNN.

Parsimonious neural networks learn interpretable physical laws

Machine learning is playing an increasing role in the physical sciences and significant progress has been made towards embedding domain knowledge into models. Less explored is its use to discover interpretable physical laws from data. We propose parsimonious neural networks (PNNs) that combine neural networks with evolutionary optimization to find models that balance accuracy with parsimony. The power and versatility of the approach is demonstrated by developing models for classical mechanics and to predict the melting temperature of materials from fundamental properties. In the first example, the resulting PNNs are easily interpretable as Newton's second law, expressed as a non-trivial time integrator that exhibits time-reversibility and conserves energy, where the parsimony is critical to extract underlying symmetries from the data. In the second case, the PNNs not only find the celebrated Lindemann melting law, but also new relationships that outperform it in the pareto sense of parsimony vs. accuracy.

Using machine learning methods to determine a typology of patients with HIV-HCV infection to be treated with antivirals

Several European countries have established criteria for prioritising initiation of treatment in patients infected with the hepatitis C virus (HCV) by grouping patients according to clinical characteristics. Based on neural network techniques, our objective was to identify those factors for HIV/HCV co-infected patients (to which clinicians have given careful consideration before treatment uptake) that have not being included among the prioritisation criteria. This study was based on the Spanish HERACLES cohort (NCT02511496) (April-September 2015, 2940 patients) and involved application of different neural network models with different basis functions (product-unit, sigmoid unit and radial basis function neural networks) for automatic classification of patients for treatment. An evolutionary algorithm was used to determine the architecture and estimate the coefficients of the model. This machine learning methodology found that radial basis neural networks provided a very simple model in terms of the number of patient characteristics to be considered by the classifier (in this case, six), returning a good overall classification accuracy of 0.767 and a minimum sensitivity (for the classification of the minority class, untreated patients) of 0.550. Finally, the area under the ROC curve was 0.802, which proved to be exceptional. The parsimony of the model makes it especially attractive, using just eight connections. The independent variable “recent PWID” is compulsory due to its importance. The simplicity of the model means that it is possible to analyse the relationship between patient characteristics and the probability of belonging to the treated group.

GA-BP in Thermal Fatigue Failure Prediction of Microelectronic Chips

A thermal fatigue life prediction model of microelectronic chips based on thermal fatigue tests and solder/substrate interfacial singularity analysis from finite element method (FEM) analysis is established in this paper. To save the calculation of interfacial singular parameters of new chips for life prediction, and improve the accuracy of prediction results in actual applications, a hybrid genetic algorithm–artificial neural network (GA–ANN) strategy is utilized. The proposed algorithm combines the local searching ability of the gradient-based back propagation (BP) strategy with the global searching ability of a genetic algorithm. A series of combinations of the dimensions and thermal mechanical properties of the solder and the corresponding singularity parameters at the failure interface are used to train the proposed GA-BP network. The results of the network, together with the established life prediction model, are used to predict the thermal fatigue lives of new chips. The comparison between the network results and thermal fatigue lives recorded in experiments shows that the GA-BP strategy is a successful prediction technique.

Reinforcement Learning-Based Differential Evolution With Cooperative Coevolution for a Compensatory Neuro-Fuzzy Controller

This paper presents the integration of reinforcement learning-based differential evolution (DE) with the cooperative coevolution (R-CCDE) method in a compensatory neuro-fuzzy controller (CNFC). The CNFC model employs compensatory fuzzy operations, which increase the adaptability and effectiveness of the controller. The R-CCDE method was used to determine an adequate control policy for nonlinear system problems. The evolution of a population involved the use of DE with cooperative coevolution to adjust CNFC parameters, and the fitness function of the R-CCDE method is used by a reinforcement signal to determine the controller that can be used to solve the control problem. This paper identified the best performing controller to solve nonlinear system problems. The simulation results of the proposed R-CCDE method were compared with those of various DE methods and the performance of the proposed R-CCDE method was superior to that of the other methods.

Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks

Biological neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifelong learning. The interplay of these elements leads to the emergence of biological intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) employ simulated evolution in-silico to breed plastic neural networks with the aim to autonomously design and create learning systems. EPANN experiments evolve networks that include both innate properties and the ability to change and learn in response to experiences in different environments and problem domains. EPANNs' aims include autonomously creating learning systems, bootstrapping learning from scratch, recovering performance in unseen conditions, testing the computational advantages of particular neural components, and deriving hypotheses on the emergence of biological learning. Thus, EPANNs may include a large variety of different neuron types and dynamics, network architectures, plasticity rules, and other factors. While EPANNs have seen considerable progress over the last two decades, current scientific and technological advances in artificial neural networks are setting the conditions for radically new approaches and results. Exploiting the increased availability of computational resources and of simulation environments, the often challenging task of hand-designing learning neural networks could be replaced by more autonomous and creative processes. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and possible developments are presented.

Optimization of Deep Neural Networks Using SoCs with OpenCL

In the optimization of deep neural networks (DNNs) via evolutionary algorithms (EAs) and the implementation of the training necessary for the creation of the objective function, there is often a trade-off between efficiency and flexibility. Pure software solutions implemented on general-purpose processors tend to be slow because they do not take advantage of the inherent parallelism of these devices, whereas hardware realizations based on heterogeneous platforms (combining central processing units (CPUs), graphics processing units (GPUs) and/or field-programmable gate arrays (FPGAs)) are designed based on different solutions using methodologies supported by different languages and using very different implementation criteria. This paper first presents a study that demonstrates the need for a heterogeneous (CPU-GPU-FPGA) platform to accelerate the optimization of artificial neural networks (ANNs) using genetic algorithms. Second, the paper presents implementations of the calculations related to the individuals evaluated in such an algorithm on different (CPU- and FPGA-based) platforms, but with the same source files written in OpenCL. The implementation of individuals on remote, low-cost FPGA systems on a chip (SoCs) is found to enable the achievement of good efficiency in terms of performance per watt.

Evolutionary Design of Convolutional Neural Networks for Human Activity Recognition in Sensor-Rich Environments

Human activity recognition is a challenging problem for context-aware systems and applications. It is gaining interest due to the ubiquity of different sensor sources, wearable smart objects, ambient sensors, etc. This task is usually approached as a supervised machine learning problem, where a label is to be predicted given some input data, such as the signals retrieved from different sensors. For tackling the human activity recognition problem in sensor network environments, in this paper we propose the use of deep learning (convolutional neural networks) to perform activity recognition using the publicly available OPPORTUNITY dataset. Instead of manually choosing a suitable topology, we will let an evolutionary algorithm design the optimal topology in order to maximize the classification F1 score. After that, we will also explore the performance of committees of the models resulting from the evolutionary process. Results analysis indicates that the proposed model was able to perform activity recognition within a heterogeneous sensor network environment, achieving very high accuracies when tested with new sensor data. Based on all conducted experiments, the proposed neuroevolutionary system has proved to be able to systematically find a classification model which is capable of outperforming previous results reported in the state-of-the-art, showing that this approach is useful and improves upon previously manually-designed architectures.

Spectrum-Diverse Neuroevolution With Unified Neural Models

Learning algorithms are being increasingly adopted in various applications. However, further expansion will require methods that work more automatically. To enable this level of automation, a more powerful solution representation is needed. However, by increasing the representation complexity, a second problem arises. The search space becomes huge, and therefore, an associated scalable and efficient searching algorithm is also required. To solve both the problems, first a powerful representation is proposed that unifies most of the neural networks features from the literature into one representation. Second, a new diversity preserving method called spectrum diversity is created based on the new concept of chromosome spectrum that creates a spectrum out of the characteristics and frequency of alleles in a chromosome. The combination of spectrum diversity with a unified neuron representation enables the algorithm to either surpass or equal NeuroEvolution of Augmenting Topologies on all of the five classes of problems tested. Ablation tests justify the good results, showing the importance of added new features in the unified neuron representation. Part of the success is attributed to the novelty-focused evolution and good scalability with a chromosome size provided by spectrum diversity. Thus, this paper sheds light on a new representation and diversity preserving mechanism that should impact algorithms and applications to come.

Evolutionary Design of Classifiers Made of Droplets Containing a Nonlinear Chemical Medium

Unconventional computing devices operating on nonlinear chemical media offer an interesting alternative to standard, semiconductor-based computers. In this work we study in-silico a chemical medium composed of communicating droplets that functions as a database classifier. The droplet network can be "programmed" by an externally provided illumination pattern. The complex relationship between the illumination pattern and the droplet behavior makes manual programming hard. We introduce an evolutionary algorithm that automatically finds the optimal illumination pattern for a given classification problem. Notably, our approach does not require us to prespecify the signals that represent the output classes of the classification problem, which is achieved by using a fitness function that measures the mutual information between chemical oscillation patterns and desired output classes. We illustrate the feasibility of our approach in computer simulations by evolving droplet classifiers for three machine learning datasets. We demonstrate that the same medium composed of 25 droplets located on a square lattice can be successfully used for different classification tasks by applying different illumination patterns as its externally supplied program.