A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks

A fluorescence probe-smartphone-machine learning integrated platform for the visual and intelligent detection of imidacloprid in agricultural products

Imidacloprid is a pesticide commonly used in agriculture production. Portable and accurate detection of imidacloprid residues is of great significance to food safety and human health. Herein, a red-emitting rare earth complex (Eu-IMDC) probe is prepared, which features low detection limit (75 nM), high selectivity and fast response speed (30 s) for imidacloprid detection. The detection mechanism is investigated through experiments and theoretical calculations. In addition, an intelligent detection platform integrating the fluorescence probe, smartphone and a feedforward neural network (FNN) model is constructed and applied to imidacloprid detection in real rice, millet, and ginger samples, achieving recovery rates range from 96.78 % to 104.77 % and relative standard deviation (RSD) values below than 3.83 %. Meanwhile, relevant parameters, such as the coefficient of determination (R2), root mean square error (RMSE) and residual prediction deviation (RPD) values, indicating excellent fitting and predictive performance of the FNN model. This work offers a rapid, portable, and intelligent sensing platform for pesticide residues in agricultural products.

A Fast Feedforward Small-World Neural Network for Nonlinear System Modeling

It is well-documented that cross-layer connections in feedforward small-world neural networks (FSWNNs) enhance the efficient transmission for gradients, thus improving its generalization ability with a fast learning. However, the merits of long-distance cross-layer connections are not fully utilized due to the random rewiring. In this study, aiming to further improve the learning efficiency, a fast FSWNN (FFSWNN) is proposed by taking into account the positive effects of long-distance cross-layer connections, and applied to nonlinear system modeling. First, a novel rewiring rule by giving priority to long-distance cross-layer connections is proposed to increase the gradient transmission efficiency when constructing FFSWNN. Second, an improved ridge regression method is put forward to determine the initial weights with high activation for the sigmoidal neurons in FFSWNN. Finally, to further improve the learning efficiency, an asynchronous learning algorithm is designed to train FFSWNN, with the weights connected to the output layer updated by the ridge regression method and other weights by the gradient descent method. Several experiments are conducted on four benchmark datasets from the University of California Irvine (UCI) machine learning repository and two datasets from real-life problems to evaluate the performance of FFSWNN on nonlinear system modeling. The results show that FFSWNN has significantly faster convergence speed and higher modeling accuracy than the comparative models, and the positive effects of the novel rewiring rule, the improved weight initialization, and the asynchronous learning algorithm on learning efficiency are demonstrated.

NGDE: A Niching-Based Gradient-Directed Evolution Algorithm for Nonconvex Optimization

Nonconvex optimization issues are prevalent in machine learning and data science. While gradient-based optimization algorithms can rapidly converge and are dimension-independent, they may, unfortunately, fall into local optimal solutions or saddle points. In contrast, evolutionary algorithms (EAs) gradually adapt the population of solutions to explore global optimal solutions. However, this approach requires substantial computational resources to perform numerous fitness function evaluations, which poses challenges for high-dimensional optimization in particular. This study introduces a novel nonconvex optimization algorithm, the niching-based gradient-directed evolution (NGDE) algorithm, designed specifically for high-dimensional nonconvex optimization. The NGDE algorithm generates potential solutions and divides them into multiple niches to explore distinct areas within the feasible region. Subsequently, each individual creates candidate offspring using the gradient-directed mutation operator we designed. The convergence properties of the NGDE algorithm are investigated in two scenarios: accessing the full gradient and approximating the gradient with mini-batch samples. The experimental studies demonstrate the superior performance of the NGDE algorithm in minimizing multimodal optimization functions. Additionally, when applied to train the neural networks of LeNet-5, NGDE shows significantly improved classification accuracy, especially in smaller training sizes.

Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective

High perceptual quality and low distortion degree are two important goals in image restoration tasks such as super-resolution (SR). Most of the existing SR methods aim to achieve these goals by minimizing the corresponding yet conflicting losses, such as the l1 loss and the adversarial loss. Unfortunately, the commonly used gradient-based optimizers, such as Adam, are hard to balance these objectives due to the opposite gradient decent directions of the contradictory losses. In this paper, we formulate the perception-distortion trade-off in SR as a multi-objective optimization problem and develop a new optimizer by integrating the gradient-free evolutionary algorithm (EA) with gradient-based Adam, where EA and Adam focus on the divergence and convergence of the optimization directions respectively. As a result, a population of optimal models with different perception-distortion preferences is obtained. We then design a fusion network to merge these models into a single stronger one for an effective perception-distortion trade-off. Experiments demonstrate that with the same backbone network, the perception-distortion balanced SR model trained by our method can achieve better perceptual quality than its competitors while attaining better reconstruction fidelity. Codes and models can be found at https://github.com/csslc/EA-Adam.

A framework of evolutionary optimized convolutional neural network for classification of shang and chow dynasties bronze decorative patterns

As a UNESCO World Cultural Heritage, the aesthetic value of bronze artifacts from the Shang and Chow Dynasties has had a profound influence on Chinese traditional culture and art. To facilitate the digital preservation and protection of these Shang and Chow bronze artifacts (SCB), it becomes imperative to categorize their decorative patterns. Therefore, a SCB pattern classification method of differential evolution called Shang and Chow Bronze Convolutional Neural Network (SCB-CNN) is proposed. Firstly, the original bronze decorative patterns of Shang and Chow dynasties are collected, and the samples are expanded through image augmentation technology to form a training dataset. Secondly, based on the classical convolutional neural network structure, the recognition and classification of bronze patterns are implemented by adjusting the network parameters. Then, the initial parameters of the convolutional neural network are optimized by differential evolution algorithm, and the optimized SCB-CNN is simulated. Finally, comparative experiments were conducted between the optimized SCB-CNN, the unoptimized model, VGG-Net, and GoogleNet. The experimental results indicate that the optimized SCB-CNN significantly reduces training time while maintaining fast prediction speed, convergence speed, and high accuracy. This study provides new insights for the inheritance and innovation research of SCB patterns.

Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications, and Open Issues

Over recent years, there has been a rapid development of deep learning (DL) in both industry and academia fields. However, finding the optimal hyperparameters of a DL model often needs high computational cost and human expertise. To mitigate the above issue, evolutionary computation (EC) as a powerful heuristic search approach has shown significant merits in the automated design of DL models, so-called evolutionary deep learning (EDL). This article aims to analyze EDL from the perspective of automated machine learning (AutoML). Specifically, we first illuminate EDL from DL and EC and regard EDL as an optimization problem. According to the DL pipeline, we systematically introduce EDL methods ranging from data preparation, model generation, to model deployment with a new taxonomy (i.e., what and how to evolve/optimize), and focus on the discussions of solution representation and search paradigm in handling the optimization problem by EC. Finally, key applications, open issues, and potentially promising lines of future research are suggested. This survey has reviewed recent developments of EDL and offers insightful guidelines for the development of EDL.