Ensemble Stochastic Configuration Networks for Estimating Prediction Intervals: A Simultaneous Robust Training Algorithm and Its Application

Dual Accuracy-Quality-Driven Neural Network for Prediction Interval Generation

Accurate uncertainty quantification is necessary to enhance the reliability of deep learning (DL) models in real-world applications. In the case of regression tasks, prediction intervals (PIs) should be provided along with the deterministic predictions of DL models. Such PIs are useful or "high-quality (HQ)" as long as they are sufficiently narrow and capture most of the probability density. In this article, we present a method to learn PIs for regression-based neural networks (NNs) automatically in addition to the conventional target predictions. In particular, we train two companion NNs: one that uses one output, the target estimate, and another that uses two outputs, the upper and lower bounds of the corresponding PI. Our main contribution is the design of a novel loss function for the PI-generation network that takes into account the output of the target-estimation network and has two optimization objectives: minimizing the mean PI width and ensuring the PI integrity using constraints that maximize the PI probability coverage implicitly. Furthermore, we introduce a self-adaptive coefficient that balances both objectives within the loss function, which alleviates the task of fine-tuning. Experiments using a synthetic dataset, eight benchmark datasets, and a real-world crop yield prediction dataset showed that our method was able to maintain a nominal probability coverage and produce significantly narrower PIs without detriment to its target estimation accuracy when compared to those PIs generated by three state-of-the-art neural-network-based methods. In other words, our method was shown to produce higher quality PIs.

Transfer Dynamic Latent Variable Modeling for Quality Prediction of Multimode Processes

Quality prediction is beneficial to intelligent inspection, advanced process control, operation optimization, and product quality improvements of complex industrial processes. Most of the existing work obeys the assumption that training samples and testing samples follow similar data distributions. The assumption is, however, not true for practical multimode processes with dynamics. In practice, traditional approaches mostly establish a prediction model using the samples from the principal operating mode (POM) with abundant samples. The model is inapplicable to other modes with a few samples. In view of this, this article will propose a novel dynamic latent variable (DLV)-based transfer learning approach, called transfer DLV regression (TDLVR), for quality prediction of multimode processes with dynamics. The proposed TDLVR can not only derive the dynamics between process variables and quality variables in the POM but also extract the co-dynamic variations among process variables between the POM and the new mode. This can effectively overcome data marginal distribution discrepancy and enrich the information of the new mode. To make full use of the available labeled samples from the new mode, an error compensation mechanism is incorporated into the established TDLVR, termed compensated TDLVR (CTDLVR), to adapt to the conditional distribution discrepancy. Empirical studies show the efficacy of the proposed TDLVR and CTDLVR methods in several case studies, including numerical simulation examples and two real-industrial process examples.

Reliable prediction intervals with directly optimized inductive conformal regression for deep learning

By generating prediction intervals (PIs) to quantify the uncertainty of each prediction in deep learning regression, the risk of wrong predictions can be effectively controlled. High-quality PIs need to be as narrow as possible, whilst covering a preset proportion of real labels. At present, many approaches to improve the quality of PIs can effectively reduce the width of PIs, but they do not ensure that enough real labels are captured. Inductive Conformal Predictor (ICP) is an algorithm that can generate effective PIs which is theoretically guaranteed to cover a preset proportion of data. However, typically ICP is not directly optimized to yield minimal PI width. In this study, we propose Directly Optimized Inductive Conformal Regression (DOICR) for neural networks that takes only the average width of PIs as the loss function and increases the quality of PIs through an optimized scheme, under the validity condition that sufficient real labels are captured in the PIs. Benchmark experiments show that DOICR outperforms current state-of-the-art algorithms for regression problems using underlying Deep Neural Network structures for both tabular and image data.

A Multi-Agent Prediction Method for Data Sampling and Transmission Reduction in Internet of Things Sensor Networks

Sensor networks can provide valuable real-time data for various IoT applications. However, the amount of sensed and transmitted data should be kept at a low level due to the limitations imposed by network bandwidth, data storage, processing capabilities, and finite energy resources. In this paper, a new method is introduced that uses the predicted intervals of possible sensor readings to efficiently suppress unnecessary transmissions and decrease the amount of data samples collected by a sensor node. In the proposed method, the intervals of possible sensor readings are determined with a multi-agent system, where each agent independently explores a historical dataset and evaluates the similarity between past and current sensor readings to make predictions. Based on the predicted intervals, it is determined whether the real sensed data can be useful for a given IoT application and when the next data sample should be transmitted. The prediction algorithm is executed by the IoT gateway or in the cloud. The presented method is applicable to IoT sensor networks that utilize low-end devices with limited processing power, memory, and energy resources. During the experiments, the advantages of the introduced method were demonstrated by considering the criteria of prediction interval width, coverage probability, and transmission reduction. The experimental results confirm that the introduced method improves the accuracy of prediction intervals and achieves a higher rate of transmission reduction compared with state-of-the-art prediction methods.

A regularized stochastic configuration network based on weighted mean of vectors for regression

The stochastic configuration network (SCN) randomly configures the input weights and biases of hidden layers under a set of inequality constraints to guarantee its universal approximation property. The SCN has demonstrated great potential for fast and efficient data modeling. However, the prediction accuracy and convergence rate of SCN are frequently impacted by the parameter settings of the model. The weighted mean of vectors (INFO) is an innovative swarm intelligence optimization algorithm, with an optimization procedure consisting of three phases: updating rule, vector combining, and a local search. This article aimed at establishing a new regularized SCN based on the weighted mean of vectors (RSCN-INFO) to optimize its parameter selection and network structure. The regularization term that combines the ridge method with the residual error feedback was introduced into the objective function in order to dynamically adjust the training parameters. Meanwhile, INFO was employed to automatically explore an appropriate four-dimensional parameter vector for RSCN. The selected parameters may lead to a compact network architecture with a faster reduction of the network residual error. Simulation results over some benchmark datasets demonstrated that the proposed RSCN-INFO showed superior performance with respect to parameter setting, fast convergence, and network compactness compared with other contrast algorithms.

Hierarchical-Bayesian-Based Sparse Stochastic Configuration Networks for Construction of Prediction Intervals

To address the architecture complexity and ill-posed problems of neural networks when dealing with high-dimensional data, this article presents a Bayesian-learning-based sparse stochastic configuration network (SCN) (BSSCN). The BSSCN inherits the basic idea of training an SCN in the Bayesian framework but replaces the common Gaussian distribution with a Laplace one as the prior distribution of the output weights of SCN. Meanwhile, a lower bound of the Laplace sparse prior distribution using a two-level hierarchical prior is adopted based on which an approximate Gaussian posterior with sparse property is obtained. It leads to the facilitation of training the BSSCN, and the analytical solution for output weights of BSSCN can be obtained. Furthermore, the hyperparameter estimation process is derived by maximizing the corresponding lower bound of the marginal likelihood function based on the expectation-maximization algorithm. In addition, considering the uncertainties caused by both noises in the real-world data and model mismatch, a bootstrap ensemble strategy using BSSCN is designed to construct the prediction intervals (PIs) of the target variables. The experimental results on three benchmark data sets and two real-world high-dimensional data sets demonstrate the effectiveness of the proposed method in terms of both prediction accuracy and quality of the constructed PIs.

Estimating Evapotranspiration of Pomegranate Trees Using Stochastic Configuration Networks (SCN) and UAV Multispectral Imagery

Evapotranspiration (ET) estimation is important in precision agriculture water management, such as evaluating soil moisture, drought monitoring, and assessing crop water stress. As a traditional method, evapotranspiration estimation using crop coefficient (Kc) has been commonly used. Since there are strong similarities between the Kc curve and the vegetation index curve, the crop coefficient Kc is usually estimated as a function of the vegetation index. Researchers have developed linear regression models for the Kc and the normalized difference vegetation index (NDVI), usually derived from satellite imagery. However, the spatial resolution of the satellite image is often insufficient for crops with clumped canopy structures, such as vines and trees. Therefore, in this article, the authors used Unmanned Aerial Vehicles (UAVs) to collect high-resolution multispectral imagery in a pomegranate orchard located at the USDA-ARS, San Joaquin Valley Agricultural Sciences Center, Parlier, CA. The Kc values were measured from a weighing lysimeter and the NDVI values were derived from UAV imagery. Then, the authors established a relationship between the NDVI and Kc by using a linear regression model and a stochastic configuration networks (SCN) model, respectively. Based on the research results, the linear regression model has an R2 of 0.975 and RMSE of 0.05. The SCN regression model has an R2 and RMSE value of 0.995 and 0.046, respectively. Compared with the linear regression model, the SCN model improved performance in predicting Kc from NDVI. Then, actual evapotranspiration was estimated and compared with lysimeter data in an experimental pomegranate orchard. The UAV imagery provided a spatial and tree-by-tree view of ET distribution.

A Lightweight Learning Method for Stochastic Configuration Networks Using Non-Inverse Solution

Stochastic configuration networks (SCNs) face time-consuming issues when dealing with complex modeling tasks that usually require a mass of hidden nodes to build an enormous network. An important reason behind this issue is that SCNs always employ the Moore–Penrose generalized inverse method with high complexity to update the output weights in each increment. To tackle this problem, this paper proposes a lightweight SCNs, called L-SCNs. First, to avoid using the Moore–Penrose generalized inverse method, a positive definite equation is proposed to replace the over-determined equation, and the consistency of their solution is proved. Then, to reduce the complexity of calculating the output weight, a low complexity method based on Cholesky decomposition is proposed. The experimental results based on both the benchmark function approximation and real-world problems including regression and classification applications show that L-SCNs are sufficiently lightweight.

Prediction Interval Estimation Methods for Artificial Neural Network (ANN)-Based Modeling of the Hydro-Climatic Processes, a Review

Despite the wide applications of artificial neural networks (ANNs) in modeling hydro-climatic processes, quantification of the ANNs’ performance is a significant matter. Sustainable management of water resources requires information about the amount of uncertainty involved in the modeling results, which is a guide for proper decision making. Therefore, in recent years, uncertainty analysis of ANN modeling has attracted noticeable attention. Prediction intervals (PIs) are one of the prevalent tools for uncertainty quantification. This review paper has focused on the different techniques of PI development in the field of hydrology and climatology modeling. The implementation of each method was discussed, and their pros and cons were investigated. In addition, some suggestions are provided for future studies. This review paper was prepared via PRISMA (preferred reporting items for systematic reviews and meta-analyses) methodology.

Optimal Randomness for Stochastic Configuration Network (SCN) with Heavy-Tailed Distributions

Stochastic Configuration Network (SCN) has a powerful capability for regression and classification analysis. Traditionally, it is quite challenging to correctly determine an appropriate architecture for a neural network so that the trained model can achieve excellent performance for both learning and generalization. Compared with the known randomized learning algorithms for single hidden layer feed-forward neural networks, such as Randomized Radial Basis Function (RBF) Networks and Random Vector Functional-link (RVFL), the SCN randomly assigns the input weights and biases of the hidden nodes in a supervisory mechanism. Since the parameters in the hidden layers are randomly generated in uniform distribution, hypothetically, there is optimal randomness. Heavy-tailed distribution has shown optimal randomness in an unknown environment for finding some targets. Therefore, in this research, the authors used heavy-tailed distributions to randomly initialize weights and biases to see if the new SCN models can achieve better performance than the original SCN. Heavy-tailed distributions, such as Lévy distribution, Cauchy distribution, and Weibull distribution, have been used. Since some mixed distributions show heavy-tailed properties, the mixed Gaussian and Laplace distributions were also studied in this research work. Experimental results showed improved performance for SCN with heavy-tailed distributions. For the regression model, SCN-Lévy, SCN-Mixture, SCN-Cauchy, and SCN-Weibull used less hidden nodes to achieve similar performance with SCN. For the classification model, SCN-Mixture, SCN-Lévy, and SCN-Cauchy have higher test accuracy of 91.5%, 91.7% and 92.4%, respectively. Both are higher than the test accuracy of the original SCN.