Comprehensive review of neural network-based prediction intervals and new advances

Deep Huber quantile regression networks

Typical machine learning regression applications aim to report the mean or the median of the predictive probability distribution, via training with a squared or an absolute error scoring function. The importance of issuing predictions of more functionals of the predictive probability distribution (quantiles and expectiles) has been recognized as a means to quantify the uncertainty of the prediction. In deep learning (DL) applications, that is possible through quantile and expectile regression neural networks (QRNN and ERNN respectively). Here we introduce deep Huber quantile regression networks (DHQRN) that nest QRNN and ERNN as edge cases. DHQRN can predict Huber quantiles, which are more general functionals in the sense that they nest quantiles and expectiles as limiting cases. The main idea is to train a DL algorithm with the Huber quantile scoring function, which is consistent for the Huber quantile functional. As a proof of concept, DHQRN are applied to predict house prices in Melbourne, Australia and Boston, United States (US). In this context, predictive performances of three DL architectures are discussed along with evidential interpretation of results from two economic case studies. Additional simulation experiments and applications to real-world case studies using open datasets demonstrate a satisfactory absolute performance of DHQRN.

Dual Accuracy-Quality-Driven Neural Network for Prediction Interval Generation

Accurate uncertainty quantification is necessary to enhance the reliability of deep learning (DL) models in real-world applications. In the case of regression tasks, prediction intervals (PIs) should be provided along with the deterministic predictions of DL models. Such PIs are useful or "high-quality (HQ)" as long as they are sufficiently narrow and capture most of the probability density. In this article, we present a method to learn PIs for regression-based neural networks (NNs) automatically in addition to the conventional target predictions. In particular, we train two companion NNs: one that uses one output, the target estimate, and another that uses two outputs, the upper and lower bounds of the corresponding PI. Our main contribution is the design of a novel loss function for the PI-generation network that takes into account the output of the target-estimation network and has two optimization objectives: minimizing the mean PI width and ensuring the PI integrity using constraints that maximize the PI probability coverage implicitly. Furthermore, we introduce a self-adaptive coefficient that balances both objectives within the loss function, which alleviates the task of fine-tuning. Experiments using a synthetic dataset, eight benchmark datasets, and a real-world crop yield prediction dataset showed that our method was able to maintain a nominal probability coverage and produce significantly narrower PIs without detriment to its target estimation accuracy when compared to those PIs generated by three state-of-the-art neural-network-based methods. In other words, our method was shown to produce higher quality PIs.

Energy-water management system based on robust predictive control for open-field cultivation

Food availability has been endangered by recent global events, where agriculture, the main food source for the global population, is expected to increase even more to fulfill the growing food demand. Along with food production, water and energy consumption are also increased, leading to over-extraction of groundwater and an excess emission of greenhouse gases due to fossil fuel consumption. In this context, a balance of these three resources is crucial; therefore, the water-energy-food nexus is considered to address the previous issues by designing an energy-water management system based on robust predictive control. This controller estimates the future worst-case scenario for multiple climatic conditions, such as solar radiation, ambient temperature, wind speed, precipitation, and groundwater recharge, to define an optimal irrigation volume, maximize crop growth, and minimize water consumption. At the same time, the controller schedules daily irrigation and groundwater extraction, considering energy availability from solar generation and storage, to fulfill the previously defined irrigation volume while minimizing operating costs. Climate prediction is done through fuzzy prediction intervals, whose lower or upper bound are used as worst-case to include climate uncertainty on the controller design. The energy-water management system is tested in different experiments, where results show that considering a robust approach ensures maximum crop development, avoids over-extraction of groundwater, and prioritizes renewable energy sources. This work proposes a robust energy-water management system designed to be sustainable. Considering the water-energy-food nexus, the system ensures food security and proper resource allocation, tackling global starvation, water availability, and energy access.

Uncertainty quantification in neural-network based pain intensity estimation

Improper pain management leads to severe physical or mental consequences, including suffering, a negative impact on quality of life, and an increased risk of opioid dependency. Assessing the presence and severity of pain is imperative to prevent such outcomes and determine the appropriate intervention. However, the evaluation of pain intensity is a challenging task because different individuals experience pain differently. To overcome this, many researchers in the field have employed machine learning models to evaluate pain intensity objectively using physiological signals. However, these efforts have primarily focused on pain point estimation, disregarding inherent uncertainty and variability in the data and model. A point estimate, which provides only partial information, is not sufficient for sound clinical decision-making. This study proposes a neural network-based method for objective pain interval estimation, and quantification of uncertainty. Our approach, which enables objective pain intensity estimation with desired confidence probabilities, affords clinicians a better understanding of a person's pain intensity. We explored three distinct algorithms: the bootstrap method, lower and upper bound estimation (LossL) optimized by genetic algorithm, and modified lower and upper bound estimation (LossS) optimized by gradient descent algorithm. Our empirical results demonstrate that LossS outperforms the other two by providing narrower prediction intervals. For 50%, 75%, 85%, and 95% prediction interval coverage probability, LossS provides average interval widths that are 22.4%, 7.9%, 16.7%, and 9.1% narrower than those of LossL, and 19.3%, 21.1%, 23.6%, and 26.9% narrower than those of bootstrap. As LossS outperforms, we assessed its performance in three different model-building approaches: (1) a generalized approach using a single model for the entire population, (2) a personalized approach with separate models for each individual, and (3) a hybrid approach with models for clusters of individuals. Results demonstrate that the hybrid model-building approach provides the best performance.

Development and Evaluation of Conformal Prediction Methods for Quantitative Structure-Activity Relationship

The quantitative structure-activity relationship (QSAR) regression model is a commonly used technique for predicting the biological activities of compounds using their molecular descriptors. Besides accurate activity estimation, obtaining a prediction uncertainty metric like a prediction interval is highly desirable. Quantifying prediction uncertainty is an active research area in statistical and machine learning (ML), but the implementation for QSAR remains challenging. However, most ML algorithms with high predictive performance require add-on companions for estimating the uncertainty of their prediction. Conformal prediction (CP) is a promising approach as its main components are agnostic to the prediction modes, and it produces valid prediction intervals under weak assumptions on the data distribution. We proposed computationally efficient CP algorithms tailored to the most widely used ML models, including random forests, deep neural networks, and gradient boosting. The algorithms use a novel approach to the derivation of nonconformity scores from the estimates of prediction uncertainty generated by the ensembles of point predictions. The validity and efficiency of proposed algorithms are demonstrated on a diverse collection of QSAR data sets as well as simulation studies. The provided software implementing our algorithms can be used as stand-alone or easily incorporated into other ML software packages for QSAR modeling.

Estimation of Prediction Intervals for Performance Assessment of Building Using Machine Learning

This study utilizes artificial neural networks (ANN) to estimate prediction intervals (PI) for seismic performance assessment of buildings subjected to long-term ground motion. To address the uncertainty quantification in structural health monitoring (SHM), the quality-driven lower upper bound estimation (QD-LUBE) has been opted for global probabilistic assessment of damage at local and global levels, unlike traditional methods. A distribution-free machine learning model has been adopted for enhanced reliability in quantifying uncertainty and ensuring robustness in post-earthquake probabilistic assessments and early warning systems. The distribution-free machine learning model is capable of quantifying uncertainty with high accuracy as compared to previous methods such as the bootstrap method, etc. This research demonstrates the efficacy of the QD-LUBE method in complex seismic risk assessment scenarios, thereby contributing significant enhancement in building resilience and disaster management strategies. This study also validates the findings through fragility curve analysis, offering comprehensive insights into structural damage assessment and mitigation strategies.

Confidence Interval Estimation for Cutting Tool Wear Prediction in Turning Using Bootstrap-Based Artificial Neural Networks

The degradation of the cutting tool and its optimal replacement is a major problem in machining given the variability in this degradation even under constant cutting conditions. Therefore, monitoring the degradation of cutting tools is an important part of the process in order to replace the tool at the optimal time and thus reduce operating costs. In this paper, a cutting tool degradation monitoring technique is proposed using bootstrap-based artificial neural networks. Different indicators from the turning operation are used as input to the approach: the RMS value of the cutting force and torque, the machining duration, and the total machined length. They are used by the approach to estimate the size of the flank wear (VB). Different neural networks are tested but the best results are achieved with an architecture containing two hidden layers: the first one containing six neurons with a Tanh activation function and the second one containing six neurons with an ReLu activation function. The novelty of the approach makes it possible, by using the bootstrap approach, to determine a confidence interval around the prediction. The results show that the networks are able to accurately track the degradation and detect the end of life of the cutting tools in a timely manner, but also that the confidence interval allows an estimate of the possible variation of the prediction to be made, thus helping in the decision for optimal tool replacement policies.

Carbon based sensors for air quality monitoring networks; middle east perspective

IoT-based Sensors networks play a pivotal role in improving air quality monitoring in the Middle East. They provide real-time data, enabling precise tracking of pollution trends, informed decision-making, and increased public awareness. Air quality and dust pollution in the Middle East region may leads to various health issues, particularly among vulnerable populations. IoT-based Sensors networks help mitigate health risks by offering timely and accurate air quality data. Air pollution affects not only human health but also the region's ecosystems and contributes to climate change. The economic implications of deteriorated air quality include healthcare costs and decreased productivity, underscore the need for effective monitoring and mitigation. IoT-based data can guide policymakers to align with Sustainable Development Goals (SDGs) related to health, clean water, and climate action. The conventional monitor based standard air quality instruments provide limited spatial coverage so there is strong need to continue research integrated with low-cost sensor technologies to make air quality monitoring more accessible, even in resource-constrained regions. IoT-based Sensors networks monitoring helps in understanding these environmental impacts. Among these IoT-based Sensors networks, sensors are of vital importance. With the evolution of sensors technologies, different types of sensors materials are available. Among this carbon based sensors are widely used for air quality monitoring. Carbon nanomaterial-based sensors (CNS) and carbon nanotubes (CNTs) as adsorbents exhibit unique capabilities in the measurement of air pollutants. These sensors are used to detect gaseous pollutants that includes oxides of nitrogen and Sulphur, and ozone, and volatile organic compounds (VOCs). This study provides comprehensive review of integration of carbon nanomaterials based sensors in IoT based network for better air quality monitoring and exploring the potential of machine learning and artificial intelligence for advanced data analysis, pollution source identification, integration of satellite and ground-based networks and future forecasting to design effective mitigation strategies. By prioritizing these recommendations, the Middle East and other regions, can further leverage IoT-based systems to improve air quality monitoring, safeguard public health, protect the environment, and contribute to sustainable development in the region.

Computationally Efficient Demographic History Inference from Allele Frequencies with Supervised Machine Learning

Inferring past demographic history of natural populations from genomic data is of central concern in many studies across research fields. Previously, our group had developed dadi, a widely used demographic history inference method based on the allele frequency spectrum (AFS) and maximum composite-likelihood optimization. However, dadi's optimization procedure can be computationally expensive. Here, we present donni (demography optimization via neural network inference), a new inference method based on dadi that is more efficient while maintaining comparable inference accuracy. For each dadi-supported demographic model, donni simulates the expected AFS for a range of model parameters then trains a set of Mean Variance Estimation neural networks using the simulated AFS. Trained networks can then be used to instantaneously infer the model parameters from future genomic data summarized by an AFS. We demonstrate that for many demographic models, donni can infer some parameters, such as population size changes, very well and other parameters, such as migration rates and times of demographic events, fairly well. Importantly, donni provides both parameter and confidence interval estimates from input AFS with accuracy comparable to parameters inferred by dadi's likelihood optimization while bypassing its long and computationally intensive evaluation process. donni's performance demonstrates that supervised machine learning algorithms may be a promising avenue for developing more sustainable and computationally efficient demographic history inference methods.

How to evaluate uncertainty estimates in machine learning for regression?

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Computationally efficient demographic history inference from allele frequencies with supervised machine learning

Inferring past demographic history of natural populations from genomic data is of central concern in many studies across research fields. Previously, our group had developed dadi, a widely used demographic history inference method based on the allele frequency spectrum (AFS) and maximum composite likelihood optimization. However, dadi's optimization procedure can be computationally expensive. Here, we developed donni (demography optimization via neural network inference), a new inference method based on dadi that is more efficient while maintaining comparable inference accuracy. For each dadi-supported demographic model, donni simulates the expected AFS for a range of model parameters then trains a set of Mean Variance Estimation neural networks using the simulated AFS. Trained networks can then be used to instantaneously infer the model parameters from future input data AFS. We demonstrated that for many demographic models, donni can infer some parameters, such as population size changes, very well and other parameters, such as migration rates and times of demographic events, fairly well. Importantly, donni provides both parameter and confidence interval estimates from input AFS with accuracy comparable to parameters inferred by dadi's likelihood optimization while bypassing its long and computationally intensive evaluation process. donni's performance demonstrates that supervised machine learning algorithms may be a promising avenue for developing more sustainable and computationally efficient demographic history inference methods.

Backtesting stochastic mortality models by prediction interval-based metrics

Human lifespan increments represent one of the main current risks for governments and pension and health benefits providers. Longevity societies imply financial sustainability challenges to guarantee adequate socioeconomic conditions for all individuals for a longer period. Consequently, modelling population dynamics and projecting future longevity scenarios are vital tasks for policymakers. As an answer, the demographic and the actuarial literature have been introduced and compared to several stochastic mortality models, although few studies have thoroughly tested the uncertainty concerning mortality projections. Forecasting mortality uncertainty levels have a central role since they reveal the potential, unexpected longevity rise and the related economic impact. Therefore, the present study poses a methodological framework to backtest uncertainty in mortality projections by exploiting uncertainty metrics not yet adopted in mortality literature. Using the data from the Human Mortality Database of the male and female populations of five countries, we present some numerical applications to illustrate how the proposed criterion works. The results show that there is no mortality model overperforming the others in all cases, and the best model choice depends on the data considered.

A real-time prediction interval correction method with an unscented Kalman filter for settlement monitoring of a power station dam

A prediction interval (PI) method is developed to quantify the model uncertainty of embankment settlement prediction. Traditional PIs are constructed based on specific past period information and remain unchanged; hence, they neglect discrepancies between previous calculations and new monitoring data. In this paper, a real-time prediction interval correction method is proposed. Time-varying PIs are built by continuously incorporating new measurements into model uncertainty calculations. The method consists of trend identification, PI construction, and real-time correction. Primarily, trend identification is carried out by wavelet analysis to eliminate early unstable noise and determine the settlement trend. Then, the Delta method is applied to construct PIs based on the characterized trend, and a comprehensive evaluation index is introduced. The model output and the upper and lower bounds of the PIs are updated by the unscented Kalman filter (UKF). The effect of the UKF is compared with that of the Kalman filter (KF) and extended Kalman filter (EKF). The method was demonstrated in the Qingyuan power station dam. The results show that the time-varying PIs based on trend data are smoother than those based on original data with better evaluation index scores. Also, the PIs are not affected by local anomalies. The proposed PIs are consistent with the actual measurements, and the UKF performs better than the KF and EKF. The approach has the potential to provide more reliable embankment safety assessments.

Method based on the support vector machine and information diffusion for prediction intervals of granary airtightness

Granaries should have good airtightness to reduce grain loss in storage. Prediction of granary airtightness at the design stage is beneficial in improving granary design. This paper proposes a method for the prediction interval (PI) of granary airtightness by using small sample data, which can guide designers with granary design. PI that the probability of the true target falling in it is markedly close or larger compared with the confidence level can be the decision basis of the granary design scheme. This study adopts support vector machine as the regression model trained by the airtightness data set of built granaries, and obtains the probability distribution of regression errors through information diffusion. The probability interval of errors is derived using a search algorithm, and PIs of granary airtightness can be acquired thereafter. Assessment indexes of PIs with confidence levels of 0.8 and 0.9 indicate that the proposed method can achieve confidence level and is superior to the comparative method using artificial neural network and bootstrap for PIs in cases of only a few samples. Thus, an innovative and feasible method is proposed for the computer-aided design of granary airtightness.

A novel hybrid wind speed interval prediction model based on mode decomposition and gated recursive neural network

Wind energy has become one of the most efficient renewable energy sources. However, the wind has the characteristics of intermittence and uncontrollability, so it is challenging to predict wind speed accurately. Considering the shortcomings of traditional wind power point predictions, a new hybrid model comprised three main modules used for data preprocessing, deterministic point prediction, and interval prediction is proposed to predict the wind speed interval. The first module, the data preprocessing module, uses variational mode decomposition (VMD), sample entropy (SE), and singular spectrum analysis (SSA) to extract the different frequency components of the initial wind speed series. The second module, the deterministic point prediction module, uses extreme learning machines (ELM), and a gated recursive unit (GRU) model to perform point prediction on the wind speed series. The third module, the interval prediction module, uses the nonparametric kernel density estimation method to construct the upper and lower bounds of the wind speed interval. In addition, the final wind speed prediction interval is obtained by integrating the prediction results of multiple interval prediction results to improve the robustness and generalization of the wind speed interval prediction. Finally, the effectiveness of the prediction performance of the proposed hybrid model is verified based on the data of two actual wind farms. The experimental results show that the proposed hybrid model can obtain the appropriate wind speed interval with high confidence and quality with different confidence levels of 95%, 90%, and 85%.

Wasserstein dropout

Despite of its importance for safe machine learning, uncertainty quantification for neural networks is far from being solved. State-of-the-art approaches to estimate neural uncertainties are often hybrid, combining parametric models with explicit or implicit (dropout-based) ensembling. We take another pathway and propose a novel approach to uncertainty quantification for regression tasks, Wasserstein dropout, that is purely non-parametric. Technically, it captures aleatoric uncertainty by means of dropout-based sub-network distributions. This is accomplished by a new objective which minimizes the Wasserstein distance between the label distribution and the model distribution. An extensive empirical analysis shows that Wasserstein dropout outperforms state-of-the-art methods, on vanilla test data as well as under distributional shift in terms of producing more accurate and stable uncertainty estimates.

Hierarchical-Bayesian-Based Sparse Stochastic Configuration Networks for Construction of Prediction Intervals

To address the architecture complexity and ill-posed problems of neural networks when dealing with high-dimensional data, this article presents a Bayesian-learning-based sparse stochastic configuration network (SCN) (BSSCN). The BSSCN inherits the basic idea of training an SCN in the Bayesian framework but replaces the common Gaussian distribution with a Laplace one as the prior distribution of the output weights of SCN. Meanwhile, a lower bound of the Laplace sparse prior distribution using a two-level hierarchical prior is adopted based on which an approximate Gaussian posterior with sparse property is obtained. It leads to the facilitation of training the BSSCN, and the analytical solution for output weights of BSSCN can be obtained. Furthermore, the hyperparameter estimation process is derived by maximizing the corresponding lower bound of the marginal likelihood function based on the expectation-maximization algorithm. In addition, considering the uncertainties caused by both noises in the real-world data and model mismatch, a bootstrap ensemble strategy using BSSCN is designed to construct the prediction intervals (PIs) of the target variables. The experimental results on three benchmark data sets and two real-world high-dimensional data sets demonstrate the effectiveness of the proposed method in terms of both prediction accuracy and quality of the constructed PIs.

Assessing the Impact of Features on Probabilistic Modeling of Photovoltaic Power Generation

Photovoltaic power generation has high variability and uncertainty because it is affected by uncertain factors such as weather conditions. Therefore, probabilistic forecasting is useful for optimal operation and risk hedging in power systems with large amounts of photovoltaic power generation. However, deterministic forecasting is the mainstay of photovoltaic generation forecasting; there are few studies on probabilistic forecasting and feature selection from weather or time-oriented features in such forecasting. In this study, prediction intervals were generated by the lower upper bound estimation (LUBE) using neural networks with two outputs to make probabilistic modeling for predictions. The objective was to improve prediction interval coverage probability (PICP), mean prediction interval width (MPIW), continuous ranked probability score (CRPS), and loss, which is the integration of PICP and MPIW, by removing unnecessary features through feature selection. When features with high gain were selected by random forest (RF), in the modeling of 14.7 kW PV systems, loss improved by 1.57 kW, CRPS by 0.03 kW, PICP by 0.057 kW, and MPIW by 0.12 kW on average over two weeks compared to the case where all features were used without feature selection. Therefore, the low gain features from RF act as noise and reduce the modeling accuracy.

A multi-scale time-series dataset with benchmark for machine learning in decarbonized energy grids

The electric grid is a key enabling infrastructure for the ambitious transition towards carbon neutrality as we grapple with climate change. With deepening penetration of renewable resources, the reliable operation of the electric grid becomes increasingly challenging. In this paper, we present PSML, a first-of-its-kind open-access multi-scale time-series dataset, to aid in the development of data-driven machine learning (ML)-based approaches towards reliable operation of future electric grids. The dataset is synthesized from a joint transmission and distribution electric grid to capture the increasingly important interactions and uncertainties of the grid dynamics, containing power, voltage and current measurements over multiple spatio-temporal scales. Using PSML, we provide state-of-the-art ML benchmarks on three challenging use cases of critical importance to achieve: (i) early detection, accurate classification and localization of dynamic disturbances; (ii) robust hierarchical forecasting of load and renewable energy; and (iii) realistic synthetic generation of physical-law-constrained measurements. We envision that this dataset will provide use-inspired ML research in safety-critical systems, while simultaneously enabling ML researchers to contribute towards decarbonization of energy sectors.

Evidential Extreme Learning Machine Algorithm-Based Day-Ahead Photovoltaic Power Forecasting

The gradually increased penetration of photovoltaic (PV) power into electric power systems brings an urgent requirement for accurate and stable PV power forecasting methods. The existing forecasting methods are built to explore the function between weather data and power generation, which ignore the uncertainty of historical PV power. To manage the uncertainty in the forecasting process, a novel ensemble method, named the evidential extreme learning machine (EELM) algorithm, for deterministic and probabilistic PV power forecasting based on the extreme learning machine (ELM) and evidential regression, is proposed in this paper. The proposed EELM algorithm builds ELM models for each neighbor in the k-nearest neighbors initially, and subsequently integrates multiple models through an evidential discounting and combination process. The results can be accessed through forecasting outcomes from corresponding models of nearest neighbors and the mass function determined by the distance between the predicted point and neighbors. The proposed EELM algorithm is verified with the real data series of a rooftop PV plant in Macau. The deterministic forecasting results demonstrate that the proposed EELM algorithm exhibits 15.45% lower nRMSE than ELM. In addition, the forecasting prediction intervals obtain better performance in PICP and CWC than normal distribution.

Deep Neural Networks for QSAR

Quantitative structure-activity relationship (QSAR) models are routinely applied computational tools in the drug discovery process. QSAR models are regression or classification models that predict the biological activities of molecules based on the features derived from their molecular structures. These models are usually used to prioritize a list of candidate molecules for future laboratory experiments and to help chemists gain better insights into how structural changes affect a molecule's biological activities. Developing accurate and interpretable QSAR models is therefore of the utmost importance in the drug discovery process. Deep neural networks, which are powerful supervised learning algorithms, have shown great promise for addressing regression and classification problems in various research fields, including the pharmaceutical industry. In this chapter, we briefly review the applications of deep neural networks in QSAR modeling and describe commonly used techniques to improve model performance.

Epistemic uncertainty quantification in deep learning classification by the Delta method

The Delta method is a classical procedure for quantifying epistemic uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters P. We propose a low cost approximation of the Delta method applicable to L₂-regularized deep neural networks based on the top K eigenpairs of the Fisher information matrix. We address efficient computation of full-rank approximate eigendecompositions in terms of the exact inverse Hessian, the inverse outer-products of gradients approximation and the so-called Sandwich estimator. Moreover, we provide bounds on the approximation error for the uncertainty of the predictive class probabilities. We show that when the smallest computed eigenvalue of the Fisher information matrix is near the L₂-regularization rate, the approximation error will be close to zero even when K≪P. A demonstration of the methodology is presented using a TensorFlow implementation, and we show that meaningful rankings of images based on predictive uncertainty can be obtained for two LeNet and ResNet-based neural networks using the MNIST and CIFAR-10 datasets. Further, we observe that false positives have on average a higher predictive epistemic uncertainty than true positives. This suggests that there is supplementing information in the uncertainty measure not captured by the classification alone.

Intra-hour irradiance forecasting techniques for solar power integration: a review

The ever-growing installation of solar power systems imposes severe challenges on the operations of local and regional power grids due to the inherent intermittency and variability of ground-level solar irradiance. In recent decades, solar forecasting methodologies for intra-hour, intra-day and day-ahead energy markets have been extensively explored as cost-effective technologies to mitigate the negative effects on the power grids caused by solar power instability. In this work, the progress in intra-hour solar forecasting methodologies are comprehensively reviewed and concisely summarized. The theories behind the forecasting methodologies and how these theories are applied in various forecasting models are presented. The reviewed mathematical tools include regressive methods, stochastic learning methods, deep learning methods, and genetic algorithm. The reviewed forecasting methodologies include data-driven methods, local-sensing methods, hybrid forecasting methods, and application orientated methods that generate probabilistic forecasts and spatial forecasts. Furthermore, suggestions to accelerate the development of future intra-hour forecasting methods are provided.

Hotspot Temperature Prediction of Dry-Type Transformers Based on Particle Filter Optimization with Support Vector Regression

Both poor cooling methods and complex heat dissipation lead to prominent asymmetry in transformer temperature distribution. Both the operating life and load capacity of a power transformer are closely related to the winding hotspot temperature. Realizing accurate prediction of the hotspot temperature of transformer windings is the key to effectively preventing thermal faults in transformers, thus ensuring the reliable operation of transformers and accurately predicting transformer operating lifetimes. In this paper, a hot spot temperature prediction method is proposed based on the transformer operating parameters through the particle filter optimization support vector regression model. Based on the monitored transformer temperature, load rate, transformer cooling type, and ambient temperature, the hotspot temperature of a dry-type transformer can be predicted by a support vector regression method. The hyperparameters of the support vector regression are dynamically optimized here according to the particle filter to improve the optimization accuracy. The validity and accuracy of the proposed method are verified by comparing the proposed method with a traditional support vector regression method based on the real operating data of a 35 kV dry-type transformer.

Bootstrapped Ensemble of Artificial Neural Networks Technique for Quantifying Uncertainty in Prediction of Wind Energy Production

The accurate prediction of wind energy production is crucial for an affordable and reliable power supply to consumers. Prediction models are used as decision-aid tools for electric grid operators to dynamically balance the energy production provided by a pool of diverse sources in the energy mix. However, different sources of uncertainty affect the predictions, providing the decision-makers with non-accurate and possibly misleading information for grid operation. In this regard, this work aims to quantify the possible sources of uncertainty that affect the predictions of wind energy production provided by an ensemble of Artificial Neural Network (ANN) models. The proposed Bootstrap (BS) technique for uncertainty quantification relies on estimating Prediction Intervals (PIs) for a predefined confidence level. The capability of the proposed BS technique is verified, considering a 34 MW wind plant located in Italy. The obtained results show that the BS technique provides a more satisfactory quantification of the uncertainty of wind energy predictions than that of a technique adopted by the wind plant owner and the Mean-Variance Estimation (MVE) technique of literature. The PIs obtained by the BS technique are also analyzed in terms of different weather conditions experienced by the wind plant and time horizons of prediction.

A Comparative Study of Machine Learning-Based Methods for Global Horizontal Irradiance Forecasting

The proliferation of photovoltaic (PV) power generation in power distribution grids induces increasing safety and service quality concerns for grid operators. The inherent variability, essentially due to meteorological conditions, of PV power generation affects the power grid reliability. In order to develop efficient monitoring and control schemes for distribution grids, reliable forecasting of the solar resource at several time horizons that are related to regulation, scheduling, dispatching, and unit commitment, is necessary. PV power generation forecasting can result from forecasting global horizontal irradiance (GHI), which is the total amount of shortwave radiation received from above by a surface horizontal to the ground. A comparative study of machine learning methods is given in this paper, with a focus on the most widely used: Gaussian process regression (GPR), support vector regression (SVR), and artificial neural networks (ANN). Two years of GHI data with a time step of 10 min are used to train the models and forecast GHI at varying time horizons, ranging from 10 min to 4 h. Persistence on the clear-sky index, also known as scaled persistence model, is included in this paper as a reference model. Three criteria are used for in-depth performance estimation: normalized root mean square error (nRMSE), dynamic mean absolute error (DMAE) and coverage width-based criterion (CWC). Results confirm that machine learning-based methods outperform the scaled persistence model. The best-performing machine learning-based methods included in this comparative study are the long short-term memory (LSTM) neural network and the GPR model using a rational quadratic kernel with automatic relevance determination.

About Model Validation in Bioprocessing

In bioprocess engineering the Qualtiy by Design (QbD) initiative encourages the use of models to define design spaces. However, clear guidelines on how models for QbD are validated are still missing. In this review we provide a comprehensive overview of the validation methods, mathematical approaches, and metrics currently applied in bioprocess modeling. The methods cover analytics for data used for modeling, model training and selection, measures for predictiveness, and model uncertainties. We point out the general issues in model validation and calibration for different types of models and put this into the context of existing health authority recommendations. This review provides a starting point for developing a guide for model validation approaches. There is no one-fits-all approach, but this review should help to identify the best fitting validation method, or combination of methods, for the specific task and the type of bioprocess model that is being developed.

Predicting Remaining Useful Life of Rolling Bearings Based on Reliable Degradation Indicator and Temporal Convolution Network with the Quantile Regression

High precision and multi information prediction results of bearing remaining useful life (RUL) can effectively describe the uncertainty of bearing health state and operation state. Aiming at the problem of feature efficient extraction and RUL prediction during rolling bearings operation degradation process, through data reduction and key features mining analysis, a new feature vector based on time-frequency domain joint feature is found to describe the bearings degradation process more comprehensively. In order to keep the effective information without increasing the scale of neural network, a joint feature compression calculation method based on redefined degradation indicator (DI) was proposed to determine the input data set. By combining the temporal convolution network with the quantile regression (TCNQR) algorithm, the probability density forecasting at any time is achieved based on kernel density estimation (KDE) for the conditional distribution of predicted values. The experimental results show that the proposed method can obtain the point prediction results with smaller errors. Compared with the existing quantile regression of long short-term memory network(LSTMQR), the proposed method can construct more accurate prediction interval and probability density curve, which can effectively quantify the uncertainty of bearing running state.

Towards a mathematical framework to inform neural network modelling via polynomial regression

Even when neural networks are widely used in a large number of applications, they are still considered as black boxes and present some difficulties for dimensioning or evaluating their prediction error. This has led to an increasing interest in the overlapping area between neural networks and more traditional statistical methods, which can help overcome those problems. In this article, a mathematical framework relating neural networks and polynomial regression is explored by building an explicit expression for the coefficients of a polynomial regression from the weights of a given neural network, using a Taylor expansion approach. This is achieved for single hidden layer neural networks in regression problems. The validity of the proposed method depends on different factors like the distribution of the synaptic potentials or the chosen activation function. The performance of this method is empirically tested via simulation of synthetic data generated from polynomials to train neural networks with different structures and hyperparameters, showing that almost identical predictions can be obtained when certain conditions are met. Lastly, when learning from polynomial generated data, the proposed method produces polynomials that approximate correctly the data locally.

Prediction Interval Estimation Methods for Artificial Neural Network (ANN)-Based Modeling of the Hydro-Climatic Processes, a Review

Despite the wide applications of artificial neural networks (ANNs) in modeling hydro-climatic processes, quantification of the ANNs’ performance is a significant matter. Sustainable management of water resources requires information about the amount of uncertainty involved in the modeling results, which is a guide for proper decision making. Therefore, in recent years, uncertainty analysis of ANN modeling has attracted noticeable attention. Prediction intervals (PIs) are one of the prevalent tools for uncertainty quantification. This review paper has focused on the different techniques of PI development in the field of hydrology and climatology modeling. The implementation of each method was discussed, and their pros and cons were investigated. In addition, some suggestions are provided for future studies. This review paper was prepared via PRISMA (preferred reporting items for systematic reviews and meta-analyses) methodology.

Long-Term Mechanical Ventilation in Neonates: A 10-Year Overview and Predictive Model

Objectives: Significant resources are devoted to neonatal prolonged mechanical ventilation (NPMV), but little is known about the outcomes in those children. Our primary objective was to describe the NPMV respiratory, digestive, and neurological outcomes at 18 months corrected age. Our second objective was on the early identification of which patients, among the NPMV cohort, will need to be ventilated for ≥125 days, which corresponded to the 75th percentile in the preliminary data, and to describe that subgroup. Methods: In this retrospective cohort study, we included all children born between 2004 and 2013 who had a NPMV (≥21 days of invasive or noninvasive respiratory support reached between 40 and 44 weeks of postconceptional age). We used random forests, logistic regression with penalization, naive Bayes, and XGBoost to predict which patients will need ≥125 days of ventilation. We used a Monte Carlo cross validation. Results: We included 164 patients. Of which, 40% (n = 66) were female, and the median gestational age was 29 weeks [interquartile range (IQR): 26-36 weeks] with a bimodal distribution. Median ventilation days were 104 (IQR: 66-139 days). The most frequently associated diagnoses were pulmonary hypertension (43%), early pulmonary dysplasia (41%), and lobar emphysema (37%). At 18 months corrected age, 29% (n = 47) had died, 59% (n = 97) were free of any respiratory support, and 45% (n = 74) were exclusively orally fed. A moderate area under the ROC curve of 0.65 (95% CI: 0.54-0.72) for identifying patients in need of ≥125 days of ventilation at inclusion was achieved by random forests classifiers. Among the 26 measured at inclusion, the most contributive ones were PCO₂, inspired O₂ concentration, and gestational age. At 18 months corrected age, patients ventilated for ≥125 days had a lower respiratory weaning success (76 vs. 87%, P = 0.05), lower exclusive oral feeding proportion (51 vs. 84%, P < 0.001), and a higher neurological impairment (median Pediatric Cerebral Performance Category score 3 vs. 2, P = 0.008) than patients ventilated for < 125 days. Conclusion: NPMV is a severe condition with a high risk of mortality, neurological impairment, and oral feed delay at 18 months. Most survivors are weaned of any respiratory support. We identified the risk factors that allow for the early identification of the most at-risk children of long-term ventilation with a moderate discrimination.

Addressing model uncertainty in probabilistic forecasting using Monte Carlo dropout

In recent years, deep learning models have been developed to address probabilistic forecasting tasks, assuming an implicit stochastic process that relates past observed values to uncertain future values. These models are capable of capturing the inherent uncertainty of the underlying process, but they ignore the model uncertainty that comes from the fact of not having infinite data. This work proposes addressing the model uncertainty problem using Monte Carlo dropout, a variational approach that assigns distributions to the weights of a neural network instead of simply using fixed values. This allows to easily adapt common deep learning models currently in use to produce better probabilistic forecasting estimates, in terms of their consideration of uncertainty. The proposal is validated for prediction intervals estimation on seven energy time series, using a popular probabilistic model called Mean Variance Estimation (MVE), as the deep model adapted using the technique.

Ensemble Stochastic Configuration Networks for Estimating Prediction Intervals: A Simultaneous Robust Training Algorithm and Its Application

Obtaining accurate point prediction of industrial processes' key variables is challenging due to the outliers and noise that are common in industrial data. Hence the prediction intervals (PIs) have been widely adopted to quantify the uncertainty related to the point prediction. In order to improve the prediction accuracy and quantify the level of uncertainty associated with the point prediction, this article estimates the PIs by using ensemble stochastic configuration networks (SCNs) and bootstrap method. The estimated PIs can guarantee both the modeling stability and computational efficiency. To encourage the cooperation among the base SCNs and improve the robustness of the ensemble SCNs when the training data are contaminated with noise and outliers, a simultaneous robust training method of the ensemble SCNs is developed based on the Bayesian ridge regression and M-estimate. Moreover, the hyperparameters of the assumed distributions over noise and output weights of the ensemble SCNs are estimated by the expectation-maximization (EM) algorithm, which can result in the optimal PIs and better prediction accuracy. Finally, the performance of the proposed approach is evaluated on three benchmark data sets and a real-world data set collected from a refinery. The experimental results demonstrate that the proposed approach exhibits better performance in terms of the quality of PIs, prediction accuracy, and robustness.

Beyond Expectation: Deep Joint Mean and Quantile Regression for Spatiotemporal Problems

Spatiotemporal problems are ubiquitous and of vital importance in many research fields. Despite the potential already demonstrated by deep learning methods in modeling spatiotemporal data, typical approaches tend to focus solely on conditional expectations of the output variables being modeled. In this article, we propose a multioutput multiquantile deep learning approach for jointly modeling several conditional quantiles together with the conditional expectation as a way to provide a more complete "picture" of the predictive density in spatiotemporal problems. Using two large-scale data sets from the transportation domain, we empirically demonstrate that, by approaching the quantile regression problem from a multitask learning perspective, it is possible to solve the embarrassing quantile crossings problem while simultaneously significantly outperforming state-of-the-art quantile regression methods. Moreover, we show that jointly modeling the mean and several conditional quantiles not only provides a rich description about the predictive density that can capture heteroscedastic properties at a neglectable computational overhead but also leads to improved predictions of the conditional expectation due to the extra information and the regularization effect induced by the added quantiles.

A Survey of Computational Intelligence Techniques for Wind Power Uncertainty Quantification in Smart Grids

The high penetration level of renewable energy is thought to be one of the basic characteristics of future smart grids. Wind power, as one of the most increasing renewable energy, has brought a large number of uncertainties into the power systems. These uncertainties would require system operators to change their traditional ways of decision-making. This article provides a comprehensive survey of computational intelligence techniques for wind power uncertainty quantification in smart grids. First, prediction intervals (PIs) are introduced as a means to quantify the uncertainties in wind power forecasts. Various PI evaluation indices, including the latest trends in comprehensive evaluation techniques, are compared. Furthermore, computational intelligence-based PI construction methods are summarized and classified into traditional methods (parametric) and direct PI construction methods (nonparametric). In the second part of this article, methods of incorporating wind power forecast uncertainties into power system decision-making processes are investigated. Three techniques, namely, stochastic models, fuzzy logic models, and robust optimization, and different power system applications using these techniques are reviewed. Finally, future research directions, such as spatiotemporal and hierarchical forecasting, deep learning-based methods, and integration of predictive uncertainty estimates into the decision-making process, are discussed. This survey can benefit the readers by providing a complete technical summary of wind power uncertainty quantification and decision-making in smart grids.

Ensemble deep learning with multi-objective optimization for prognosis of rotating machinery

With the emerging of Internet of Things and smart sensing techniques, enormous monitoring data has been collected by prognostics and health management (PHM) systems. Predicting the Remaining useful life (RUL) of mechanical components from monitoring data has always been a challenging task in many industries, yet determining RUL accurately is identified as one of the most demanded outcomes of PHM systems. In this study, an ensemble deep learning with multi-objective optimization (EDL-MO) method is proposed for RUL prediction. A novel ensemble deep learning algorithm for RUL prediction is designed by combining accuracy and diversity. By introducing the diversity, uncorrelated error is produced in each individual iteration, and performance of prediction will be improved by evolving deep networks. The presented EDL-MO employs evolutionary optimization to optimize the two conflicting objectives, that is, diversity and accuracy. To validate the proposed algorithm, bearing run-to-failure experiments were carried out under constant load. The vibration signals are recorded and utilized to predict the RUL by using the proposed EDL-MO method, as well as other existing methods for performance comparison. The effectiveness and superiority of EDL-MO are analyzed, which outperforms the current algorithms in predicting RUL on rotation machineries.