Deep Adaptive Input Normalization for Time Series Forecasting

Enhanced framework embedded with data transformation and multi-objective feature selection algorithm for forecasting wind power

The increasing global interest in utilizing wind turbines for power generation emphasizes the importance of accurate wind power forecasting in managing wind power. This paper proposed a framework that integrates a data transformation mechanism with a multi-objective none-dominated sorting genetic algorithm III (NSGA-III), coupled with a hybrid deep Recurrent Network (DRN) and Long Short-Term Memory (LSTM) architecture for modeling wind power. The feature selection algorithm, multi-objective NSGA-III, identifies the optimal subset features from wind energy datasets. These selected features undergo a data transformation process before being input into the hybrid DRN-LSTM for wind power forecasting. A comparative study demonstrates the proposal's superior effectiveness and robustness compared to existing frameworks with the proposal achieving 2.6593e-10 and 1.630e-05 in terms of MSE and RMSE respectively whereas the classical algorithm recorded 8.8814e-07 and 9.424e-04. The study's contributions lie in its approach integration of data transformation mechanism and the notable enhancements in wind power forecasting accuracy. Furthermore, the study offers valuable insights to guide research efforts in the future.

Distributional Drift Adaptation With Temporal Conditional Variational Autoencoder for Multivariate Time Series Forecasting

Due to the nonstationary nature, the distribution of real-world multivariate time series (MTS) changes over time, which is known as distribution drift. Most existing MTS forecasting models greatly suffer from distribution drift and degrade the forecasting performance over time. Existing methods address distribution drift via adapting to the latest arrived data or self-correcting per the meta knowledge derived from future data. Despite their great success in MTS forecasting, these methods hardly capture the intrinsic distribution changes, especially from a distributional perspective. Accordingly, we propose a novel framework temporal conditional variational autoencoder (TCVAE) to model the dynamic distributional dependencies over time between historical observations and future data in MTSs and infer the dependencies as a temporal conditional distribution to leverage latent variables. Specifically, a novel temporal Hawkes attention (THA) mechanism represents temporal factors that subsequently fed into feedforward networks to estimate the prior Gaussian distribution of latent variables. The representation of temporal factors further dynamically adjusts the structures of Transformer-based encoder and decoder to distribution changes by leveraging a gated attention mechanism (GAM). Moreover, we introduce conditional continuous normalization flow (CCNF) to transform the prior Gaussian to a complex and form-free distribution to facilitate flexible inference of the temporal conditional distribution. Extensive experiments conducted on six real-world MTS datasets demonstrate the TCVAE's superior robustness and effectiveness over the state-of-the-art MTS forecasting baselines. We further illustrate the TCVAE applicability through multifaceted case studies and visualization in real-world scenarios.

Multivariate Time Series Forecasting Using Multiscale Recurrent Networks With Scale Attention and Cross-Scale Guidance

Multivariate time series (MTS) forecasting is considered as a challenging task due to complex and nonlinear interdependencies between time steps and series. With the advance of deep learning, significant efforts have been made to model long-term and short-term temporal patterns hidden in historical information by recurrent neural networks (RNNs) with a temporal attention mechanism. Although various forecasting models have been developed, most of them are single-scale oriented, resulting in scale information loss. In this article, we seamlessly integrate multiscale analysis into deep learning frameworks to build scale-aware recurrent networks and propose two multiscale recurrent network (MRN) models for MTS forecasting. The first model called MRN-SA adopts a scale attention mechanism to dynamically select the most relevant information from different scales and simultaneously employs input attention and temporal attention to make predictions. The second one named as MRN-CSG introduces a novel cross-scale guidance mechanism to exploit the information from coarse scale to guide the decoding process at fine scale, which results in a lightweight and more easily trained model without obvious loss of accuracy. Extensive experimental results demonstrate that both MRN-SA and MRN-CSG can achieve state-of-the-art performance on five typical MTS datasets in different domains. The source codes will be publicly available at https://github.com/qguo2010/MRN.

Federated Learning for Data Trading Portfolio Allocation With Autonomous Economic Agents

In the rapidly advancing ubiquitous intelligence society, the role of data as a valuable resource has become paramount. As a result, there is a growing need for the development of autonomous economic agents (AEAs) capable of intelligently and autonomously trading data. These AEAs are responsible for acquiring, processing, and selling data to entities such as software companies. To ensure optimal profitability, an intelligent AEA must carefully allocate its portfolio, relying on accurate return estimation and well-designed models. However, a significant challenge arises due to the sensitive and confidential nature of data trading. Each AEA possesses only limited local information, which may not be sufficient for training a robust and effective portfolio allocation model. To address this limitation, we propose a novel data trading market where AEAs exclusively possess local market information. To overcome the information constraint, AEAs employ federated learning (FL) that allows multiple AEAs to jointly train a model capable of generating promising portfolio allocations for multiple data products. To account for the dynamic and ever-changing revenue returns, we introduce an integration of the histogram of oriented gradients (HoGs) with the discrete wavelet transformation (DWT). This innovative combination serves to redefine the representation of local market information to effectively handle the inherent nonstationarity of revenue patterns associated with data products. Furthermore, we leverage the transform domain of local model drifts in the global model update process, effectively reducing the communication burden and significantly improving training efficiency. Through simulations, we provide compelling evidence that our proposed schemes deliver superior performance across multiple evaluation metrics, including test loss, cumulative return, portfolio risk, and Sharpe ratio.

Machine learning-based real-time anomaly detection using data pre-processing in the telemetry of server farms

Fast and accurate anomaly detection is critical in telemetry systems because it helps operators take appropriate actions in response to abnormal behaviours. However, recent techniques are accurate but not fast enough to deal with real-time data. There is a need to reduce the anomaly detection time, which motivates us to propose two new algorithms called AnDePeD (Anomaly Detector on Periodic Data) and AnDePed Pro. The novelty of the proposed algorithms lies in exploiting the periodic nature of data in anomaly detection. Our proposed algorithms apply a variational mode decomposition technique to find and extract periodic components from the original data before using Long Short-Term Memory neural networks to detect anomalies in the remainder time series. Furthermore, our methods include advanced techniques to eliminate prediction errors and automatically tune operational parameters. Extensive numerical results show that the proposed algorithms achieve comparable performance in terms of Precision, Recall, F-score, and MCC metrics while outperforming most of the state-of-the-art anomaly detection approaches in terms of initialisation delay and detection delay, which is favourable for practical applications.

TCDformer: A transformer framework for non-stationary time series forecasting based on trend and change-point detection

Although time series prediction models based on Transformer architecture have achieved significant advances, concerns have arisen regarding their performance with non-stationary real-world data. Traditional methods often use stabilization techniques to boost predictability, but this often results in the loss of non-stationarity, notably underperforming when tackling major events in practical applications. To address this challenge, this research introduces an innovative method named TCDformer (Trend and Change-point Detection Transformer). TCDformer employs a unique strategy, initially encoding abrupt changes in non-stationary time series using the local linear scaling approximation (LLSA) module. The reconstructed contextual time series is then decomposed into trend and seasonal components. The final prediction results are derived from the additive combination of a multilayer perceptron (MLP) for predicting trend components and wavelet attention mechanisms for seasonal components. Comprehensive experimental results show that on standard time series prediction datasets, TCDformer significantly surpasses existing benchmark models in terms of performance, reducing MSE by 47.36% and MAE by 31.12%. This approach offers an effective framework for managing non-stationary time series, achieving a balance between performance and interpretability, making it especially suitable for addressing non-stationarity challenges in real-world scenarios.

A filter-augmented auto-encoder with learnable normalization for robust multivariate time series anomaly detection

While existing reconstruction-based multivariate time series (MTS) anomaly detection methods demonstrate advanced performance on many challenging real-world datasets, they generally assume the data only consists of normal samples when training models. However, real-world MTS data may contain significant noise and even be contaminated by anomalies. As a result, most existing approaches easily capture the pattern of the contaminated data, making identifying anomalies more difficult. Although a few studies have aimed to mitigate the interference of the noise and anomalies by introducing various regularizations, they still employ the objective of fully reconstructing the input data, impeding the model from learning an accurate profile of the MTS's normal pattern. Moreover, it is difficult for existing methods to apply the most appropriate normalization schemes for each dataset in various complex scenarios, particularly for mixed-feature MTS. This paper proposes a filter-augmented auto-encoder with learnable normalization (NormFAAE) for robust MTS anomaly detection. Firstly, NormFAAE designs a deep hybrid normalization module. It is trained with the backbone end-to-end in the current training task to perform the optimal normalization scheme. Meanwhile, it integrates two learnable normalization sub-modules to deal with the mixed-feature MTS effectively. Secondly, NormFAAE proposes a filter-augmented auto-encoder with a dual-phase task. It separates the noise and anomalies from the input data by a deep filter module, which facilitates the model to only reconstruct the normal data, achieving a more robust latent representation of MTS. Experimental results demonstrate that NormFAAE outperforms 17 typical baselines on five real-world industrial datasets from diverse fields.

Explainable Artificial Intelligence (XAI) in Pain Research: Understanding the Role of Electrodermal Activity for Automated Pain Recognition

Artificial intelligence and especially deep learning methods have achieved outstanding results for various applications in the past few years. Pain recognition is one of them, as various models have been proposed to replace the previous gold standard with an automated and objective assessment. While the accuracy of such models could be increased incrementally, the understandability and transparency of these systems have not been the main focus of the research community thus far. Thus, in this work, several outcomes and insights of explainable artificial intelligence applied to the electrodermal activity sensor data of the PainMonit and BioVid Heat Pain Database are presented. For this purpose, the importance of hand-crafted features is evaluated using recursive feature elimination based on impurity scores in Random Forest (RF) models. Additionally, Gradient-weighted class activation mapping is applied to highlight the most impactful features learned by deep learning models. Our studies highlight the following insights: (1) Very simple hand-crafted features can yield comparative performances to deep learning models for pain recognition, especially when properly selected with recursive feature elimination. Thus, the use of complex neural networks should be questioned in pain recognition, especially considering their computational costs; and (2) both traditional feature engineering and deep feature learning approaches rely on simple characteristics of the input time-series data to make their decision in the context of automated pain recognition.

A Hybrid Residual Dilated LSTM and Exponential Smoothing Model for Midterm Electric Load Forecasting

This work presents a hybrid and hierarchical deep learning model for midterm load forecasting. The model combines exponential smoothing (ETS), advanced long short-term memory (LSTM), and ensembling. ETS extracts dynamically the main components of each individual time series and enables the model to learn their representation. Multilayer LSTM is equipped with dilated recurrent skip connections and a spatial shortcut path from lower layers to allow the model to better capture long-term seasonal relationships and ensure more efficient training. A common learning procedure for LSTM and ETS, with a penalized pinball loss, leads to simultaneous optimization of data representation and forecasting performance. In addition, ensembling at three levels ensures a powerful regularization. A simulation study performed on the monthly electricity demand time series for 35 European countries confirmed the high performance of the proposed model and its competitiveness with classical models such as ARIMA and ETS as well as state-of-the-art models based on machine learning.

Scalable Room Occupancy Prediction with Deep Transfer Learning Using Indoor Climate Sensor

An important instrument for achieving smart and high-performance buildings is Machine Learning (ML). A lot of research has been done in exploring the ML models for various applications in the built environment such as occupancy prediction. Nevertheless, the research focused mostly on analyzing the feasibility and performance of different supervised ML models but has rarely focused on practical applications and the scalability of those models. In this study, a transfer learning method is proposed as a solution to typical problems in the practical application of ML in buildings. Such problems are scaling a model to a different building, collecting ground truth data necessary for training the supervised model, and assuring the model is robust when conditions change. The practical application examined in this work is a deep learning model used for predicting room occupancy using indoor climate IoT sensors. This work proved that it is possible to significantly reduce the length of ground truth data collection to only two days. The robustness of the transferred model was tested as well, where performance stayed on a similar level if a suitable normalization technique was used. In addition, the proposed methodology was tested with room occupancy level prediction, showing slightly lower performance. Finally, the importance of understanding the performance metrics is crucial for market adoption of ML-based solutions in the built environment. Therefore, in this study, additional analysis was done by presenting the occupancy prediction model performance in understandable ways from the practical perspective.