Time series forecasting for nonlinear and non-stationary processes: a review and comparative study

Frequency-specific dual-attention based adversarial network for blood oxygen level-dependent time series prediction

Functional magnetic resonance imaging (fMRI) is currently one of the most popular technologies for measuring brain activity in both research and clinical contexts. However, clinical constraints often result in short fMRI scan durations, limiting the diagnostic performance for brain disorders. To address this limitation, we developed an end-to-end frequency-specific dual-attention-based adversarial network (FDAA-Net) to extend the time series of existing blood oxygen level-dependent (BOLD) data, enhancing their diagnostic utility. Our approach leverages the frequency-dependent nature of fMRI signals using variational mode decomposition (VMD), which adaptively tracks brain activity across different frequency bands. We integrated the generative adversarial network (GAN) with a spatial-temporal attention mechanism to fully capture relationships among spatially distributed brain regions and temporally continuous time windows. We also introduced a novel loss function to estimate the upward and downward trends of each frequency component. We validated FDAA-Net on the Human Connectome Project (HCP) database by comparing the original and predicted time series of brain regions in the default mode network (DMN), a key network activated during rest. FDAA-Net effectively overcame linear frequency-specific challenges and outperformed other popular prediction models. Test-retest reliability experiments demonstrated high consistency between the functional connectivity of predicted outcomes and targets. Furthermore, we examined the clinical applicability of FDAA-Net using short-term fMRI data from individuals with autism spectrum disorder (ASD) and major depressive disorder (MDD). The model achieved a maximum predicted sequence length of 40% of the original scan durations. The prolonged time series improved diagnostic performance by 8.0% for ASD and 11.3% for MDD compared with the original sequences. These findings highlight the potential of fMRI time series prediction to enhance diagnostic power of brain disorders in short fMRI scans.

A Bayesian non-stationary heteroskedastic time series model for multivariate critical care data

We propose a multivariate GARCH model for non-stationary health time series by modifying the observation-level variance of the standard state space model. The proposed model provides an intuitive and novel way of dealing with heteroskedastic data using the conditional nature of state-space models. We follow the Bayesian paradigm to perform the inference procedure. In particular, we use Markov chain Monte Carlo methods to obtain samples from the resultant posterior distribution. We use the forward filtering backward sampling algorithm to efficiently obtain samples from the posterior distribution of the latent state. The proposed model also handles missing data in a fully Bayesian fashion. We validate our model on synthetic data and analyze a data set obtained from an intensive care unit in a Montreal hospital and the MIMIC dataset. We further show that our proposed models offer better performance, in terms of WAIC than standard state space models. The proposed model provides a new way to model multivariate heteroskedastic non-stationary time series data. Model comparison can then be easily performed using the WAIC.

A comparison between machine and deep learning models on high stationarity data

Advances in sensor, computing, and communication technologies are enabling big data analytics by providing time series data. However, conventional models struggle to identify sequence features and forecast accuracy. This paper investigates time series features and shows that some machine learning algorithms can outperform deep learning models. In particular, the problem analyzed concerned predicting the number of vehicles passing through an Italian tollbooth in 2021. The dataset, composed of 8766 rows and 6 columns relating to additional tollbooths, proved to have high stationarity and was treated through machine learning methods such as support vector machine, random forest, and eXtreme gradient boosting (XGBoost), as well as deep learning through recurrent neural networks with long short-term memory (RNN-LSTM) cells. From the comparison of these models, the prediction through the XGBoost algorithm outperforms competing algorithms, particularly in terms of MAE and MSE. The result highlights how a shallower algorithm than a neural network is, in this case, able to obtain a better adaptation to the time series instead of a much deeper model that tends to develop a smoother prediction.

A TCN-Linear Hybrid Model for Chaotic Time Series Forecasting

The applications of deep learning and artificial intelligence have permeated daily life, with time series prediction emerging as a focal area of research due to its significance in data analysis. The evolution of deep learning methods for time series prediction has progressed from the Convolutional Neural Network (CNN) and the Recurrent Neural Network (RNN) to the recently popularized Transformer network. However, each of these methods has encountered specific issues. Recent studies have questioned the effectiveness of the self-attention mechanism in Transformers for time series prediction, prompting a reevaluation of approaches to LTSF (Long Time Series Forecasting) problems. To circumvent the limitations present in current models, this paper introduces a novel hybrid network, Temporal Convolutional Network-Linear (TCN-Linear), which leverages the temporal prediction capabilities of the Temporal Convolutional Network (TCN) to enhance the capacity of LSTF-Linear. Time series from three classical chaotic systems (Lorenz, Mackey-Glass, and Rossler) and real-world stock data serve as experimental datasets. Numerical simulation results indicate that, compared to classical networks and novel hybrid models, our model achieves the lowest RMSE, MAE, and MSE with the fewest training parameters, and its R2 value is the closest to 1.

Forecasting with an N-dimensional Langevin equation and a neural-ordinary differential equation

Accurate prediction of electricity day-ahead prices is essential in competitive electricity markets. Although stationary electricity-price forecasting techniques have received considerable attention, research on non-stationary methods is comparatively scarce, despite the common prevalence of non-stationary features in electricity markets. Specifically, existing non-stationary techniques will often aim to address individual non-stationary features in isolation, leaving aside the exploration of concurrent multiple non-stationary effects. Our overarching objective here is the formulation of a framework to systematically model and forecast non-stationary electricity-price time series, encompassing the broader scope of non-stationary behavior. For this purpose, we develop a data-driven model that combines an N-dimensional Langevin equation (LE) with a neural-ordinary differential equation (NODE). The LE captures fine-grained details of the electricity-price behavior in stationary regimes but is inadequate for non-stationary conditions. To overcome this inherent limitation, we adopt a NODE approach to learn, and at the same time predict, the difference between the actual electricity-price time series and the simulated price trajectories generated by the LE. By learning this difference, the NODE reconstructs the non-stationary components of the time series that the LE is not able to capture. We exemplify the effectiveness of our framework using the Spanish electricity day-ahead market as a prototypical case study. Our findings reveal that the NODE nicely complements the LE, providing a comprehensive strategy to tackle both stationary and non-stationary electricity-price behavior. The framework's dependability and robustness is demonstrated through different non-stationary scenarios by comparing it against a range of basic naïve methods.

Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics

We address a three-tier numerical framework based on nonlinear manifold learning for the forecasting of high-dimensional time series, relaxing the "curse of dimensionality" related to the training phase of surrogate/machine learning models. At the first step, we embed the high-dimensional time series into a reduced low-dimensional space using nonlinear manifold learning (local linear embedding and parsimonious diffusion maps). Then, we construct reduced-order surrogate models on the manifold (here, for our illustrations, we used multivariate autoregressive and Gaussian process regression models) to forecast the embedded dynamics. Finally, we solve the pre-image problem, thus lifting the embedded time series back to the original high-dimensional space using radial basis function interpolation and geometric harmonics. The proposed numerical data-driven scheme can also be applied as a reduced-order model procedure for the numerical solution/propagation of the (transient) dynamics of partial differential equations (PDEs). We assess the performance of the proposed scheme via three different families of problems: (a) the forecasting of synthetic time series generated by three simplistic linear and weakly nonlinear stochastic models resembling electroencephalography signals, (b) the prediction/propagation of the solution profiles of a linear parabolic PDE and the Brusselator model (a set of two nonlinear parabolic PDEs), and (c) the forecasting of a real-world data set containing daily time series of ten key foreign exchange rates spanning the time period 3 September 2001-29 October 2020.

Bayesian auxiliary variable model for birth records data with qualitative and quantitative responses

Many applications involve data with qualitative and quantitative responses. When there is an association between the two responses, a joint model will produce improved results than fitting them separately. In this paper, a Bayesian method is proposed to jointly model such data. The joint model links the qualitative and quantitative responses and can assess their dependency strength via a latent variable. The posterior distributions of parameters are obtained through an efficient MCMC sampling algorithm. The simulation is conducted to show that the proposed method improves the prediction capacity for both responses. Further, the proposed joint model is applied to the birth records data acquired by the Virginia Department of Health for studying the mutual dependence between preterm birth of infants and their birth weights.

Optimal forecasting accuracy using Lp-norm combination

A well-known result in statistics is that a linear combination of two-point forecasts has a smaller Mean Square Error (MSE) than the two competing forecasts themselves (Bates and Granger in J Oper Res Soc 20(4):451-468, 1969). The only case in which no improvements are possible is when one of the single forecasts is already the optimal one in terms of MSE. The kinds of combination methods are various, ranging from the simple average (SA) to more robust methods such as the one based on median or Trimmed Average (TA) or Least Absolute Deviations or optimization techniques (Stock and Watson in J Forecast 23(6):405-430, 2004). Standard regression-based combination approaches may fail to get a realistic result if the forecasts show high collinearity in several situations or the data distribution is not Gaussian. Therefore, we propose a forecast combination method based on Lp-norm estimators. These estimators are based on the Generalized Error Distribution, which is a generalization of the Gaussian distribution, and they can be used to solve the cases of multicollinearity and non-Gaussianity. In order to demonstrate the potential of Lp-norms, we conducted a simulated and an empirical study, comparing its performance with other standard-regression combination approaches. We carried out the simulation study with different values of the autoregressive parameter, by alternating heteroskedasticity and homoskedasticity. On the other hand, the real data application is based on the daily Bitfinex historical series of bitcoins (2014-2020) and the 25 historical series relating to companies included in the Dow Jonson, were subsequently considered. We showed that, by combining different GARCH and the ARIMA models, assuming both Gaussian and non-Gaussian distributions, the Lp-norm scheme improves the forecasting accuracy with respect to other regression-based combination procedures.

ECG Signal Modeling Using Volatility Properties: Its Application in Sleep Apnea Syndrome

This study presents and evaluates the mathematical model to estimate the mean and variance of single-lead ECG signals in sleep apnea syndrome. Our objective is to use the volatility property of the ECG signal for modeling. ECG signal is a stochastic signal whose mean and variance are time-varying. So, we propose to decompose this nonstationarity into two additive components; a homoscedastic Autoregressive Integrated Moving Average (ARIMA) and a heteroscedastic time series in terms of Exponential Generalized Autoregressive Conditional Heteroskedasticity (EGARCH), where the former captures the linearity property and the latter the nonlinear characteristics of the ECG signal. First, ECG signals are segmented into one-minute segments. The heteroskedasticity property is then examined through various tests such as the ARCH/GARCH test, kurtosis, skewness, and histograms. Next, the ARIMA model is applied to signals as a linear model and EGARCH as a nonlinear model. The appropriate orders of models are estimated by using the Bayesian Information Criterion (BIC). We assess the effectiveness of our model in terms of mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE). The data in this article is obtained from the Physionet Apnea-ECG database. Results show that the ARIMA-EGARCH model performs better than other models for modeling both apneic and normal ECG signals in sleep apnea syndrome.

Brain Connectivity Studies on Structure-Function Relationships: A Short Survey with an Emphasis on Machine Learning

This short survey reviews the recent literature on the relationship between the brain structure and its functional dynamics. Imaging techniques such as diffusion tensor imaging (DTI) make it possible to reconstruct axonal fiber tracks and describe the structural connectivity (SC) between brain regions. By measuring fluctuations in neuronal activity, functional magnetic resonance imaging (fMRI) provides insights into the dynamics within this structural network. One key for a better understanding of brain mechanisms is to investigate how these fast dynamics emerge on a relatively stable structural backbone. So far, computational simulations and methods from graph theory have been mainly used for modeling this relationship. Machine learning techniques have already been established in neuroimaging for identifying functionally independent brain networks and classifying pathological brain states. This survey focuses on methods from machine learning, which contribute to our understanding of functional interactions between brain regions and their relation to the underlying anatomical substrate.

Performance Analysis Using a Deep Belief Network by a Self-Organizing Map

Since resource consumption is the main reason for software aging occurrences, many methods have been applied to accurately predict the resource consumption series. Among these methods, neural networks are powerfully applied to forecast the series data. For some existing problems of artificial neural networks such as the choice of initialization and local optimization, the improvements of neural networks are not only a hot research topic in the field of time series prediction but also a research hotspot in resource consumption prediction of software aging. In this paper, we propose a method for resource consumption prediction of software aging using deep belief nets (DBNs) with the restricted Boltzmann machine (RBM). This presented method contains the following steps. First, a pre-processing is introduced by two parts: smoothing data by a self-organizing map (SOM) and removing a linear trend by a difference method. Second, a method, DBN with two RBMs, is presented to capture the features and forecast future values. Third, a glowworm swarm optimization (GSO) method is used to learn the hyper-parameters of DBN with two RBMs. In the experiments, two types of resource consumption series are used to validate our proposed method compared with some state-of-the-art algorithms.

On unified framework for discrete-time grey models: Extensions and applications

Grey theory-based time series models are widely used in various fields and disciplines. While most of the research is focused on the development and improvement of novel discrete-time models, very limited attention has been paid to the relationships among diverse models. The current paper proposes a methodological and practical framework to unify the single-variable, multi-variable, and multi-output discrete-time grey models, making it easier for practitioners to select an appropriate model for a given time-series forecasting problem. The recursive extrapolation strategy is present to generate multi-step ahead forecasts and four model families are deduced within the universal framework. Large-scale simulation studies are conducted to evaluate the finite-sample fitting and multi-step ahead forecasting performance. The proposed approach is illustrated using application examples from the indirect measurement of the tensile strength of materials.

Intrinsic recurrence quantification analysis of nonlinear and nonstationary short-term time series

Recurrence analysis is a powerful tool to appraise the nonlinear dynamics of complex systems and delineate the inherent laminar, divergent, or transient behaviors. Oftentimes, the effectiveness of recurrence quantification hinges upon the accurate reconstruction of the state space from a univariate time series with a uniform sampling rate. Few, if any, existing approaches quantify the recurrence properties from a short-term time series, particularly those sampled at a non-uniform rate, which are fairly ubiquitous in studies of rare or extreme events. This paper presents a novel intrinsic recurrence quantification analysis to portray the recurrence behaviors in complex dynamical systems with only short-term observations. As opposed to the traditional recurrence analysis, the proposed approach represents recurrence dynamics of a short-term time series in an intrinsic state space formed by proper rotations, attained from intrinsic time-scale decomposition (ITD) of the short time series. It is shown that intrinsic recurrence quantification analysis (iRQA), patterns harnessed from the corresponding recurrence plot, captures the underlying nonlinear and nonstationary dynamics of those short time series. In addition, as ITD does not require uniform sampling of the time series, iRQA is also applicable to unevenly spaced temporal data. Our findings are corroborated in two case studies: change detection in the Lorenz time series and early-stage identification of atrial fibrillation using short-term electrocardiogram signals.

A Dynamic Systems Approach for Detecting and Localizing of Infarct-Related Artery in Acute Myocardial Infarction Using Compressed Paper-Based Electrocardiogram (ECG)

Timely evaluation and reperfusion have improved the myocardial salvage and the subsequent recovery rate of the patients hospitalized with acute myocardial infarction (MI). Long waiting time and time-consuming procedures of in-hospital diagnostic testing severely affect the timeliness. We present a Poincare pattern ensemble-based method with the consideration of multi-correlated non-stationary stochastic system dynamics to localize the infarct-related artery (IRA) in acute MI by fully harnessing information from paper-based Electrocardiogram (ECG). The vectorcardiogram (VCG) diagnostic features extracted from only 2.5-s long paper ECG recordings were used to hierarchically localize the IRA-not mere localization of the infarcted cardiac tissues-in acute MI. Paper ECG records and angiograms of 106 acute MI patients collected at the Heart Artery and Vein Center at Fresno California and the 12-lead ECG signals from the Physionet PTB online database were employed to validate the proposed approach. We reported the overall accuracies of 97.41% for healthy control (HC) vs. MI, 89.41 ± 9.89 for left and right culprit arteries vs. others, 88.2 ± 11.6 for left main arteries vs. right-coronary-ascending (RCA) and 93.67 ± 4.89 for left-anterior-descending (LAD) vs. left-circumflex (LCX). The IRA localization from paper ECG can be used to timely triage the patients with acute coronary syndromes to the percutaneous coronary intervention facilities.

Machine Learning for Quantitative Finance Applications: A Survey

The analysis of financial data represents a challenge that researchers had to deal with. The rethinking of the basis of financial markets has led to an urgent demand for developing innovative models to understand financial assets. In the past few decades, researchers have proposed several systems based on traditional approaches, such as autoregressive integrated moving average (ARIMA) and the exponential smoothing model, in order to devise an accurate data representation. Despite their efficacy, the existing works face some drawbacks due to poor performance when managing a large amount of data with intrinsic complexity, high dimensionality and casual dynamicity. Furthermore, these approaches are not suitable for understanding hidden relationships (dependencies) between data. This paper proposes a review of some of the most significant works providing an exhaustive overview of recent machine learning (ML) techniques in the field of quantitative finance showing that these methods outperform traditional approaches. Finally, the paper also presents comparative studies about the effectiveness of several ML-based systems.

Machine learning and statistical models for predicting indoor air quality

Indoor air quality (IAQ), as determined by the concentrations of indoor air pollutants, can be predicted using either physically based mechanistic models or statistical models that are driven by measured data. In comparison with mechanistic models mostly used in unoccupied or scenario-based environments, statistical models have great potential to explore IAQ captured in large measurement campaigns or in real occupied environments. The present study carried out the first literature review of the use of statistical models to predict IAQ. The most commonly used statistical modeling methods were reviewed and their strengths and weaknesses discussed. Thirty-seven publications, in which statistical models were applied to predict IAQ, were identified. These studies were all published in the past decade, indicating the emergence of the awareness and application of machine learning and statistical modeling in the field of IAQ. The concentrations of indoor particulate matter (PM_2.5 and PM₁₀ ) were the most frequently studied parameters, followed by carbon dioxide and radon. The most popular statistical models applied to IAQ were artificial neural networks, multiple linear regression, partial least squares, and decision trees.

Development of Sustainable Recycling Investment Framework Considering Uncertain Demand and Nonlinear Recycling Cost

This paper presents a more active and efficient recycling investment strategy that considers the balances among the current production constraints, manufacturing profits, and recycling investments for a sustainable circular economy as compared to the current methods. While existing production planning has numerous uncertainties and nonlinear characteristics, the circular economy-based production planning constitutes more complex uncertainties and nonlinear characteristics that result from an uncertain return rate, demand uncertainties, and nonlinear return on investment costs. This paper suggests a stochastic nonlinear programming model-based active recycling investment framework so as to generate a more effective process plan to handle these characteristics. In the proposed framework, recycling investment strategies are quantitatively analyzed when considering uncertain demand and unclear production conditions. In addition, the effective solving techniques for the circular economy based production framework are obtained while using Monte-Carlo based sample average approximation and memetic algorithm. To prove the effectiveness of the proposed framework, it is implemented for a given system and the numerical analyses that were conducted for the various sustainable manufacturing scenarios.

High-Order Temporal Correlation Model Learning for Time-Series Prediction

Time-series prediction has become a prominent challenge, especially when the data are described as sequences of multiway arrays. Because noise and redundancy may exist in the tensor representation of a time series, we focus on solving the problem of high-order time-series prediction under a tensor decomposition framework and develop two novel multilinear models: 1) the multilinear orthogonal autoregressive (MOAR) model and 2) the multilinear constrained autoregressive (MCAR) model. The MOAR model is designed to preserve as much information as possible from the original tensorial data under orthogonal constraints. The MCAR model is an enhanced version that is developed by replacing orthogonal constraints with an inverse decomposition error term. For both models, we project the original tensor into subspaces spanned by basis matrices to facilitate the discovery of the intrinsic temporal structure embedded in the original tensor. To build connections among consecutive slices of the tensor, we generalize a traditional autoregressive model to tensor form to better preserve the temporal smoothness. Experiments conducted on four publicly available datasets demonstrate that our proposed methods converge within a small number of iterations during the training stage and achieve promising results compared with state-of-the-art methods.

Recovery of signal loss adopting the residual bootstrap method in fetal heart rate dynamics

Fetal heart rate (FHR) data obtained from a non-stress test (NST) can be presented in a type of time series, which is accompanied by signal loss due to physical and biological causes. To recover or estimate FHR data, which is subjected to a high rate of signal loss, time series models [second-order autoregressive (AR(2)), first-order autoregressive conditional heteroscedasticity (ARCH(1)) and empirical mode decomposition and vector autoregressive (EMD-VAR)] and the residual bootstrap method were applied. The ARCH(1) model with the residual bootstrap technique was the most accurate [root mean square error (RMSE), 2.065] as it reflects the nonlinearity of the FHR data [mean absolute error (MAE) for approximate entropy (ApEn), 0.081]. As a result, the goal of predicting fetal health and identifying a high-risk pregnancy could be achieved. These trials may be effectively used to save the time and cost of repeating the NST when the fetal diagnosis is impossible owing to a large amount of signal loss.

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery

PURPOSE

With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge.

METHODS

We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels.

RESULTS

We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial.

CONCLUSION

This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

Heterogeneous recurrence analysis of heartbeat dynamics for the identification of sleep apnea events

Obstructive sleep apnea (OSA) is a common sleep disorder that affects 24% of adult men and 9% of adult women. It occurs due to the occlusion of the upper airway during sleep, thereby leading to a decrease of blood oxygen level that triggers arousals and sleep fragmentation. OSA significantly impacts the quality of sleep and it is known to be responsible for a number of health complications, such as high blood pressure and type 2 diabetes. Traditional diagnosis of OSA relies on polysomnography, which is expensive, time-consuming and inaccessible to the general population. Recent advancement of sensing provides an unprecedented opportunity for the screening of OSA events using single-channel electrocardiogram (ECG). However, existing approaches are limited in their ability to characterize nonlinear dynamics underlying ECG signals. As such, hidden patterns of OSA-altered cardiac electrical activity cannot be fully revealed and understood. This paper presents a new heterogeneous recurrence model to characterize the heart rate variability for the identification of OSA. A nonlinear state space is firstly reconstructed from a time series of RR intervals that are extracted from single-channel ECGs. Further, the state space is recursively partitioned into a hierarchical structure of local recurrence regions. A new fractal representation is designed to efficiently characterize state transitions among segmented sub-regions. Statistical measures are then developed to quantify heterogeneous recurrence patterns. In addition, we integrate classification models with heterogeneous recurrence features to differentiate healthy subjects from OSA patients. Experimental results show that the proposed approach captures heterogeneous recurrence patterns in the transformed space and provides an effective tool to detect OSA using one-lead ECG signals.

Inferring sparse networks for noisy transient processes

Inferring causal structures of real world complex networks from measured time series signals remains an open issue. The current approaches are inadequate to discern between direct versus indirect influences (i.e., the presence or absence of a directed arc connecting two nodes) in the presence of noise, sparse interactions, as well as nonlinear and transient dynamics of real world processes. We report a sparse regression (referred to as the l1-min) approach with theoretical bounds on the constraints on the allowable perturbation to recover the network structure that guarantees sparsity and robustness to noise. We also introduce averaging and perturbation procedures to further enhance prediction scores (i.e., reduce inference errors), and the numerical stability of l1-min approach. Extensive investigations have been conducted with multiple benchmark simulated genetic regulatory network and Michaelis-Menten dynamics, as well as real world data sets from DREAM5 challenge. These investigations suggest that our approach can significantly improve, oftentimes by 5 orders of magnitude over the methods reported previously for inferring the structure of dynamic networks, such as Bayesian network, network deconvolution, silencing and modular response analysis methods based on optimizing for sparsity, transients, noise and high dimensionality issues.