Robust initialization of a Jordan network with recurrent constrained learning

Applications of Recurrent Neural Networks in Environmental Factor Forecasting: A Review

Analysis and forecasting of sequential data, key problems in various domains of engineering and science, have attracted the attention of many researchers from different communities. When predicting the future probability of events using time series, recurrent neural networks (RNNs) are an effective tool that have the learning ability of feedforward neural networks and expand their expression ability using dynamic equations. Moreover, RNNs are able to model several computational structures. Researchers have developed various RNNs with different architectures and topologies. To summarize the work of RNNs in forecasting and provide guidelines for modeling and novel applications in future studies, this review focuses on applications of RNNs for time series forecasting in environmental factor forecasting. We present the structure, processing flow, and advantages of RNNs and analyze the applications of various RNNs in time series forecasting. In addition, we discuss limitations and challenges of applications based on RNNs and future research directions. Finally, we summarize applications of RNNs in forecasting.

Robust Recurrent Kernel Online Learning

We propose a robust recurrent kernel online learning (RRKOL) algorithm based on the celebrated real-time recurrent learning approach that exploits the kernel trick in a recurrent online training manner. The novel RRKOL algorithm guarantees weight convergence with regularized risk management through the use of adaptive recurrent hyperparameters for superior generalization performance. Based on a new concept of the structure update error with a variable parameter length, we are the first one to propose the detailed structure update error, such that the weight convergence and robust stability proof can be integrated with a kernel sparsification scheme based on a solid theoretical ground. The RRKOL algorithm automatically weighs the regularized term in the recurrent loss function, such that we not only minimize the estimation error but also improve the generalization performance through sparsification with simulation support.

A linear recurrent kernel online learning algorithm with sparse updates

In this paper, we propose a recurrent kernel algorithm with selectively sparse updates for online learning. The algorithm introduces a linear recurrent term in the estimation of the current output. This makes the past information reusable for updating of the algorithm in the form of a recurrent gradient term. To ensure that the reuse of this recurrent gradient indeed accelerates the convergence speed, a novel hybrid recurrent training is proposed to switch on or off learning the recurrent information according to the magnitude of the current training error. Furthermore, the algorithm includes a data-dependent adaptive learning rate which can provide guaranteed system weight convergence at each training iteration. The learning rate is set as zero when the training violates the derived convergence conditions, which makes the algorithm updating process sparse. Theoretical analyses of the weight convergence are presented and experimental results show the good performance of the proposed algorithm in terms of convergence speed and estimation accuracy.

Reinforced two-step-ahead weight adjustment technique for online training of recurrent neural networks

A reliable forecast of future events possesses great value. The main purpose of this paper is to propose an innovative learning technique for reinforcing the accuracy of two-step-ahead (2SA) forecasts. The real-time recurrent learning (RTRL) algorithm for recurrent neural networks (RNNs) can effectively model the dynamics of complex processes and has been used successfully in one-step-ahead forecasts for various time series. A reinforced RTRL algorithm for 2SA forecasts using RNNs is proposed in this paper, and its performance is investigated by two famous benchmark time series and a streamflow during flood events in Taiwan. Results demonstrate that the proposed reinforced 2SA RTRL algorithm for RNNs can adequately forecast the benchmark (theoretical) time series, significantly improve the accuracy of flood forecasts, and effectively reduce time-lag effects.