Statistical methods of parameter estimation for deterministically chaotic time series

Unwinding the model manifold: Choosing similarity measures to remove local minima in sloppy dynamical systems

In this paper, we consider the problem of parameter sensitivity in models of complex dynamical systems through the lens of information geometry. We calculate the sensitivity of model behavior to variations in parameters. In most cases, models are sloppy, that is, exhibit an exponential hierarchy of parameter sensitivities. We propose a parameter classification scheme based on how the sensitivities scale at long observation times. We show that for oscillatory models, either with a limit cycle or a strange attractor, sensitivities can become arbitrarily large, which implies a high effective dimensionality on the model manifold. Sloppy models with a single fixed point have model manifolds with low effective dimensionality, previously described as a "hyper-ribbon." In contrast, models with high effective dimensionality translate into multimodal fitting problems. We define a measure of curvature on the model manifold which we call the winding frequency that estimates the density of local minima in the model's parameter space. We then show how alternative choices of fitting metrics can "unwind" the model manifold and give low winding frequencies. This prescription translates the model manifold from one of high effective dimensionality into the hyper-ribbon structures observed elsewhere. This translation opens the door for applications of sloppy model analysis and model reduction methods developed for models with low effective dimensionality.

Why preferring parametric forecasting to nonparametric methods?

A recent series of papers by Charles T. Perretti and collaborators have shown that nonparametric forecasting methods can outperform parametric methods in noisy nonlinear systems. Such a situation can arise because of two main reasons: the instability of parametric inference procedures in chaotic systems which can lead to biased parameter estimates, and the discrepancy between the real system dynamics and the modeled one, a problem that Perretti and collaborators call "the true model myth". Should ecologists go on using the demanding parametric machinery when trying to forecast the dynamics of complex ecosystems? Or should they rely on the elegant nonparametric approach that appears so promising? It will be here argued that ecological forecasting based on parametric models presents two key comparative advantages over nonparametric approaches. First, the likelihood of parametric forecasting failure can be diagnosed thanks to simple Bayesian model checking procedures. Second, when parametric forecasting is diagnosed to be reliable, forecasting uncertainty can be estimated on virtual data generated with the fitted to data parametric model. In contrast, nonparametric techniques provide forecasts with unknown reliability. This argumentation is illustrated with the simple theta-logistic model that was previously used by Perretti and collaborators to make their point. It should convince ecologists to stick to standard parametric approaches, until methods have been developed to assess the reliability of nonparametric forecasting.

Parameter estimation through ignorance

Dynamical modeling lies at the heart of our understanding of physical systems. Its role in science is deeper than mere operational forecasting, in that it allows us to evaluate the adequacy of the mathematical structure of our models. Despite the importance of model parameters, there is no general method of parameter estimation outside linear systems. A relatively simple method of parameter estimation for nonlinear systems is introduced, based on variations in the accuracy of probability forecasts. It is illustrated on the logistic map, the Henon map, and the 12-dimensional Lorenz96 flow, and its ability to outperform linear least squares in these systems is explored at various noise levels and sampling rates. As expected, it is more effective when the forecast error distributions are non-Gaussian. The method selects parameter values by minimizing a proper, local skill score for continuous probability forecasts as a function of the parameter values. This approach is easier to implement in practice than alternative nonlinear methods based on the geometry of attractors or the ability of the model to shadow the observations. Direct measures of inadequacy in the model, the "implied ignorance," and the information deficit are introduced.

Random dynamical models from time series

In this work we formulate a consistent Bayesian approach to modeling stochastic (random) dynamical systems by time series and implement it by means of artificial neural networks. The feasibility of this approach for both creating models adequately reproducing the observed stationary regime of system evolution, and predicting changes in qualitative behavior of a weakly nonautonomous stochastic system, is demonstrated on model examples. In particular, a successful prognosis of stochastic system behavior as compared to the observed one is illustrated on model examples, including discrete maps disturbed by non-Gaussian and nonuniform noise and a flow system with Langevin force.

Nonlinear systems identification by combining regression with bootstrap resampling

A new parameter estimation method for nonlinear systems from time series data is proposed. For the purpose of unbiased estimation, we employ the idea of bootstrap method on regression problems. Our method can be applied into even short and noisy data and is expected to give us a robust estimation. Some benchmarks of estimating chaotic models show its practical applicability. We also try to apply this method to analysis for intermittent hormonal therapy for prostate cancer by using a mathematical model and real clinical data.

Seeker optimization algorithm for parameter estimation of time-delay chaotic systems

Time-delay chaotic systems have some very interesting properties, and their parameter estimation has received increasing interest in the recent years. It is well known that parameter estimation of a chaotic system is a nonlinear, multivariable, and multimodal optimization problem for which global optimization techniques are required in order to avoid local minima. In this work, a seeker-optimization-algorithm (SOA)-based method is proposed to address this issue. In the SOA, search direction is based on the empirical gradients by evaluating the response to the position changes, and step length is based on uncertainty reasoning by using a simple fuzzy rule. The performance of the algorithm is evaluated on two typical test systems. Moreover, two state-of-the-art algorithms (i.e., particle swarm optimization and differential evolution) are also considered for comparison. The simulation results show that the proposed algorithm is better than or at least as good as the other two algorithms and can effectively solve the parameter estimation problem of time-delay chaotic systems.

Adaptive scheme for synchronization-based multiparameter estimation from a single chaotic time series and its applications

Chaos-synchronization-based multiparameter estimation of a multiply delayed feedback system is investigated. We propose an adaptive method that can estimate all the parameters of the response system using the driving signal only. In the past few years, various methods have been developed for estimation of multiparameters of a chaotic system but most of them require more than one time series to estimate all the parameters of a chaotic or hyperchaotic system. The proposed method requires only a single chaotic time series to estimate all the parameters. A sufficient condition for synchronization is derived and it is shown that the numerical results well support the analytic calculations. The synchronized system has applications in cryptographic encoding for digital and analog signals, which is shown with an example.

Markov chain Monte Carlo method in Bayesian reconstruction of dynamical systems from noisy chaotic time series

The impossibility to use the MCMC (Markov chain Monte Carlo) methods for long noisy chaotic time series (TS) (due to high computational complexity) is a serious limitation for reconstruction of dynamical systems (DSs). In particular, it does not allow one to use the universal Bayesian approach for reconstruction of a DS in the most interesting case of the unknown evolution operator of the system. We propose a technique that makes it possible to use the MCMC methods for Bayesian reconstruction of a DS from noisy chaotic TS of arbitrary long duration.

Parameter identification technique for uncertain chaotic systems using state feedback and steady-state analysis

A technique is introduced for identifying uncertain and/or unknown parameters of chaotic dynamical systems via using simple state feedback. The proposed technique is based on bringing the system into a stable steady state and then solving for the unknown parameters using a simple algebraic method that requires access to the complete or partial states of the system depending on the dynamical model of the chaotic system. The choice of the state feedback is optimized in terms of practicality and causality via employing a single feedback signal and tuning the feedback gain to ensure both stability and identifiability. The case when only a single scalar time series of one of the states is available is also considered and it is demonstrated that a synchronization-based state observer can be augmented to the state feedback to address this problem. A detailed case study using the Lorenz system is used to exemplify the suggested technique. In addition, both the Rössler and Chua systems are examined as possible candidates for utilizing the proposed methodology when partial identification of the unknown parameters is considered. Finally, the dependence of the proposed technique on the structure of the chaotic dynamical model and the operating conditions is discussed and its advantages and limitations are highlighted via comparing it with other methods reported in the literature.

Monte Carlo method for multiparameter estimation in coupled chaotic systems

We address the problem of estimating multiple parameters of a chaotic dynamical model from the observation of a scalar time series. We assume that the series is produced by a chaotic system with the same functional form as the model, so that synchronization between the two systems can be achieved by an adequate coupling. In this scenario, we propose an efficient Monte Carlo optimization algorithm that iteratively updates the model parameters in order to minimize the synchronization error. As an example, we apply it to jointly estimate the three static parameters of a chaotic Lorenz system.

Chaos synchronization and parameter estimation from a scalar output signal

We propose an observer-based approach for chaos synchronization and parameter estimation from a scalar output signal. To begin with, we use geometric control to transform the master system into a standard form with zero dynamics. Then we construct a slaver to synchronize with the master using a combination of slide mode control and linear feedback control. Within a finite time, partial synchronization is realized, which further results in complete synchronization as time tends to infinity. Even if there exists model uncertainty in the slaver, we can also estimate the unknown model parameter by a simple adaptive rule.

Failure of maximum likelihood methods for chaotic dynamical systems

The maximum likelihood method is a basic statistical technique for estimating parameters and variables, and is the starting point for many more sophisticated methods, like Bayesian methods. This paper shows that maximum likelihood fails to identify the true trajectory of a chaotic dynamical system, because there are trajectories that appear to be far more (infinitely more) likely than truth. This failure occurs for unbounded noise and for bounded noise when it is sufficiently large and will almost certainly have consequences for parameter estimation in such systems. The reason for the failure is rather simple; in chaotic dynamical systems there can be trajectories that are consistently closer to the observations than the true trajectory being observed, and hence their likelihood dominates truth. The residuals of these truth-dominating trajectories are not consistent with the noise distribution; they would typically have too small standard deviation and many outliers, and hence the situation may be remedied by using methods that examine the distribution of residuals and are not entirely maximum likelihood based.

Modified Bayesian approach for the reconstruction of dynamical systems from time series

Some recent papers were concerned with applicability of the Bayesian (statistical) approach to reconstruction of dynamic systems (DS) from experimental data. A significant merit of the approach is its universality. But, being correct in terms of meeting conditions of the underlying theorem, the Bayesian approach to reconstruction of DS is hard to realize in the most interesting case of noisy chaotic time series (TS). In this work we consider a modification of the Bayesian approach that can be used for reconstruction of DS from noisy TS. We demonstrate efficiency of the modified approach for solution of two types of problems: (1) finding values of parameters of a known DS by noisy TS; (2) classification of modes of behavior of such a DS by short TS with pronounced noise.

Geometric noise reduction for multivariate time series

We propose an algorithm for the reduction of observational noise in chaotic multivariate time series. The algorithm is based on a maximum likelihood criterion, and its goal is to reduce the mean distance of the points of the cleaned time series to the attractor. We give evidence of the convergence of the empirical measure associated with the cleaned time series to the underlying invariant measure, implying the possibility to predict the long run behavior of the true dynamics.

Medium-term prediction of chaos

We study prediction of chaotic time series when a perfect model is available but the initial condition is measured with uncertainty. A common approach for predicting future data given these circumstances is to apply the model despite the uncertainty. In systems with fold dynamics, we find prediction is improved over this strategy by recognizing this behavior. A systematic study of the Logistic map demonstrates prediction of the most likely trajectory can be extended three time steps. Finally, we discuss application of these ideas to the Rössler attractor.

Adaptive approximation method for joint parameter estimation and identical synchronization of chaotic systems

We introduce a numerical approximation method for estimating an unknown parameter of a (primary) chaotic system which is partially observed through a scalar time series. Specifically, we show that the recursive minimization of a suitably designed cost function that involves the dynamic state of a fully observed (secondary) system and the observed time series can lead to the identical synchronization of the two systems and the accurate estimation of the unknown parameter. The salient feature of the proposed technique is that the only external input to the secondary system is the unknown parameter which needs to be adjusted. We present numerical examples for the Lorenz system which show how our algorithm can be considerably faster than some previously proposed methods.