Prediction of spatiotemporal time series based on reconstructed local states

Learning spatiotemporal chaos using next-generation reservoir computing

Forecasting the behavior of high-dimensional dynamical systems using machine learning requires efficient methods to learn the underlying physical model. We demonstrate spatiotemporal chaos prediction using a machine learning architecture that, when combined with a next-generation reservoir computer, displays state-of-the-art performance with a computational time ¹⁰- ¹⁰ times faster for training process and training data set ∼ ¹⁰ times smaller than other machine learning algorithms. We also take advantage of the translational symmetry of the model to further reduce the computational cost and training data, each by a factor of ∼10.

Role of assortativity in predicting burst synchronization using echo state network

In this study, we use a reservoir computing based echo state network (ESN) to predict the collective burst synchronization of neurons. Specifically, we investigate the ability of ESN in predicting the burst synchronization of an ensemble of Rulkov neurons placed on a scale-free network. We have shown that a limited number of nodal dynamics used as input in the machine can capture the real trend of burst synchronization in this network. Further, we investigate the proper selection of nodal inputs of degree-degree (positive and negative) correlated networks. We show that for a disassortative network, selection of different input nodes based on degree has no significant role in the machine's prediction. However, in the case of assortative network, training the machine with the information (i.e., time series) of low degree nodes gives better results in predicting the burst synchronization. The results are found to be consistent with the investigation carried out with a continuous time Hindmarsh-Rose neuron model. Furthermore, the role of hyperparameters like spectral radius and leaking parameter of ESN on the prediction process has been examined. Finally, we explain the underlying mechanism responsible for observing these differences in the prediction in a degree correlated network.

Approaches to Parameter Estimation from Model Neurons and Biological Neurons

Model optimization in neuroscience has focused on inferring intracellular parameters from time series observations of the membrane voltage and calcium concentrations. These parameters constitute the fingerprints of ion channel subtypes and may identify ion channel mutations from observed changes in electrical activity. A central question in neuroscience is whether computational methods may obtain ion channel parameters with sufficient consistency and accuracy to provide new information on the underlying biology. Finding single-valued solutions in particular, remains an outstanding theoretical challenge. This note reviews recent progress in the field. It first covers well-posed problems and describes the conditions that the model and data need to meet to warrant the recovery of all the original parameters—even in the presence of noise. The main challenge is model error, which reflects our lack of knowledge of exact equations. We report on strategies that have been partially successful at inferring the parameters of rodent and songbird neurons, when model error is sufficiently small for accurate predictions to be made irrespective of stimulation.

Sequence-to-sequence prediction of spatiotemporal systems

We propose a novel type of neural networks known as "attention-based sequence-to-sequence architecture" for a model-free prediction of spatiotemporal systems. This architecture is composed of an encoder and a decoder in which the encoder acts upon a given input sequence and then the decoder yields another output sequence to make a multistep prediction at a time. In order to demonstrate the potential of this approach, we train the neural network using data numerically sampled from the Korteweg-de Vries equation-which describes the interaction between solitary waves-and then predict its future evolution. Furthermore, we validate the applicability of the approach on datasets sampled from the chaotic Lorenz system and three other partial differential equations. The results show that the proposed method can achieve good performance in predicting the evolutionary behavior of studied spatiotemporal dynamics. To the best of our knowledge, this work is the first attempt at applying attention-based sequence-to-sequence architecture to the prediction task of solitary waves.

Optimal solid state neurons

Bioelectronic medicine is driving the need for neuromorphic microcircuits that integrate raw nervous stimuli and respond identically to biological neurons. However, designing such circuits remains a challenge. Here we estimate the parameters of highly nonlinear conductance models and derive the ab initio equations of intracellular currents and membrane voltages embodied in analog solid-state electronics. By configuring individual ion channels of solid-state neurons with parameters estimated from large-scale assimilation of electrophysiological recordings, we successfully transfer the complete dynamics of hippocampal and respiratory neurons in silico. The solid-state neurons are found to respond nearly identically to biological neurons under stimulation by a wide range of current injection protocols. The optimization of nonlinear models demonstrates a powerful method for programming analog electronic circuits. This approach offers a route for repairing diseased biocircuits and emulating their function with biomedical implants that can adapt to biofeedback.

Convolutional autoencoder and conditional random fields hybrid for predicting spatial-temporal chaos

We present an approach for data-driven prediction of high-dimensional chaotic time series generated by spatially-extended systems. The algorithm employs a convolutional autoencoder for dimension reduction and feature extraction combined with a probabilistic prediction scheme operating in the feature space, which consists of a conditional random field. The future evolution of the spatially-extended system is predicted using a feedback loop and iterated predictions. The excellent performance of this method is illustrated and evaluated using Lorenz-96 systems and Kuramoto-Sivashinsky equations of different size generating time series of different dimensionality and complexity.

Optimal neural network feature selection for spatial-temporal forecasting

Neural networks, and in general machine learning techniques, have been widely employed in forecasting time series and more recently in predicting spatial-temporal signals. All of these approaches involve some kind of feature selection regarding what past data and what neighbor data to use for forecasting. In this article, we show extensive empirical evidence on how to independently construct the optimal feature selection or input representation used by the input layer of a feed forward neural network for the purpose of forecasting spatial-temporal signals. The approach is based on results from the dynamical systems theory, namely, nonlinear embedding theorems. We demonstrate it for a variety of spatial-temporal signals and show that the optimal input layer representation consists of a grid, with spatial-temporal lags determined by the minimum of the mutual information of the spatial-temporal signals and the number of points taken in space-time decided by the embedding dimension of the signal. We present evidence of this proposal by running a Monte Carlo simulation of several combinations of input layer feature designs and show that the one predicted by the nonlinear embedding theorems seems to be optimal or close to being optimal. In total, we show evidence in four unrelated systems: a series of coupled Hénon maps, a series of coupled ordinary differential equations (Lorenz-96) phenomenologically modeling atmospheric dynamics, the Kuramoto-Sivashinsky equation, a partial differential equation used in studies of instabilities in laminar flame fronts, and finally real physical data from sunspot areas in the Sun (in latitude and time) from 1874 to 2015. These four examples cover the range from simple toy models to complex nonlinear dynamical simulations and real data. Finally, we also compare our proposal against alternative feature selection methods and show that it also works for other machine learning forecasting models.

Influence of aggregation on benthic coral reef spatio-temporal dynamics

Spatial patterning of coral reef sessile benthic organisms can constrain competitive and demographic rates, with implications for dynamics over a range of time scales. However, techniques for quantifying and analysing reefscape behaviour, particularly at short to intermediate time scales (weeks to decades), are lacking. An analysis of the dynamics of coral reefscapes simulated with a lattice model shows consistent trends that can be categorized into four stages: a repelling stage that moves rapidly away from an unstable initial condition, a transient stage where spatial rearrangements bring key competitors into contact, an attracting stage where the reefscape decays to a steady-state attractor, and an attractor stage. The transient stage exhibits nonlinear dynamics, whereas the other stages are linear. The relative durations of the stages are affected by the initial spatial configuration as characterized by coral aggregation-a measure of spatial clumpiness, which together with coral and macroalgae fractional cover, more completely describe modelled reefscape dynamics. Incorporating diffusional processes results in aggregated patterns persisting in the attractor. Our quantitative characterization of reefscape dynamics has possible applications to other spatio-temporal systems and implications for reef restoration: high initial aggregation patterns slow losses in herbivore-limited systems and low initial aggregation configurations accelerate growth in herbivore-dominated systems.

Observing spatio-temporal dynamics of excitable media using reservoir computing

We present a dynamical observer for two dimensional partial differential equation models describing excitable media, where the required cross prediction from observed time series to not measured state variables is provided by Echo State Networks receiving input from local regions in space, only. The efficacy of this approach is demonstrated for (noisy) data from a (cubic) Barkley model and the Bueno-Orovio-Cherry-Fenton model describing chaotic electrical wave propagation in cardiac tissue.

Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach

We demonstrate the effectiveness of using machine learning for model-free prediction of spatiotemporally chaotic systems of arbitrarily large spatial extent and attractor dimension purely from observations of the system's past evolution. We present a parallel scheme with an example implementation based on the reservoir computing paradigm and demonstrate the scalability of our scheme using the Kuramoto-Sivashinsky equation as an example of a spatiotemporally chaotic system.

Estimating varying coefficients for partial differential equation models

Partial differential equations (PDEs) are used to model complex dynamical systems in multiple dimensions, and their parameters often have important scientific interpretations. In some applications, PDE parameters are not constant but can change depending on the values of covariates, a feature that we call varying coefficients. We propose a parameter cascading method to estimate varying coefficients in PDE models from noisy data. Our estimates of the varying coefficients are shown to be consistent and asymptotically normally distributed. The performance of our method is evaluated by a simulation study and by an empirical study estimating three varying coefficients in a PDE model arising from LIDAR data.

Nonlinear forecasting of intertidal shoreface evolution

Natural systems dominated by sediment transport are notoriously difficult to forecast. This is particularly true along the ocean coastline, a region that draws considerable human attention as economic investment and infrastructure are threatened by both persistent, long-term and acute, event driven processes (i.e., sea level rise and storm damage, respectively). Forecasting the coastline's evolution over intermediate time (daily) and space (tens of meters) scales is hindered by the complexity of sediment transport and hydrodynamics, and limited access to the detailed local forcing that drives fast scale processes. Modern remote sensing systems provide an efficient, economical means to collect data within these regions. A solar-powered digital camera installation is used to capture the coast's evolution, and machine learning algorithms are implemented to extract the shoreline and estimate the daily mean intertidal coastal profile. Methods in nonlinear time series forecasting and genetic programming applied to these data corroborate that coastal morphology at these scales is predominately driven by nonlinear internal dynamics, which partially mask external forcing signatures. Results indicate that these forecasting techniques achieve nontrivial predictive skill for spatiotemporal forecast of the upper coastline profile (as much as 43% of variance in data explained for one day predictions). This analysis provides evidence that societally relevant coastline forecasts can be achieved without knowing the forcing environment or the underlying dynamical equations that govern coastline evolution.

Data-driven prediction and prevention of extreme events in a spatially extended excitable system

Extreme events occur in many spatially extended dynamical systems, often devastatingly affecting human life, which makes their reliable prediction and efficient prevention highly desirable. We study the prediction and prevention of extreme events in a spatially extended system, a system of coupled FitzHugh-Nagumo units, in which extreme events occur in a spatially and temporally irregular way. Mimicking typical constraints faced in field studies, we assume not to know the governing equations of motion and to be able to observe only a subset of all phase-space variables for a limited period of time. Based on reconstructing the local dynamics from data and despite being challenged by the rareness of events, we are able to predict extreme events remarkably well. With small, rare, and spatiotemporally localized perturbations which are guided by our predictions, we are able to completely suppress extreme events in this system.

Nonlinear time-series analysis revisited

In 1980 and 1981, two pioneering papers laid the foundation for what became known as nonlinear time-series analysis: the analysis of observed data-typically univariate-via dynamical systems theory. Based on the concept of state-space reconstruction, this set of methods allows us to compute characteristic quantities such as Lyapunov exponents and fractal dimensions, to predict the future course of the time series, and even to reconstruct the equations of motion in some cases. In practice, however, there are a number of issues that restrict the power of this approach: whether the signal accurately and thoroughly samples the dynamics, for instance, and whether it contains noise. Moreover, the numerical algorithms that we use to instantiate these ideas are not perfect; they involve approximations, scale parameters, and finite-precision arithmetic, among other things. Even so, nonlinear time-series analysis has been used to great advantage on thousands of real and synthetic data sets from a wide variety of systems ranging from roulette wheels to lasers to the human heart. Even in cases where the data do not meet the mathematical or algorithmic requirements to assure full topological conjugacy, the results of nonlinear time-series analysis can be helpful in understanding, characterizing, and predicting dynamical systems.

Parameter Estimation of Partial Differential Equation Models

Partial differential equation (PDE) models are commonly used to model complex dynamic systems in applied sciences such as biology and finance. The forms of these PDE models are usually proposed by experts based on their prior knowledge and understanding of the dynamic system. Parameters in PDE models often have interesting scientific interpretations, but their values are often unknown, and need to be estimated from the measurements of the dynamic system in the present of measurement errors. Most PDEs used in practice have no analytic solutions, and can only be solved with numerical methods. Currently, methods for estimating PDE parameters require repeatedly solving PDEs numerically under thousands of candidate parameter values, and thus the computational load is high. In this article, we propose two methods to estimate parameters in PDE models: a parameter cascading method and a Bayesian approach. In both methods, the underlying dynamic process modeled with the PDE model is represented via basis function expansion. For the parameter cascading method, we develop two nested levels of optimization to estimate the PDE parameters. For the Bayesian method, we develop a joint model for data and the PDE, and develop a novel hierarchical model allowing us to employ Markov chain Monte Carlo (MCMC) techniques to make posterior inference. Simulation studies show that the Bayesian method and parameter cascading method are comparable, and both outperform other available methods in terms of estimation accuracy. The two methods are demonstrated by estimating parameters in a PDE model from LIDAR data.

Evidence-based modeling of network discharge dynamics during periodic pacing to control epileptiform activity

Deep brain stimulation (DBS) is a promising therapeutic approach for epilepsy treatment. Recently, research has focused on the implementation of stimulation protocols that would adapt to the patients need (adaptive stimulation) and deliver electrical stimuli only when it is most useful. A formal mathematical description of the effects of electrical stimulation on neuronal networks is a prerequisite for the development of adaptive DBS algorithms. Using tools from non-linear dynamic analysis, we describe an evidence-based, mathematical modeling approach that (1) accurately simulates epileptiform activity at time-scales of single and multiple ictal discharges, (2) simulates modulation of neural dynamics during epileptiform activity in response to fixed, low-frequency electrical stimulation, (3) defines a mapping from real-world observations to model state, and (4) defines a mapping from model state to real-world observations. We validate the real-world utility of the model's properties by statistical comparison between the number, duration, and interval of ictal-like discharges observed in vitro and those simulated in silica under conditions of repeated stimuli at fixed-frequency. These validation results confirm that the evidence-based modeling approach captures robust, informative features of neural network dynamics of in vitro epileptiform activity under periodic pacing and support its use for further implementation of adaptive DBS protocols for epilepsy treatment.

An adaptive wavelet neural network for spatio-temporal system identification

Starting from the basic concept of coupled map lattices, a new family of adaptive wavelet neural networks (AWNN) is introduced for spatio-temporal system identification, by combining an efficient wavelet representation with a coupled map lattice model. A new orthogonal projection pursuit (OPP) method, coupled with a particle swarm optimization (PSO) algorithm, is proposed for augmenting the proposed network. A novel two-stage hybrid training scheme is developed for constructing a parsimonious network model. In the first stage, by applying the orthogonal projection pursuit algorithm, significant wavelet neurons are adaptively and successively recruited into the network, where adjustable parameters of the associated wavelet neurons are optimized using a particle swarm optimizer. The resultant network model, obtained in the first stage, may however be redundant. In the second stage, an orthogonal least squares algorithm is then applied to refine and improve the initially trained network by removing redundant wavelet neurons from the network. The proposed two-stage hybrid training procedure can generally produce a parsimonious network model, where a ranked list of wavelet neurons, according to the capability of each neuron to represent the total variance in the system output signal is produced. Two spatio-temporal system identification examples are presented to demonstrate the performance of the proposed new modelling framework.

Inferring local dynamics and connectivity of spatially extended systems with long-range links based on steady-state stabilization

A method is presented for system identification of spatially extended systems with structural inhomogeneities of local dynamics and additional long-range links. The proposed identification procedure is based on steady-state stabilization and is illustrated with an inhomogeneous two-dimensional grid of coupled FitzHugh-Nagumo models.

Spatiotemporal system identification on nonperiodic domains using Chebyshev spectral operators and system reduction algorithms

A system identification methodology based on Chebyshev spectral operators and an orthogonal system reduction algorithm is proposed, leading to a new approach for data-driven modeling of nonlinear spatiotemporal systems on nonperiodic domains. A continuous model structure is devised allowing for terms of arbitrary derivative order and nonlinearity degree. Chebyshev spectral operators are introduced to realm of inverse problems to discretize that continuous structure and arrive with spectral accuracy at a discrete form. Finally, least squares combined with an orthogonal system reduction algorithm are employed to solve for the parameters and eliminate the redundancies to achieve a parsimonious model. A numerical case study of identifying the Allen-Cahn metastable equation demonstrates the superior accuracy of the proposed Chebyshev spectral identification over its finite difference counterpart.

Spatiotemporal system reconstruction using Fourier spectral operators and structure selection techniques

A technique based on trigonometric spectral methods and structure selection is proposed for the reconstruction, from observed time series, of spatiotemporal systems governed by nonlinear partial differential equations of polynomial type with terms of arbitrary derivative order and nonlinearity degree. The system identification using Fourier spectral differentiation operators in conjunction with a structure selection procedure leads to a parsimonious model of the original system by detecting and eliminating the redundant parameters using orthogonal decomposition of the state data. Implementation of the technique is exemplified for a highly stiff reaction-diffusion system governed by the Kuramoto-Sivashinsky equation. Numerical experiments demonstrate the superior performance of the proposed technique in terms of accuracy as well as robustness, even with smaller sets of sampling data.

Neighborhood detection for the identification of spatiotemporal systems

Neighborhood detection and local state vector construction for the identification of spatiotemporal systems is considered in this paper. Determining the neighborhood size both in the space and time domain can considerably reduce the complexity of the set of candidate model terms for the identification of coupled map lattice models. The computation requirements of the model identification algorithm can also be greatly reduced instead of the more direct identification approach of searching over the entire spatiotemporal neighborhood in the original space. In this paper, a new neighborhood detection method is introduced based on embedding theory for nonlinear dynamical systems to produce an initial spatiotemporal neighborhood for the identification of spatiotemporal systems. Numerical examples are provided to demonstrate the feasibility and applicability of the new neighborhood detection method.

Nonlinear spatial-temporal prediction based on optimal fusion

The problem of spatial-temporal signal processing and modeling has been of great interest in recent years. A new spatial-temporal prediction method is presented in this paper. An optimal fusion scheme based on fourth-order statistic is first employed to combine the received signals at different spatial domains. The fused signal is then used to construct a spatial-temporal predictor by a support vector machine. It is shown theoretically that the proposed method has an improved performance even in non-Gaussian environments. To demonstrate the practicality of this spatial-temporal predictor, we apply it to model real-life radar sea scattered signals. Experimental results show that the proposed method can provide a more accurate model for sea clutter than the conventional methods.

Automatic filters for the detection of coherent structure in spatiotemporal systems

Most current methods for identifying coherent structures in spatially extended systems rely on prior information about the form which those structures take. Here we present two approaches to automatically filter the changing configurations of spatial dynamical systems and extract coherent structures. One, local sensitivity filtering, is a modification of the local Lyapunov exponent approach suitable to cellular automata and other discrete spatial systems. The other, local statistical complexity filtering, calculates the amount of information needed for optimal prediction of the system's behavior in the vicinity of a given point. By examining the changing spatiotemporal distributions of these quantities, we can find the coherent structures in a variety of pattern-forming cellular automata, without needing to guess or postulate the form of that structure. We apply both filters to elementary and cyclical cellular automata (ECA and CCA) and find that they readily identify particles, domains, and other more complicated structures. We compare the results from ECA with earlier ones based upon the theory of formal languages and the results from CCA with a more traditional approach based on an order parameter and free energy. While sensitivity and statistical complexity are equally adept at uncovering structure, they are based on different system properties (dynamical and probabilistic, respectively) and provide complementary information.

Quantifying self-organization with optimal predictors

Despite broad interest in self-organizing systems, there are few quantitative, experimentally applicable criteria for self-organization. The existing criteria all give counter-intuitive results for important cases. In this Letter, we propose a new criterion, namely, an internally generated increase in the statistical complexity, the amount of information required for optimal prediction of the system's dynamics. We precisely define this complexity for spatially extended dynamical systems, using the probabilistic ideas of mutual information and minimal sufficient statistics. This leads to a general method for predicting such systems and a simple algorithm for estimating statistical complexity. The results of applying this algorithm to a class of models of excitable media (cyclic cellular automata) strongly support our proposal.

Estimating model parameters by chaos synchronization

Using chaos synchronization and a proposed iterative method of parameter adaptation, we precisely estimate the model parameters of chaotic systems and synchronize two chaotic systems with originally mismatching model parameters. This parameter adaptation method can be applied to a spatiotemporal chaotic system with a one-way-coupled map lattice. As a biomedical application, this method is capable of estimating the asymmetric tension parameter of a vocal fold model.

Identification of nonlinear spatiotemporal systems via partitioned filtering

The problem of identifying continuous spatiotemporal nonlinear systems from noisy and indirect observations is determined by its computational complexity. We propose a solution by means of nonlinear state space filtering along with a state partition technique. The method is demonstrated to be computationally feasible for spatiotemporal data with properties that occur typically in experimental recordings. It is applied to one component of the simulated chaotic data of a two-component reaction diffusion system, yielding estimates of both the unobserved state component and the diffusion constant.

Parameter estimation in spatially extended systems: the Karhunen-Lóeve and Galerkin multiple shooting approach

Parameter estimation for spatiotemporal dynamics for coupled map lattices and continuous time domain systems is shown using a combination of multiple shooting, Karhunen-Loéve decomposition and Galerkin's projection methodologies. The resulting advantages in estimating parameters have been studied and discussed for chaotic and turbulent dynamics using small amounts of data from subsystems, availability of only scalar and noisy time series data, effects of space-time parameter variations, and in the presence of multiple time scales.

Forecasting confined spatiotemporal chaos with genetic algorithms

A technique to forecast spatiotemporal time series is presented. It uses a proper orthogonal or Karhunen-Loève decomposition to encode large spatiotemporal data sets in a few time series, and genetic algorithms to efficiently extract dynamical rules from the data. The method works very well for confined systems displaying spatiotemporal chaos, as exemplified here by forecasting the evolution of the one-dimensional complex Ginzburg-Landau equation in a finite domain.