Model Error, Information Barriers, State Estimation and Prediction in Complex Multiscale Systems

Unambiguous Models and Machine Learning Strategies for Anomalous Extreme Events in Turbulent Dynamical System

Data-driven modeling methods are studied for turbulent dynamical systems with extreme events under an unambiguous model framework. New neural network architectures are proposed to effectively learn the key dynamical mechanisms including the multiscale coupling and strong instability, and gain robust skill for long-time prediction resistive to the accumulated model errors from the data-driven approximation. The machine learning model overcomes the inherent limitations in traditional long short-time memory networks by exploiting a conditional Gaussian structure informed of the essential physical dynamics. The model performance is demonstrated under a prototype model from idealized geophysical flow and passive tracers, which exhibits analytical solutions with representative statistical features. Many attractive properties are found in the trained model in recovering the hidden dynamics using a limited dataset and sparse observation time, showing uniformly high skill with persistent numerical stability in predicting both the trajectory and statistical solutions among different statistical regimes away from the training regime. The model framework is promising to be applied to a wider class of turbulent systems with complex structures.

Estimating the time-evolving refractivity of a turbulent medium using optical beam measurements: a data assimilation approach

In applications such as free-space optical communication, a signal is often recovered after propagation through a turbulent medium. In this setting, it is common to assume that limited information is known about the turbulent medium, such as a space- and time-averaged statistic (e.g., root-mean-square), but without information about the state of the spatial variations. It could be helpful to gain more information if the state of the turbulent medium can be characterized with the spatial variations and evolution in time described. Here, we propose to investigate the use of data assimilation techniques for this purpose. A computational setting is used with the paraxial wave equation, and the extended Kalman filter is used to conduct data assimilation using intensity measurements. To reduce computational cost, the evolution of the turbulent medium is modeled as a stochastic process. Following some past studies, the process has only a small number of Fourier wavelengths for spatial variations. The results show that the spatial and temporal variations of the medium are recovered accurately in many cases. In some time windows in some cases, the error is large for the recovery. Finally, we discuss the potential use of the spatial variation information for aiding the recovery of the transmitted signal or beam source.

Optimal parameterizing manifolds for anticipating tipping points and higher-order critical transitions

A general, variational approach to derive low-order reduced models from possibly non-autonomous systems is presented. The approach is based on the concept of optimal parameterizing manifold (OPM) that substitutes more classical notions of invariant or slow manifolds when the breakdown of "slaving" occurs, i.e., when the unresolved variables cannot be expressed as an exact functional of the resolved ones anymore. The OPM provides, within a given class of parameterizations of the unresolved variables, the manifold that averages out optimally these variables as conditioned on the resolved ones. The class of parameterizations retained here is that of continuous deformations of parameterizations rigorously valid near the onset of instability. These deformations are produced through the integration of auxiliary backward-forward systems built from the model's equations and lead to analytic formulas for parameterizations. In this modus operandi, the backward integration time is the key parameter to select per scale/variable to parameterize in order to derive the relevant parameterizations which are doomed to be no longer exact away from instability onset due to the breakdown of slaving typically encountered, e.g., for chaotic regimes. The selection criterion is then made through data-informed minimization of a least-square parameterization defect. It is thus shown through optimization of the backward integration time per scale/variable to parameterize, that skilled OPM reduced systems can be derived for predicting with accuracy higher-order critical transitions or catastrophic tipping phenomena, while training our parameterization formulas for regimes prior to these transitions takes place.

CEBoosting: Online sparse identification of dynamical systems with regime switching by causation entropy boosting

Regime switching is ubiquitous in many complex dynamical systems with multiscale features, chaotic behavior, and extreme events. In this paper, a causation entropy boosting (CEBoosting) strategy is developed to facilitate the detection of regime switching and the discovery of the dynamics associated with the new regime via online model identification. The causation entropy, which can be efficiently calculated, provides a logic value of each candidate function in a pre-determined library. The reversal of one or a few such causation entropy indicators associated with the model calibrated for the current regime implies the detection of regime switching. Despite the short length of each batch formed by the sequential data, the accumulated value of causation entropy corresponding to a sequence of data batches leads to a robust indicator. With the detected rectification of the model structure, the subsequent parameter estimation becomes a quadratic optimization problem, which is solved using closed analytic formulas. Using the Lorenz 96 model, it is shown that the causation entropy indicator can be efficiently calculated, and the method applies to moderately large dimensional systems. The CEBoosting algorithm is also adaptive to the situation with partial observations. It is shown via a stochastic parameterized model that the CEBoosting strategy can be combined with data assimilation to identify regime switching triggered by the unobserved latent processes. In addition, the CEBoosting method is applied to a nonlinear paradigm model for topographic mean flow interaction, demonstrating the online detection of regime switching in the presence of strong intermittency and extreme events.

Model-free forecasting of partially observable spatiotemporally chaotic systems

Reservoir computing is a powerful tool for forecasting turbulence because its simple architecture has the computational efficiency to handle high-dimensional systems. Its implementation, however, often requires full state-vector measurements and knowledge of the system nonlinearities. We use nonlinear projector functions to expand the system measurements to a high dimensional space and then feed them to a reservoir to obtain forecasts. We demonstrate the application of such reservoir computing networks on spatiotemporally chaotic systems, which model several features of turbulence. We show that using radial basis functions as nonlinear projectors enables complex system nonlinearities to be captured robustly even with only partial observations and without knowing the governing equations. Finally, we show that when measurements are sparse or incomplete and noisy, such that even the governing equations become inaccurate, our networks can still produce reasonably accurate forecasts, thus paving the way towards model-free forecasting of practical turbulent systems.

A random batch method for efficient ensemble forecasts of multiscale turbulent systems

A new efficient ensemble prediction strategy is developed for a multiscale turbulent model framework with emphasis on the nonlinear interactions between large and small-scale variables. The high computational cost in running large ensemble simulations of high-dimensional equations is effectively avoided by adopting a random batch decomposition of the wide spectrum of the fluctuation states, which is a characteristic feature of the multiscale turbulent systems. The time update of each ensemble sample is then only subject to a small portion of the small-scale fluctuation modes in one batch, while the true model dynamics with multiscale coupling is respected by frequent random resampling of the batches at each time updating step. We investigate both theoretical and numerical properties of the proposed method. First, the convergence of statistical errors in the random batch model approximation is shown rigorously independent of the sample size and full dimension of the system. Next, the forecast skill of the computational algorithm is tested on two representative models of turbulent flows exhibiting many key statistical phenomena with a direct link to realistic turbulent systems. The random batch method displays robust performance in capturing a series of crucial statistical features with general interests, including highly non-Gaussian fat-tailed probability distributions and intermittent bursts of instability, while requires a much lower computational cost than the direct ensemble approach. The efficient random batch method also facilitates the development of new strategies in uncertainty quantification and data assimilation for a wide variety of general complex turbulent systems in science and engineering.

Conditional Gaussian nonlinear system: A fast preconditioner and a cheap surrogate model for complex nonlinear systems

Developing suitable approximate models for analyzing and simulating complex nonlinear systems is practically important. This paper aims at exploring the skill of a rich class of nonlinear stochastic models, known as the conditional Gaussian nonlinear system (CGNS), as both a cheap surrogate model and a fast preconditioner for facilitating many computationally challenging tasks. The CGNS preserves the underlying physics to a large extent and can reproduce intermittency, extreme events, and other non-Gaussian features in many complex systems arising from practical applications. Three interrelated topics are studied. First, the closed analytic formulas of solving the conditional statistics provide an efficient and accurate data assimilation scheme. It is shown that the data assimilation skill of a suitable CGNS approximate forecast model outweighs that by applying an ensemble method even to the perfect model with strong nonlinearity, where the latter suffers from filter divergence. Second, the CGNS allows the development of a fast algorithm for simultaneously estimating the parameters and the unobserved variables with uncertainty quantification in the presence of only partial observations. Utilizing an appropriate CGNS as a preconditioner significantly reduces the computational cost in accurately estimating the parameters in the original complex system. Finally, the CGNS advances rapid and statistically accurate algorithms for computing the probability density function and sampling the trajectories of the unobserved state variables. These fast algorithms facilitate the development of an efficient and accurate data-driven method for predicting the linear response of the original system with respect to parameter perturbations based on a suitable CGNS preconditioner.

Shock trace prediction by reduced models for a viscous stochastic Burgers equation

Viscous shocks are a particular type of extreme event in nonlinear multiscale systems, and their representation requires small scales. Model reduction can thus play an essential role in reducing the computational cost for the prediction of shocks. Yet, reduced models typically aim to approximate large-scale dominating dynamics, which do not resolve the small scales by design. To resolve this representation barrier, we introduce a new qualitative characterization of the space-time locations of shocks, named the "shock trace," via a space-time indicator function based on an empirical resolution-adaptive threshold. Unlike exact shocks, the shock traces can be captured within the representation capacity of the large scales, thus facilitating the forecast of the timing and locations of the shocks utilizing reduced models. Within the context of a viscous stochastic Burgers equation, we show that a data-driven reduced model, in the form of nonlinear autoregression (NAR) time series models, can accurately predict the random shock traces, with relatively low rates of false predictions. Furthermore, the NAR model, which includes nonlinear closure terms to approximate the feedback from the small scales, significantly outperforms the corresponding Galerkin truncated model in the scenario of either noiseless or noisy observations. The results illustrate the importance of the data-driven closure terms in the NAR model, which account for the effects of the unresolved dynamics brought by nonlinear interactions.

BAMCAFE: A Bayesian machine learning advanced forecast ensemble method for complex turbulent systems with partial observations

Ensemble forecast based on physics-informed models is one of the most widely used forecast algorithms for complex turbulent systems. A major difficulty in such a method is the model error that is ubiquitous in practice. Data-driven machine learning (ML) forecasts can reduce the model error, but they often suffer from partial and noisy observations. In this article, a simple but effective Bayesian machine learning advanced forecast ensemble (BAMCAFE) method is developed, which combines an available imperfect physics-informed model with data assimilation (DA) to facilitate the ML ensemble forecast. In the BAMCAFE framework, a Bayesian ensemble DA is applied to create the training data of the ML model, which reduces the intrinsic error in the imperfect physics-informed model simulations and provides the training data of the unobserved variables. Then a generalized DA is employed for the initialization of the ML ensemble forecast. In addition to forecasting the optimal point-wise value, the BAMCAFE also provides an effective approach of quantifying the forecast uncertainty utilizing a non-Gaussian probability density function that characterizes the intermittency and extreme events. It is shown using a two-layer Lorenz 96 model that the BAMCAFE method can significantly improve the forecasting skill compared to the typical reduced-order imperfect models with bare truncation or stochastic parameterization for both the observed and unobserved large-scale variables. It is also shown via a nonlinear conceptual model that the BAMCAFE leads to a comparable non-Gaussian forecast uncertainty as the perfect model while the associated imperfect physics-informed model suffers from large forecast biases.

An efficient continuous data assimilation algorithm for the Sabra shell model of turbulence

Complex nonlinear turbulent dynamical systems are ubiquitous in many areas of research. Recovering unobserved state variables is an important topic for the data assimilation of turbulent systems. In this article, an efficient continuous-in-time data assimilation scheme is developed, which exploits closed analytic formulas for updating the unobserved state variables. Therefore, it is computationally efficient and accurate. The new data assimilation scheme is combined with a simple reduced order modeling technique that involves a cheap closure approximation and noise inflation. In such a way, many complicated turbulent dynamical systems can satisfy the requirements of the mathematical structures for the proposed efficient data assimilation scheme. The new data assimilation scheme is then applied to the Sabra shell model, which is a conceptual model for nonlinear turbulence. The goal is to recover the unobserved shell velocities across different spatial scales. It is shown that the new data assimilation scheme is skillful in capturing the nonlinear features of turbulence including the intermittency and extreme events in both the chaotic and the turbulent dynamical regimes. It is also shown that the new data assimilation scheme is more accurate and computationally cheaper than the standard ensemble Kalman filter and nudging data assimilation schemes for assimilating the Sabra shell model with partial observations.

Feasibility of Laser Communication Beacon Light Compressed Sensing

The Compressed Sensing (CS) camera can compress images in real time without consuming computing resources. Applying CS theory in the Laser Communication (LC) system can minimize the assumed transmission bandwidth (normally from a satellite to a ground station) and minimize the storage costs of beacon light-spot images; this can save more than ten times the typical bandwidth or storage space. However, the CS compressive process affects the light-spot tracking and key parameters in the images. In this study, we quantitatively explored the feasibility of the CS technique to capture light-spots in LC systems. We redesigned the measurement matrix to adapt to the requirement of light-tracking. We established a succinct structured deep network, the Compressed Sensing Denoising Center Net (CSD-Center Net) for denoising tracking computation from compressed image information. A series of simulations was made to test the performance of information preservation in beacon light spot image storage. With the consideration of CS ratio and application scenarios, coupled with CSD-Center Net and standard centroid, CS can achieve the tracking function well. The information preserved in compressed information correlates with the CS ratio; higher CS ratio can preserve more details. In fact, when the data rate is up than 10%, the accuracy could meet the requirements what we need in most application scenarios.

Lagrangian Reduced Order Modeling Using Finite Time Lyapunov Exponents

There are two main strategies for improving the projection-based reduced order model (ROM) accuracy—(i) improving the ROM, that is, adding new terms to the standard ROM; and (ii) improving the ROM basis, that is, constructing ROM bases that yield more accurate ROMs. In this paper, we use the latter. We propose two new Lagrangian inner products that we use together with Eulerian and Lagrangian data to construct two new Lagrangian ROMs, which we denote α-ROM and λ-ROM. We show that both Lagrangian ROMs are more accurate than the standard Eulerian ROMs, that is, ROMs that use standard Eulerian inner product and data to construct the ROM basis. Specifically, for the quasi-geostrophic equations, we show that the new Lagrangian ROMs are more accurate than the standard Eulerian ROMs in approximating not only Lagrangian fields (e.g., the finite time Lyapunov exponent (FTLE)), but also Eulerian fields (e.g., the streamfunction). In particular, the α-ROM can be orders of magnitude more accurate than the standard Eulerian ROMs. We emphasize that the new Lagrangian ROMs do not employ any closure modeling to model the effect of discarded modes (which is standard procedure for low-dimensional ROMs of complex nonlinear systems). Thus, the dramatic increase in the new Lagrangian ROMs’ accuracy is entirely due to the novel Lagrangian inner products used to build the Lagrangian ROM basis.

Generalized Nonlinear Least Squares Method for the Calibration of Complex Computer Code Using a Gaussian Process Surrogate

The approximated nonlinear least squares (ALS) method has been used for the estimation of unknown parameters in the complex computer code which is very time-consuming to execute. The ALS calibrates or tunes the computer code by minimizing the squared difference between real observations and computer output using a surrogate such as a Gaussian process model. When the differences (residuals) are correlated or heteroscedastic, the ALS may result in a distorted code tuning with a large variance of estimation. Another potential drawback of the ALS is that it does not take into account the uncertainty in the approximation of the computer model by a surrogate. To address these problems, we propose a generalized ALS (GALS) by constructing the covariance matrix of residuals. The inverse of the covariance matrix is multiplied to the residuals, and it is minimized with respect to the tuning parameters. In addition, we consider an iterative version for the GALS, which is called as the max-minG algorithm. In this algorithm, the parameters are re-estimated and updated by the maximum likelihood estimation and the GALS, by using both computer and experimental data repeatedly until convergence. Moreover, the iteratively re-weighted ALS method (IRWALS) was considered for a comparison purpose. Five test functions in different conditions are examined for a comparative analysis of the four methods. Based on the test function study, we find that both the bias and variance of estimates obtained from the proposed methods (the GALS and the max-minG) are smaller than those from the ALS and the IRWALS methods. Especially, the max-minG works better than others including the GALS for the relatively complex test functions. Lastly, an application to a nuclear fusion simulator is illustrated and it is shown that the abnormal pattern of residuals in the ALS can be resolved by the proposed methods.

Predicting observed and hidden extreme events in complex nonlinear dynamical systems with partial observations and short training time series

Extreme events appear in many complex nonlinear dynamical systems. Predicting extreme events has important scientific significance and large societal impacts. In this paper, a new mathematical framework of building suitable nonlinear approximate models is developed, which aims at predicting both the observed and hidden extreme events in complex nonlinear dynamical systems for short-, medium-, and long-range forecasting using only short and partially observed training time series. Different from many ad hoc data-driven regression models, these new nonlinear models take into account physically motivated processes and physics constraints. They also allow efficient and accurate algorithms for parameter estimation, data assimilation, and prediction. Cheap stochastic parameterizations, judicious linear feedback control, and suitable noise inflation strategies are incorporated into the new nonlinear modeling framework, which provide accurate predictions of both the observed and hidden extreme events as well as the strongly non-Gaussian statistics in various highly intermittent nonlinear dyad and triad models, including the Lorenz 63 model. Then, a stochastic mode reduction strategy is applied to a 21-dimensional nonlinear paradigm model for topographic mean flow interaction. The resulting five-dimensional physics-constrained nonlinear approximate model is able to accurately predict extreme events and the regime switching between zonally blocked and unblocked flow patterns. Finally, incorporating judicious linear stochastic processes into a simple nonlinear approximate model succeeds in learning certain complicated nonlinear effects of a six-dimensional low-order Charney-DeVore model with strong chaotic and regime switching behavior. The simple nonlinear approximate model then allows accurate online state estimation and the short- and medium-range forecasting of extreme events.

Linear and nonlinear statistical response theories with prototype applications to sensitivity analysis and statistical control of complex turbulent dynamical systems

Statistical response theory provides an effective tool for the analysis and statistical prediction of high-dimensional complex turbulent systems involving a large number of unresolved unstable modes, for example, in climate change science. Recently, the linear and nonlinear response theories have shown promising developments in overcoming the curse-of-dimensionality in uncertain quantification and statistical control of turbulent systems by identifying the most sensitive response directions. We offer an extensive illustration of using the statistical response theory for a wide variety of challenging problems under a hierarchy of prototype models ranging from simple solvable equations to anisotropic geophysical turbulence. Directly applying the linear response operator for statistical responses is shown to only have limited skill for small perturbation ranges. For stronger nonlinearity and perturbations, a nonlinear reduced-order statistical model reduction strategy guaranteeing model fidelity and sensitivity provides a systematic framework to recover the multiscale variability in leading order statistics. The linear response operator is applied in the training phase for the optimal nonlinear model responses requiring only the unperturbed equilibrium statistics. The statistical response theory is further applied to the statistical control of inherently high-dimensional systems. The statistical response in the mean offers an efficient way to recover the control forcing from the statistical energy equation without the need to run the expensive model. Among all the testing examples, the statistical response strategy displays uniform robust skill in various dynamical regimes with distinct statistical features. Further applications of the statistical response theory include the prediction of extreme events and intermittency in turbulent passive transport and a rigorous saturation bound governing the total statistical growth from initial and external uncertainties.

Understanding and Predicting Nonlinear Turbulent Dynamical Systems with Information Theory

Complex nonlinear turbulent dynamical systems are ubiquitous in many areas. Quantifying the model error and model uncertainty plays an important role in understanding and predicting complex dynamical systems. In the first part of this article, a simple information criterion is developed to assess the model error in imperfect models. This effective information criterion takes into account the information in both the equilibrium statistics and the temporal autocorrelation function, where the latter is written in the form of the spectrum density that permits the quantification via information theory. This information criterion facilitates the study of model reduction, stochastic parameterizations, and intermittent events. In the second part of this article, a new efficient method is developed to improve the computation of the linear response via the Fluctuation Dissipation Theorem (FDT). This new approach makes use of a Gaussian Mixture (GM) to describe the unperturbed probability density function in high dimensions and avoids utilizing Gaussian approximations in computing the statistical response, as is widely used in the quasi-Gaussian (qG) FDT. Testing examples show that this GM FDT outperforms qG FDT in various strong non-Gaussian regimes.

Statistical dynamical model to predict extreme events and anomalous features in shallow water waves with abrupt depth change

Understanding and predicting extreme events and their anomalous statistics in complex nonlinear systems are a grand challenge in climate, material, and neuroscience as well as for engineering design. Recent laboratory experiments in weakly turbulent shallow water reveal a remarkable transition from Gaussian to anomalous behavior as surface waves cross an abrupt depth change (ADC). Downstream of the ADC, probability density functions of surface displacement exhibit strong positive skewness accompanied by an elevated level of extreme events. Here, we develop a statistical dynamical model to explain and quantitatively predict the above anomalous statistical behavior as experimental control parameters are varied. The first step is to use incoming and outgoing truncated Korteweg-de Vries (TKdV) equations matched in time at the ADC. The TKdV equation is a Hamiltonian system, which induces incoming and outgoing statistical Gibbs invariant measures. The statistical matching of the known nearly Gaussian incoming Gibbs state at the ADC completely determines the predicted anomalous outgoing Gibbs state, which can be calculated by a simple sampling algorithm verified by direct numerical simulations, and successfully captures key features of the experiment. There is even an analytic formula for the anomalous outgoing skewness. The strategy here should be useful for predicting extreme anomalous statistical behavior in other dispersive media.

Conditional Gaussian Systems for Multiscale Nonlinear Stochastic Systems: Prediction, State Estimation and Uncertainty Quantification

A conditional Gaussian framework for understanding and predicting complex multiscale nonlinear stochastic systems is developed. Despite the conditional Gaussianity, such systems are nevertheless highly nonlinear and are able to capture the non-Gaussian features of nature. The special structure of the system allows closed analytical formulae for solving the conditional statistics and is thus computationally efficient. A rich gallery of examples of conditional Gaussian systems are illustrated here, which includes data-driven physics-constrained nonlinear stochastic models, stochastically coupled reaction–diffusion models in neuroscience and ecology, and large-scale dynamical models in turbulence, fluids and geophysical flows. Making use of the conditional Gaussian structure, efficient statistically accurate algorithms involving a novel hybrid strategy for different subspaces, a judicious block decomposition and statistical symmetry are developed for solving the Fokker–Planck equation in large dimensions. The conditional Gaussian framework is also applied to develop extremely cheap multiscale data assimilation schemes, such as the stochastic superparameterization, which use particle filters to capture the non-Gaussian statistics on the large-scale part whose dimension is small whereas the statistics of the small-scale part are conditional Gaussian given the large-scale part. Other topics of the conditional Gaussian systems studied here include designing new parameter estimation schemes and understanding model errors.