Neural Granger Causality

Explicit Representation and Customized Fault Isolation Framework for Learning Temporal and Spatial Dependencies in Industrial Processes

Typically, industrial processes possess both temporal and spatial dependencies due to intravariable dynamics and intervariable couplings. The two dependencies have different manifestations, indicating diverse process characteristics. However, the existing methods fail to separate temporal and spatial information well, leading to inappropriate representation and inaccurate fault detection and isolation results. This study proposes an explicit representation and customized fault isolation framework to tackle temporal and spatial characteristics, so as to identify and locate anomalies affecting different dependencies. First, we design a double-level separation method for temporal and spatial information. In the first level, we construct two independent auto-encoding modules to extract temporal correlation and spatial graph structure in parallel. In the second level, we propose an information aliasing loss function to guild the two modules to distinguish between temporal and spatial characteristics, further facilitating information separation. By monitoring the explicit temporal and spatial statistics obtained by the two modules, spatiotemporal dependencies of anomalies can be determined for subsequent isolation. Furthermore, we propose a customized isolation strategy for anomalies in temporal and spatial characteristics. By quantifying changes in intravariable temporal dynamics and intervariable spatial graph structure individually, temporal impact and spatial propagation of faults can be finely characterized and isolated. Three examples are adopted to verify the performance of the proposed framework, including a numerical example, a real condensing system of the thermal power plant process, and the Tennessee Eastman benchmark process.

The causality measure of partial mutual information from mixed embedding (PMIME) revisited

The measure of partial mutual information from mixed embedding (PMIME) is an information theory-based measure to accurately identify the direct and directional coupling, termed Granger causality or simply causality, between the observed variables or subsystems of a high-dimensional dynamical and complex system, without any a priori assumptions about the nature of the coupling relationship. In its core, it is a forward selection procedure that aims to iteratively identify the lag-dependence structure of a given observed variable (response) to all the other observed variables (candidate drivers). This model-free approach is capable of detecting nonlinear interactions, abundantly present in real-world complex systems, and it was shown to perform well on multivariate time series of moderately high dimension. However, the PMIME presents some inefficiencies in its performance mainly when applied on strongly stochastic (linear or nonlinear) systems as it may falsely detect non-existent relationships. Moreover, and by construction, the measure cannot extract purely synergetic relationships present in a system. In the current work, the issue of false detections is addressed by introducing an improved resampling significance test and a procedure of rechecking the identified drivers (backward revision). Regarding the inability to detect synergetic relationships, the PMIME is further enhanced by checking pairs as candidate drivers for the response variable after having considered all drivers individually. The effects of these modifications are investigated in a systematic simulation study on properly designed systems involving strong stochasticity, regressor terms with synergetic effects, and a system dimension ranging from 3 to 30. The overall results of the simulations indicate that these modifications indeed improve the performance of PMIME and alleviate to a significant degree the issues of the original algorithm. Guidelines for balancing between accuracy and computational efficiency are also given, particularly relevant for real-world applications. Finally, the measure performance is investigated in the study of futures of various government bonds and stock market indices in the period around COVID-19 pandemic.

Nonlinear causal network learning via Granger causality based on extreme support vector regression

For complex networked systems, based on the consideration of nonlinearity and causality, a novel general method of nonlinear causal network learning, termed extreme support vector regression Granger causality (ESVRGC), is proposed. The nonuniform time-delayed influence of the driving nodes on the target node is particularly considered. Then, the restricted model and the unrestricted model of Granger causality are, respectively, formulated based on extreme support vector regression, which uses the selected time-delayed components of system variables as the inputs of kernel functions. The nonlinear conditional Granger causality index is finally calculated to confirm the strength of a causal interaction. Generally, based on the simulation of a nonlinear vector autoregressive model and nonlinear discrete time-delayed dynamic systems, ESVRGC demonstrates better performance than other popular methods. Also, the validity and robustness of ESVRGC are also verified by the different cases of network types, sample sizes, noise intensities, and coupling strengths. Finally, the superiority of ESVRGC is successful verified by the experimental study on real benchmark datasets.

A historical perspective of biomedical explainable AI research

The black-box nature of most artificial intelligence (AI) models encourages the development of explainability methods to engender trust into the AI decision-making process. Such methods can be broadly categorized into two main types: post hoc explanations and inherently interpretable algorithms. We aimed at analyzing the possible associations between COVID-19 and the push of explainable AI (XAI) to the forefront of biomedical research. We automatically extracted from the PubMed database biomedical XAI studies related to concepts of causality or explainability and manually labeled 1,603 papers with respect to XAI categories. To compare the trends pre- and post-COVID-19, we fit a change point detection model and evaluated significant changes in publication rates. We show that the advent of COVID-19 in the beginning of 2020 could be the driving factor behind an increased focus concerning XAI, playing a crucial role in accelerating an already evolving trend. Finally, we present a discussion with future societal use and impact of XAI technologies and potential future directions for those who pursue fostering clinical trust with interpretable machine learning models.

Identification of interacting neural populations: methods and statistical considerations

As improved recording technologies have created new opportunities for neurophysiological investigation, emphasis has shifted from individual neurons to multiple populations that form circuits, and it has become important to provide evidence of cross-population coordinated activity. We review various methods for doing so, placing them in six major categories while avoiding technical descriptions and instead focusing on high-level motivations and concerns. Our aim is to indicate what the methods can achieve and the circumstances under which they are likely to succeed. Toward this end, we include a discussion of four cross-cutting issues: the definition of neural populations, trial-to-trial variability and Poisson-like noise, time-varying dynamics, and causality.

Time series causal relationships discovery through feature importance and ensemble models

Inferring causal relationships from observational data is a key challenge in understanding the interpretability of Machine Learning models. Given the ever-increasing amount of observational data available in many areas, Machine Learning algorithms used for forecasting have become more complex, leading to a less understandable path of how a decision is made by the model. To address this issue, we propose leveraging ensemble models, e.g., Random Forest, to assess which input features the trained model prioritizes when making a forecast and, in this way, establish causal relationships between the variables. The advantage of these algorithms lies in their ability to provide feature importance, which allows us to build the causal network. We present our methodology to estimate causality in time series from oil field production. As it is difficult to extract causal relations from a real field, we also included a synthetic oil production dataset and a weather dataset, which is also synthetic, to provide the ground truth. We aim to perform causal discovery, i.e., establish the existing connections between the variables in each dataset. Through an iterative process of improving the forecasting of a target's value, we evaluate whether the forecasting improves by adding information from a new potential driver; if so, we state that the driver causally affects the target. On the oil field-related datasets, our causal analysis results agree with the interwell connections already confirmed by tracer information; whenever the tracer data are available, we used it as our ground truth. This consistency between both estimated and confirmed connections provides us the confidence about the effectiveness of our proposed methodology. To our knowledge, this is the first time causal analysis using solely production data is employed to discover interwell connections in an oil field dataset.

MPGE and RootRank: A sufficient root cause characterization and quantification framework for industrial process faults

Root cause diagnosis can locate abnormalities of industrial processes, ensuring production safety and manufacturing efficiency. However, existing root cause diagnosis models only consider pairwise direct causality and ignore the multi-level fault propagation, which may lead to incomplete root cause descriptions and ambiguous root cause candidates. To address the above issue, a novel framework, named multi-level predictive graph extraction (MPGE) and RootRank scoring, is proposed and applied to the root cause diagnosis for industrial processes. In this framework, both direct and indirect Granger causalities are characterized by multi-level predictive relationships to provide a sufficient characterization of root cause variables. First, a predictive graph structure with a sparse constrained adjacency matrix is constructed to describe the information transmission between variables. The information of variables is deeply fused according to the adjacency matrix to consider multi-level fault propagation. Then, a hierarchical adjacency pruning (HAP) mechanism is designed to automatically capture vital predictive relationships through adjacency redistribution. In this way, the multi-level causalities between variables are extracted to fully describe both direct and indirect fault propagation and highlight the root cause. Further, a RootRank scoring algorithm is proposed to analyze the predictive graph and quantify the fault propagation contribution of each variable, thereby giving definite root cause identification results. Three examples are adopted to verify the diagnostic performance of the proposed framework, including a numerical example, the Tennessee Eastman benchmark process, and a real cut-made process of cigarette. Both theoretical analysis and experimental verification show the high interpretability and reliability of the proposed framework.

Granger causality using Jacobian in neural networks

Granger causality is a commonly used method for uncovering information flow and dependencies in a time series. Here, we introduce JGC (Jacobian Granger causality), a neural network-based approach to Granger causality using the Jacobian as a measure of variable importance, and propose a variable selection procedure for inferring Granger causal variables with this measure, using criteria of significance and consistency. The resulting approach performs consistently well compared to other approaches in identifying Granger causal variables, the associated time lags, as well as interaction signs. In addition, we also discuss the need for contemporaneous variables in Granger causal modeling as well as how these neural network-based approaches reduce the impact of nonseparability in dynamical systems, a problem where predictive information on a target variable is not unique to its causes, but also contained in the history of the target variable itself.

Which return regime induces overconfidence behavior? Artificial intelligence and a nonlinear approach

Overconfidence behavior, one form of positive illusion, has drawn considerable attention throughout history because it is viewed as the main reason for many crises. Investors' overconfidence, which can be observed as overtrading following positive returns, may lead to inefficiencies in stock markets. To the best of our knowledge, this is the first study to examine the presence of investor overconfidence by employing an artificial intelligence technique and a nonlinear approach to impulse responses to analyze the impact of different return regimes on the overconfidence attitude. We examine whether investors in an emerging stock market (Borsa Istanbul) exhibit overconfidence behavior using a feed-forward, neural network, nonlinear Granger causality test and nonlinear impulse-response functions based on local projections. These are the first applications in the relevant literature due to the novelty of these models in forecasting high-dimensional, multivariate time series. The results obtained from distinguishing between the different market regimes to analyze the responses of trading volume to return shocks contradict those in the literature, which is the key contribution of the study. The empirical findings imply that overconfidence behavior exhibits asymmetries in different return regimes and is persistent during the 20-day forecasting horizon. Overconfidence is more persistent in the low- than in the high-return regime. In the negative interest-rate period, a high-return regime induces overconfidence behavior, whereas in the positive interest-rate period, a low-return regime induces overconfidence behavior. Based on the empirical findings, investors should be aware that portfolio gains may result in losses depending on aggressive and excessive trading strategies, particularly in low-return regimes.

Altered nonlinear Granger causality interactions in the large-scale brain networks of patients with schizophrenia

Objective.It has been demonstrated that schizophrenia (SZ) is characterized by functional dysconnectivity involving extensive brain networks. However, the majority of previous studies utilizing resting-state functional magnetic resonance imaging (fMRI) to infer abnormal functional connectivity (FC) in patients with SZ have focused on the linear correlation that one brain region may influence another, ignoring the inherently nonlinear properties of fMRI signals.Approach. In this paper, we present a neural Granger causality (NGC) technique for examining the changes in SZ's nonlinear causal couplings. We develop static and dynamic NGC-based analyses of large-scale brain networks at several network levels, estimating complicated temporal and causal relationships in SZ patients.Main results. We find that the NGC-based FC matrices can detect large and significant differences between the SZ and healthy control groups at both the regional and subnetwork scales. These differences are persistent and significantly overlapped at various network sparsities regardless of whether the brain networks were built using static or dynamic techniques. In addition, compared to controls, patients with SZ exhibited extensive NGC confusion patterns throughout the entire brain.Significance. These findings imply that the NGC-based FCs may be a useful method for quantifying the abnormalities in the causal influences of patients with SZ, hence shedding fresh light on the pathophysiology of this disorder.

Neural Information Squeezer for Causal Emergence

Conventional studies of causal emergence have revealed that stronger causality can be obtained on the macro-level than the micro-level of the same Markovian dynamical systems if an appropriate coarse-graining strategy has been conducted on the micro-states. However, identifying this emergent causality from data is still a difficult problem that has not been solved because the appropriate coarse-graining strategy can not be found easily. This paper proposes a general machine learning framework called Neural Information Squeezer to automatically extract the effective coarse-graining strategy and the macro-level dynamics, as well as identify causal emergence directly from time series data. By using invertible neural network, we can decompose any coarse-graining strategy into two separate procedures: information conversion and information discarding. In this way, we can not only exactly control the width of the information channel, but also can derive some important properties analytically. We also show how our framework can extract the coarse-graining functions and the dynamics on different levels, as well as identify causal emergence from the data on several exampled systems.

CGCNImp: a causal graph convolutional network for multivariate time series imputation

BACKGROUND

Multivariate time series data generally contains missing values, which can be an obstacle to subsequent analysis and may compromise downstream applications. One challenge in this endeavor is the presence of the missing values brought about by sensor failure and transmission packet loss. Imputation is the usual remedy in such circumstances. However, in some multivariate time series data, the complex correlation and temporal dependencies, coupled with the non-stationarity of the data, make imputation difficult.

MEHODS

To address this problem, we propose a novel model for multivariate time series imputation called CGCNImp that considers both correlation and temporal dependency modeling. The correlation dependency module leverages neural Granger causality and a GCN to capture the correlation dependencies among different attributes of the time series data, while the temporal dependency module relies on an attention-driven long short term memory (LSTM) and a time lag matrix to learn its dependencies. Missing values and noise are addressed with total variation reconstruction.

RESULTS

We conduct thorough empirical analyses on two real-world datasets. Imputation results show that CGCNImp achieves state-of-the-art performance when compared to previous methods.

Granger causality test with nonlinear neural-network-based methods: Python package and simulation study

BACKGROUND AND OBJECTIVE

Causality defined by Granger in 1969 is a widely used concept, particularly in neuroscience and economics. As there is an increasing interest in nonlinear causality research, a Python package with a neural-network-based causality analysis approach was created. It allows performing causality tests using neural networks based on Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or Multilayer Perceptron (MLP). The aim of this paper is to present the nonlinear method for causality analysis and the created Python package.

METHODS

The created functions with the autoregressive (AR) and Generalized Radial Basis Functions (GRBF) neural network models were tested on simulated signals in two cases: with nonlinear dependency and with absence of causality from Y to X signal. The train-test split (70/30) was used. Errors obtained on the test set were compared using the Wilcoxon signed-rank test to determine the presence of the causality. For the chosen model, the proposed method of study the change of causality over time was presented.

RESULTS

In the case when X was a polynomial of Y, nonlinear methods were able to detect the causality, while the AR model did not manage to indicate it. The best results (in terms of the prediction accuracy) were obtained for the MLP for the lag of 150 (MSE equal to 0.011, compared to 0.041 and 0.036 for AR and GRBF, respectively). When there was no causality between the signals, none of the proposed and AR models did indicate false causality, while it was detected by GRBF models in one case. Only the proposed models gave the expected results in each of the tested scenarios.

CONCLUSIONS

The proposed method appeared to be superior to the compared methods. They were able to detect non-linear causality, make accurate forecasting and not indicate false causality. The created package enables easy usage of neural networks to study the causal relationship between signals. The neural-networks-based approach is a suitable method that allows the detection of a nonlinear causal relationship, which cannot be detected by the classical Granger method. Unlike other similar tools, the package allows for the study of changes in causality over time.

How do mobility restrictions and social distancing during COVID-19 affect oil price?

This paper provides an analysis of the effect of the COVID-19 outbreak on the crude oil price. Using a newly developed air mobility index and Apple's driving trends index, we assess the effect of human mobility restrictions and social distancing during the COVID-19 pandemic on the crude oil price. We apply a quantile regression model, which evaluates different quantiles of the crude oil price. We also conduct an extreme value modeling, which examines the lower tail of the crude oil price distribution. We find that both the air mobility index and driving trends index significantly influence lower and upper quantiles of the WTI crude oil price. The extreme value models suggest that there is a potential risk of a negative crude oil price for a sudden extreme fall of air mobility.

Neural Networks for Directed Connectivity Estimation in Source-Reconstructed EEG Data

Directed connectivity between brain sources identified from scalp electroencephalography (EEG) can shed light on the brain’s information flows and provide a biomarker of neurological disorders. However, as volume conductance results in scalp activity being a mix of activities originating from multiple sources, the correct interpretation of their connectivity is a formidable challenge despite source localization being applied with some success. Traditional connectivity approaches rely on statistical assumptions that usually do not hold for EEG, calling for a model-free approach. We investigated several types of Artificial Neural Networks in estimating Directed Connectivity between Reconstructed EEG Sources and assessed their accuracy with respect to several ground truths. We show that a Long Short-Term Memory neural network with Non-Uniform Embedding yields the most promising results due to its relative robustness to differing dipole locations. We conclude that certain network architectures can compete with the already established methods for brain connectivity analysis.

Granger Causality: A Review and Recent Advances

Introduced more than a half-century ago, Granger causality has become a popular tool for analyzing time series data in many application domains, from economics and finance to genomics and neuroscience. Despite this popularity, the validity of this framework for inferring causal relationships among time series has remained the topic of continuous debate. Moreover, while the original definition was general, limitations in computational tools have constrained the applications of Granger causality to primarily simple bivariate vector autoregressive processes. Starting with a review of early developments and debates, this article discusses recent advances that address various shortcomings of the earlier approaches, from models for high-dimensional time series to more recent developments that account for nonlinear and non-Gaussian observations and allow for subsampled and mixed-frequency time series.

Impacts of COVID-19 local spread and Google search trend on the US stock market

We develop a novel temporal complex network approach to quantify the US county level spread dynamics of COVID-19. We use both conventional econometric and Machine Learning (ML) models that incorporate the local spread dynamics, COVID-19 cases and death, and Google search activities to assess if incorporating information about local spreads improves the predictive accuracy of models for the US stock market. The results suggest that COVID-19 cases and deaths, its local spread, and Google searches have impacts on abnormal stock prices between January 2020 to May 2020. Furthermore, incorporating information about local spread significantly improves the performance of forecasting models of the abnormal stock prices at longer forecasting horizons.