Inferring the directionality of coupling with conditional mutual information

Identifying key drivers in a stochastic dynamical system through estimation of transfer entropy between univariate and multivariate time series

Transfer entropy (TE) is a widely used tool for quantifying causal relationships in stochastic dynamical systems. Traditionally, TE and its conditional variants are applied pairwise between dynamic variables to infer these relationships. However, identifying key drivers in such systems requires a measure of the causal influence exerted by each component on the entire system. I propose using outgoing transfer entropy (OutTE), the transfer entropy from a given variable to the collection of remaining variables, to quantify the causal influence of the variable on the rest of the system. Conversely, the incoming transfer entropy (InTE) is also defined to quantify the causal influence received by a component from the rest of the system. Since OutTE and InTE involve transfer entropy between univariate and multivariate time series, naive estimation methods can result in significant errors, especially when the number of variables is large relative to the number of samples. To address this, I introduce a novel estimation scheme that computes outgoing and incoming TE only between significantly interacting partners. The feasibility and effectiveness of this approach are demonstrated using synthetic data and real oral microbiota data. The method successfully identifies the bacterial species known to be key players in the bacterial community, highlighting its potential for uncovering causal drivers in complex systems.

Physiological signal analysis and open science using the Julia language and associated software

In this mini review, we propose the use of the Julia programming language and its software as a strong candidate for reproducible, efficient, and sustainable physiological signal analysis. First, we highlight available software and Julia communities that provide top-of-the-class algorithms for all aspects of physiological signal processing despite the language's relatively young age. Julia can significantly accelerate both research and software development due to its high-level interactive language and high-performance code generation. It is also particularly suited for open and reproducible science. Openness is supported and welcomed because the overwhelming majority of Julia software programs are open source and developed openly on public platforms, primarily through individual contributions. Such an environment increases the likelihood that an individual not (originally) associated with a software program would still be willing to contribute their code, further promoting code sharing and reuse. On the other hand, Julia's exceptionally strong package manager and surrounding ecosystem make it easy to create self-contained, reproducible projects that can be instantly installed and run, irrespective of processor architecture or operating system.

Linear and nonlinear causality in financial markets

Identifying and quantifying co-dependence between financial instruments is a key challenge for researchers and practitioners in the financial industry. Linear measures such as the Pearson correlation are still widely used today, although their limited explanatory power is well known. In this paper, we present a much more general framework for assessing co-dependencies by identifying linear and nonlinear causalities in the complex system of financial markets. To do so, we use two different causal inference methods, transfer entropy and convergent cross-mapping, and employ Fourier transform surrogates to separate their linear and nonlinear contributions. We find that stock indices in Germany and the U.S. exhibit a significant degree of nonlinear causality and that correlation, while a very good proxy for linear causality, disregards nonlinear effects and hence underestimates causality itself. The presented framework enables the measurement of nonlinear causality, the correlation-causality fallacy, and motivates how causality can be used for inferring market signals, pair trading, and risk management of portfolios. Our results suggest that linear and nonlinear causality can be used as early warning indicators of abnormal market behavior, allowing for better trading strategies and risk management.

Inferring Dealer Networks in the Foreign Exchange Market Using Conditional Transfer Entropy: Analysis of a Central Bank Announcement

The foreign exchange (FX) market has evolved into a complex system where locally generated information percolates through the dealer network via high-frequency interactions. Information related to major events, such as economic announcements, spreads rapidly through this network, potentially inducing volatility, liquidity disruptions, and contagion effects across financial markets. Yet, research on the mechanics of information flows in the FX market is limited. In this paper, we introduce a novel approach employing conditional transfer entropy to construct networks of information flows. Leveraging a unique, high-resolution dataset of bid and ask prices, we investigate the impact of an announcement by the European Central Bank on the information transfer within the market. During the announcement, we identify key dealers as information sources, conduits, and sinks, and, through comparison to a baseline, uncover shifts in the network topology.

Causal dynamics of sleep, circadian rhythm, and mood symptoms in patients with major depression and bipolar disorder: insights from longitudinal wearable device data

BACKGROUND

Sleep and circadian rhythm disruptions are common in patients with mood disorders. The intricate relationship between these disruptions and mood has been investigated, but their causal dynamics remain unknown.

METHODS

We analysed data from 139 patients (76 female, mean age = 23.5 ± 3.64 years) with mood disorders who participated in a prospective observational study in South Korea. The patients wore wearable devices to monitor sleep and engaged in smartphone-delivered ecological momentary assessment of mood symptoms. Using a mathematical model, we estimated their daily circadian phase based on sleep data. Subsequently, we obtained daily time series for sleep/circadian phase estimates and mood symptoms spanning >40,000 days. We analysed the causal relationship between the time series using transfer entropy, a non-linear causal inference method.

FINDINGS

The transfer entropy analysis suggested causality from circadian phase disturbance to mood symptoms in both patients with MDD (n = 45) and BD type I (n = 35), as 66.7% and 85.7% of the patients with a large dataset (>600 days) showed causality, but not in patients with BD type II (n = 59). Surprisingly, no causal relationship was suggested between sleep phase disturbances and mood symptoms.

INTERPRETATION

Our findings suggest that in patients with mood disorders, circadian phase disturbances directly precede mood symptoms. This underscores the potential of targeting circadian rhythms in digital medicine, such as sleep or light exposure interventions, to restore circadian phase and thereby manage mood disorders effectively.

FUNDING

Institute for Basic Science, the Human Frontiers Science Program Organization, the National Research Foundation of Korea, and the Ministry of Health & Welfare of South Korea.

Quantifying the efficacy of voltage protocols in characterising ion channel kinetics: A novel information-theoretic approach

Voltage-clamp experiments are commonly utilised to characterise cellular ion channel kinetics. In these experiments, cells are stimulated using a known time-varying voltage, referred to as the voltage protocol, and the resulting cellular response, typically in the form of current, is measured. Parameters of models that describe ion channel kinetics are then estimated by solving an inverse problem which aims to minimise the discrepancy between the predicted response of the model and the actual measured cell response. In this paper, a novel framework to evaluate the information content of voltage-clamp protocols in relation to ion channel model parameters is presented. Additional quantitative information metrics that allow for comparisons among various voltage protocols are proposed. These metrics offer a foundation for future optimal design frameworks to devise novel, information-rich protocols. The efficacy of the proposed framework is evidenced through the analysis of seven voltage protocols from the literature. By comparing known numerical results for inverse problems using these protocols with the information-theoretic metrics, the proposed approach is validated. The essential steps of the framework are: (i) generate random samples of the parameters from chosen prior distributions; (ii) run the model to generate model output (current) for all samples; (iii) construct reduced-dimensional representations of the time-varying current output using proper orthogonal decomposition (POD); (iv) estimate information-theoretic metrics such as mutual information, entropy equivalent variance, and conditional mutual information using non-parametric methods; (v) interpret the metrics; for example, a higher mutual information between a parameter and the current output suggests the protocol yields greater information about that parameter, resulting in improved identifiability; and (vi) integrate the information-theoretic metrics into a single quantitative criterion, encapsulating the protocol's efficacy in estimating model parameters.

Information transfers and flows in Markov chains as dynamical causal effects

A logical sequence of information-theoretic quantifiers of directional (causal) couplings in Markov chains is generated within the framework of dynamical causal effects (DCEs), starting from the simplest DCEs (in terms of localization of their functional elements) and proceeding step-by-step to more complex ones. Thereby, a system of 11 quantifiers is readily obtained, some of them coinciding with previously known causality measures widely used in time series analysis and often called "information transfers" or "flows" (transfer entropy, Ay-Polani information flow, Liang-Kleeman information flow, information response, etc.,) By construction, this step-by-step generation reveals logical relationships between all these quantifiers as specific DCEs. As a further concretization, diverse quantitative relationships between the transfer entropy and the Liang-Kleeman information flow are found both rigorously and numerically for coupled two-state Markov chains.

Connectivity Analysis in EEG Data: A Tutorial Review of the State of the Art and Emerging Trends

Understanding how different areas of the human brain communicate with each other is a crucial issue in neuroscience. The concepts of structural, functional and effective connectivity have been widely exploited to describe the human connectome, consisting of brain networks, their structural connections and functional interactions. Despite high-spatial-resolution imaging techniques such as functional magnetic resonance imaging (fMRI) being widely used to map this complex network of multiple interactions, electroencephalographic (EEG) recordings claim high temporal resolution and are thus perfectly suitable to describe either spatially distributed and temporally dynamic patterns of neural activation and connectivity. In this work, we provide a technical account and a categorization of the most-used data-driven approaches to assess brain-functional connectivity, intended as the study of the statistical dependencies between the recorded EEG signals. Different pairwise and multivariate, as well as directed and non-directed connectivity metrics are discussed with a pros-cons approach, in the time, frequency, and information-theoretic domains. The establishment of conceptual and mathematical relationships between metrics from these three frameworks, and the discussion of novel methodological approaches, will allow the reader to go deep into the problem of inferring functional connectivity in complex networks. Furthermore, emerging trends for the description of extended forms of connectivity (e.g., high-order interactions) are also discussed, along with graph-theory tools exploring the topological properties of the network of connections provided by the proposed metrics. Applications to EEG data are reviewed. In addition, the importance of source localization, and the impacts of signal acquisition and pre-processing techniques (e.g., filtering, source localization, and artifact rejection) on the connectivity estimates are recognized and discussed. By going through this review, the reader could delve deeply into the entire process of EEG pre-processing and analysis for the study of brain functional connectivity and learning, thereby exploiting novel methodologies and approaches to the problem of inferring connectivity within complex networks.

Improving the Process of Early-Warning Detection and Identifying the Most Affected Markets: Evidence from Subprime Mortgage Crisis and COVID-19 Outbreak-Application to American Stock Markets

Stock-market-crash predictability is of particular interest in the field of financial time-series analysis. Famous examples of major stock-market crashes are the real-estate bubble in 2008 and COVID-19 in 2020. Several studies have studied the prediction process without taking into consideration which markets might be falling into a crisis. To this end, a combination analysis is utilized in this manuscript. Firstly, the auto-regressive estimation (ARE) algorithm is successfully applied to electroencephalography (EEG) brain data for detecting diseases. The ARE algorithm is employed based on state-space modelling, which applies the expectation-maximization algorithm and Kalman filter. This manuscript introduces its application, for the first time, to stock-market data. For this purpose, a time-evolving interaction surface is constructed to observe the change in the surface topology. This enables tracking of the stock market's behavior over time and differentiates between different states. This provides a deep understanding of the underlying system behavior before, during, and after a crisis. Different patterns of the stock-market movements are recognized, providing novel information regarding detecting an early-warning sign. Secondly, a Granger-causality time-domain technique, called directed partial correlation, is employed to infer the underlying interconnectivity structure among markets. This information is crucial for investors and market players, enabling them to differentiate between those markets which will fall in a catastrophic loss, and those which will not. Consequently, they can make successful decisions towards selecting less risky portfolios, which guarantees lower losses. The results showed the effectiveness of the use of this methodology in the framework of the process of early-warning detection.

A Conditional Mutual Information Estimator for Mixed Data and an Associated Conditional Independence Test

In this study, we focus on mixed data which are either observations of univariate random variables which can be quantitative or qualitative, or observations of multivariate random variables such that each variable can include both quantitative and qualitative components. We first propose a novel method, called CMIh, to estimate conditional mutual information taking advantages of the previously proposed approaches for qualitative and quantitative data. We then introduce a new local permutation test, called LocAT for local adaptive test, which is well adapted to mixed data. Our experiments illustrate the good behaviour of CMIh and LocAT, and show their respective abilities to accurately estimate conditional mutual information and to detect conditional (in)dependence for mixed data.

Causal Inference in Time Series in Terms of Rényi Transfer Entropy

Uncovering causal interdependencies from observational data is one of the great challenges of a nonlinear time series analysis. In this paper, we discuss this topic with the help of an information-theoretic concept known as Rényi’s information measure. In particular, we tackle the directional information flow between bivariate time series in terms of Rényi’s transfer entropy. We show that by choosing Rényi’s parameter α, we can appropriately control information that is transferred only between selected parts of the underlying distributions. This, in turn, is a particularly potent tool for quantifying causal interdependencies in time series, where the knowledge of “black swan” events, such as spikes or sudden jumps, are of key importance. In this connection, we first prove that for Gaussian variables, Granger causality and Rényi transfer entropy are entirely equivalent. Moreover, we also partially extend these results to heavy-tailed α-Gaussian variables. These results allow establishing a connection between autoregressive and Rényi entropy-based information-theoretic approaches to data-driven causal inference. To aid our intuition, we employed the Leonenko et al. entropy estimator and analyzed Rényi’s information flow between bivariate time series generated from two unidirectionally coupled Rössler systems. Notably, we find that Rényi’s transfer entropy not only allows us to detect a threshold of synchronization but it also provides non-trivial insight into the structure of a transient regime that exists between the region of chaotic correlations and synchronization threshold. In addition, from Rényi’s transfer entropy, we could reliably infer the direction of coupling and, hence, causality, only for coupling strengths smaller than the onset value of the transient regime, i.e., when two Rössler systems are coupled but have not yet entered synchronization.

Beyond the Baroreflex: A New Measure of Autonomic Regulation Based on the Time-Frequency Assessment of Variability, Phase Coherence and Couplings

For decades the role of autonomic regulation and the baroreflex in the generation of the respiratory sinus arrhythmia (RSA) - modulation of heart rate by the frequency of breathing - has been under dispute. We hypothesized that by using autonomic blockers we can reveal which oscillations and their interactions are suppressed, elucidating their involvement in RSA as well as in cardiovascular regulation more generally. R-R intervals, end tidal CO2, finger arterial pressure, and muscle sympathetic nerve activity (MSNA) were measured simultaneously in 7 subjects during saline, atropine and propranolol infusion. The measurements were repeated during spontaneous and fixed-frequency breathing, and apnea. The power spectra, phase coherence and couplings were calculated to characterise the variability and interactions within the cardiovascular system. Atropine reduced R-R interval variability (p < 0.05) in all three breathing conditions, reduced MSNA power during apnea and removed much of the significant coherence and couplings. Propranolol had smaller effect on the power of oscillations and did not change the number of significant interactions. Most notably, atropine reduced R-R interval power in the 0.145-0.6 Hz interval during apnea, which supports the hypothesis that the RSA is modulated by a mechanism other than the baroreflex. Atropine also reduced or made negative the phase shift between the systolic and diastolic pressure, indicating the cessation of baroreflex-dependent blood pressure variability. This result suggests that coherent respiratory oscillations in the blood pressure can be used for the non-invasive assessment of autonomic regulation.

A Feed-Forward Neural Network for Increasing the Hopfield-Network Storage Capacity

In the hippocampal dentate gyrus (DG), pattern separation mainly depends on the concepts of 'expansion recoding', meaning random mixing of different DG input channels. However, recent advances in neurophysiology have challenged the theory of pattern separation based on these concepts. In this study, we propose a novel feed-forward neural network, inspired by the structure of the DG and neural oscillatory analysis, to increase the Hopfield-network storage capacity. Unlike the previously published feed-forward neural networks, our bio-inspired neural network is designed to take advantage of both biological structure and functions of the DG. To better understand the computational principles of pattern separation in the DG, we have established a mouse model of environmental enrichment. We obtained a possible computational model of the DG, associated with better pattern separation ability, by using neural oscillatory analysis. Furthermore, we have developed a new algorithm based on Hebbian learning and coupling direction of neural oscillation to train the proposed neural network. The simulation results show that our proposed network significantly expands the storage capacity of Hopfield network, and more effective pattern separation is achieved. The storage capacity rises from 0.13 for the standard Hopfield network to 0.32 using our model when the overlap in patterns is 10%.

Stress-sensitive inference of task controllability

Estimating the controllability of the environment enables agents to better predict upcoming events and decide when to engage controlled action selection. How does the human brain estimate controllability? Trial-by-trial analysis of choices, decision times and neural activity in an explore-and-predict task demonstrate that humans solve this problem by comparing the predictions of an 'actor' model with those of a reduced 'spectator' model of their environment. Neural blood oxygen level-dependent responses within striatal and medial prefrontal areas tracked the instantaneous difference in the prediction errors generated by these two statistical learning models. Blood oxygen level-dependent activity in the posterior cingulate, temporoparietal and prefrontal cortices covaried with changes in estimated controllability. Exposure to inescapable stressors biased controllability estimates downward and increased reliance on the spectator model in an anxiety-dependent fashion. Taken together, these findings provide a mechanistic account of controllability inference and its distortion by stress exposure.

Linear and Nonlinear Directed Connectivity Analysis of the Cardio-Respiratory System in Type 1 Diabetes

In this study, we explored the possibility of developing non-invasive biomarkers for patients with type 1 diabetes (T1D) by quantifying the directional couplings between the cardiac, vascular, and respiratory systems, treating them as interconnected nodes in a network configuration. Towards this goal, we employed a linear directional connectivity measure, the directed transfer function (DTF), estimated by a linear multivariate autoregressive modelling of ECG, respiratory and skin perfusion signals, and a nonlinear method, the dynamical Bayesian inference (DBI) analysis of bivariate phase interactions. The physiological data were recorded concurrently for a relatively short time period (5 min) from 10 healthy control subjects and 10 T1D patients. We found that, in both control and T1D subjects, breathing had greater influence on the heart and perfusion with respect to the opposite coupling direction and that, by both employed methods of analysis, the causal influence of breathing on the heart was significantly decreased (p < 0.05) in T1D patients compared to the control group. These preliminary results, although obtained from a limited number of subjects, provide a strong indication for the usefulness of a network-based multi-modal analysis for the development of biomarkers of T1D-related complications from short-duration data, as well as their potential in the exploration of the pathophysiological mechanisms that underlie this devastating and very widespread disease.

Generative formalism of causality quantifiers for processes

The concept of dynamical causal effect (DCE) is generalized and equipped with a formalism which allows one to formulate in a unified manner and interrelate a variety of causality quantifiers used in time series analysis. An elementary DCE from a subsystem Y to a subsystem X is defined within the stochastic dynamical systems framework as a response of a future X state to an appropriate variation of an initial (X,Y)-state distribution or a certain parameter of Y or of the coupling element Y→X; this response is quantified in a probabilistic sense via a certain distinction functional; elementary DCEs are assembled over a set of initial variations via an assemblage functional. To include all those aspects, a "triple brackets formula" for the general DCE is suggested and serves as a first principle to produce specific causality quantifiers as realizations of the general DCE. As an application, transfer entropy and Liang-Kleeman information flow are related surprisingly as opposite limit cases in a family of DCEs; it is shown that their "nats per time unit" may differ drastically. The suggested DCE viewpoint links any formal causality quantifier to "intervention-effect" experiments, i.e., future responses to initial variations, and so provides its dynamical interpretation, opening a way to its further physical interpretations in studies of physical systems.

A complex network approach to study the extreme precipitation patterns in a river basin

The quantification of spatial propagation of extreme precipitation events is vital in water resources planning and disaster mitigation. However, quantifying these extreme events has always been challenging as many traditional methods are insufficient to capture the nonlinear interrelationships between extreme event time series. Therefore, it is crucial to develop suitable methods for analyzing the dynamics of extreme events over a river basin with a diverse climate and complicated topography. Over the last decade, complex network analysis emerged as a powerful tool to study the intricate spatiotemporal relationship between many variables in a compact way. In this study, we employ two nonlinear concepts of event synchronization and edit distance to investigate the extreme precipitation pattern in the Ganga river basin. We use the network degree to understand the spatial synchronization pattern of extreme rainfall and identify essential sites in the river basin with respect to potential prediction skills. The study also attempts to quantify the influence of precipitation seasonality and topography on extreme events. The findings of the study reveal that (1) the network degree is decreased in the southwest to northwest direction, (2) the timing of 50th percentile precipitation within a year influences the spatial distribution of degree, (3) the timing is inversely related to elevation, and (4) the lower elevation greatly influences connectivity of the sites. The study highlights that edit distance could be a promising alternative to analyze event-like data by incorporating event time and amplitude and constructing complex networks of climate extremes.

Entropic regression with neurologically motivated applications

The ultimate goal of cognitive neuroscience is to understand the mechanistic neural processes underlying the functional organization of the brain. The key to this study is understanding the structure of both the structural and functional connectivity between anatomical regions. In this paper, we use an information theoretic approach, which defines direct information flow in terms of causation entropy, to improve upon the accuracy of the recovery of the true network structure over popularly used methods for this task such as correlation and least absolute shrinkage and selection operator regression. The method outlined above is tested on synthetic data, which is produced by following previous work in which a simple dynamical model of the brain is used, simulated on top of a real network of anatomical brain regions reconstructed from diffusion tensor imaging. We demonstrate the effectiveness of the method of AlMomani et al. [Chaos 30, 013107 (2020)] when applied to data simulated on the realistic diffusion tensor imaging network, as well as on randomly generated small-world and Erdös-Rényi networks.

Testing conditional independence in supervised learning algorithms

We propose the conditional predictive impact (CPI), a consistent and unbiased estimator of the association between one or several features and a given outcome, conditional on a reduced feature set. Building on the knockoff framework of Candès et al. (J R Stat Soc Ser B 80:551–577, 2018), we develop a novel testing procedure that works in conjunction with any valid knockoff sampler, supervised learning algorithm, and loss function. The CPI can be efficiently computed for high-dimensional data without any sparsity constraints. We demonstrate convergence criteria for the CPI and develop statistical inference procedures for evaluating its magnitude, significance, and precision. These tests aid in feature and model selection, extending traditional frequentist and Bayesian techniques to general supervised learning tasks. The CPI may also be applied in causal discovery to identify underlying multivariate graph structures. We test our method using various algorithms, including linear regression, neural networks, random forests, and support vector machines. Empirical results show that the CPI compares favorably to alternative variable importance measures and other nonparametric tests of conditional independence on a diverse array of real and synthetic datasets. Simulations confirm that our inference procedures successfully control Type I error with competitive power in a range of settings. Our method has been implemented in an package, , which can be downloaded from https://github.com/dswatson/cpi.

Principles and open questions in functional brain network reconstruction

Graph theory is now becoming a standard tool in system-level neuroscience. However, endowing observed brain anatomy and dynamics with a complex network representation involves often covert theoretical assumptions and methodological choices which affect the way networks are reconstructed from experimental data, and ultimately the resulting network properties and their interpretation. Here, we review some fundamental conceptual underpinnings and technical issues associated with brain network reconstruction, and discuss how their mutual influence concurs in clarifying the organization of brain function.

Granger-Causality Inference of the Existence of Unobserved Important Components in Network Analysis

Detecting causal interrelationships in multivariate systems, in terms of the Granger-causality concept, is of major interest for applications in many fields. Analyzing all the relevant components of a system is almost impossible, which contrasts with the concept of Granger causality. Not observing some components might, in turn, lead to misleading results, particularly if the missing components are the most influential and important in the system under investigation. In networks, the importance of a node depends on the number of nodes connected to this node. The degree of centrality is the most commonly used measure to identify important nodes in networks. There are two kinds of degree centrality, which are in-degree and out-degree. This manuscrpt is concerned with finding the highest out-degree among nodes to identify the most influential nodes. Inferring the existence of unobserved important components is critical in many multivariate interacting systems. The implications of such a situation are discussed in the Granger-causality framework. To this end, two of the most recent Granger-causality techniques, renormalized partial directed coherence and directed partial correlation, were employed. They were then compared in terms of their performance according to the extent to which they can infer the existence of unobserved important components. Sub-network analysis was conducted to aid these two techniques in inferring the existence of unobserved important components, which is evidenced in the results. By comparing the results of the two conducted techniques, it can be asserted that renormalized partial coherence outperforms directed partial correlation in the inference of existing unobserved important components that have not been included in the analysis. This measure of Granger causality and sub-network analysis emphasizes their ubiquitous successful applicability in such cases of the existence of hidden unobserved important components.

Transfer entropies within dynamical effects framework

Transfer entropy (TE) is widely used in time-series analysis to detect causal couplings between temporally evolving objects. As a coupling strength quantifier, the TE alone often seems insufficient, raising the question of its further interpretations. Here the TE is related to dynamical causal effects (DCEs) which quantify long-term responses of a coupling recipient to variations in a coupling source or in a coupling itself: Detailed relationships are established for a paradigmatic stochastic dynamical system of bidirectionally coupled linear overdamped oscillators, their practical applications and possible extensions are discussed. It is shown that two widely used versions of the TE (original and infinite-history) can become qualitatively distinct, diverging to different long-term DCEs.

The Human Organism as an Integrated Interaction Network: Recent Conceptual and Methodological Challenges

The field of Network Physiology aims to advance our understanding of how physiological systems and sub-systems interact to generate a variety of behaviors and distinct physiological states, to optimize the organism's functioning, and to maintain health. Within this framework, which considers the human organism as an integrated network, vertices are associated with organs while edges represent time-varying interactions between vertices. Likewise, vertices may represent networks on smaller spatial scales leading to a complex mixture of interacting homogeneous and inhomogeneous networks of networks. Lacking adequate analytic tools and a theoretical framework to probe interactions within and among diverse physiological systems, current approaches focus on inferring properties of time-varying interactions-namely strength, direction, and functional form-from time-locked recordings of physiological observables. To this end, a variety of bivariate or, in general, multivariate time-series-analysis techniques, which are derived from diverse mathematical and physical concepts, are employed and the resulting time-dependent networks can then be further characterized with methods from network theory. Despite the many promising new developments, there are still problems that evade from a satisfactory solution. Here we address several important challenges that could aid in finding new perspectives and inspire the development of theoretic and analytical concepts to deal with these challenges and in studying the complex interactions between physiological systems.

Bladder pressure encoding by sacral dorsal root ganglion fibres: implications for decoding

OBJECTIVE

We aim at characterising the encoding of bladder pressure (intravesical pressure) by a population of sensory fibres. This research is motivated by the possibility to restore bladder function in elderly patients or after spinal cord injury using implanted devices, so called bioelectronic medicines. For these devices, nerve-based estimation of intravesical pressure can enable a personalized and on-demand stimulation paradigm, which has promise of being more effective and efficient. In this context, a better understanding of the encoding strategies employed by the body might in the future be exploited by informed decoding algorithms that enable a precise and robust bladder-pressure estimation.

APPROACH

To this end, we apply information theory to microelectrode-array recordings from the cat sacral dorsal root ganglion while filling the bladder, conduct surrogate data studies to augment the data we have, and finally decode pressure in a simple informed approach.

MAIN RESULTS

We find an encoding scheme by different main bladder neuron types that we divide into three response types (slow tonic, phasic, and derivative fibres). We show that an encoding by different bladder neuron types, each represented by multiple cells, offers reliability through within-type redundancy and high information rates through semi-independence of different types. Our subsequent decoding study shows a more robust decoding from mean responses of homogeneous cell pools.

SIGNIFICANCE

We have here, for the first time, established a link between an information theoretic analysis of the encoding of intravesical pressure by a population of sensory neurons to an informed decoding paradigm. We show that even a simple adapted decoder can exploit the redundancy in the population to be more robust against cell loss. This work thus paves the way towards principled encoding studies in the periphery and towards a new generation of informed peripheral nerve decoders for bioelectronic medicines.

Detecting causality from time series in a machine learning framework

Detecting causality from observational data is a challenging problem. Here, we propose a machine learning based causality approach, Reservoir Computing Causality (RCC), in order to systematically identify causal relationships between variables. We demonstrate that RCC is able to identify the causal direction, coupling delay, and causal chain relations from time series. Compared to a well-known phase space reconstruction based causality method, Extended Convergent Cross Mapping, RCC does not require the estimation of the embedding dimension and delay time. Moreover, RCC has three additional advantages: (i) robustness to noisy time series; (ii) computational efficiency; and (iii) seamless causal inference from high-dimensional data. We also illustrate the power of RCC in identifying remote causal interactions of high-dimensional systems and demonstrate its usability on a real-world example using atmospheric circulation data. Our results suggest that RCC can accurately detect causal relationships in complex systems.

Learning clinical networks from medical records based on information estimates in mixed-type data

The precise diagnostics of complex diseases require to integrate a large amount of information from heterogeneous clinical and biomedical data, whose direct and indirect interdependences are notoriously difficult to assess. To this end, we propose an efficient computational approach to simultaneously compute and assess the significance of multivariate information between any combination of mixed-type (continuous/categorical) variables. The method is then used to uncover direct, indirect and possibly causal relationships between mixed-type data from medical records, by extending a recent machine learning method to reconstruct graphical models beyond simple categorical datasets. The method is shown to outperform existing tools on benchmark mixed-type datasets, before being applied to analyze the medical records of eldery patients with cognitive disorders from La Pitié-Salpêtrière Hospital, Paris. The resulting clinical network visually captures the global interdependences in these medical records and some facets of clinical diagnosis practice, without specific hypothesis nor prior knowledge on any clinically relevant information. In particular, it provides some physiological insights linking the consequence of cerebrovascular accidents to the atrophy of important brain structures associated to cognitive impairment.

We developed a machine learning approach to analyze medical records and help clinicians visualize the direct and indirect interrelations between clinical examinations and the variety of syndromes implicated in complex diseases. The reconstruction of such clinical networks is illustrated on the spectrum of cognitive disorders, originating from either neurodegenerative, cerebrovascular or psychiatric dementias. This global network analysis is also shown to uncover novel direct associations and possible cause-effect relationships between clinically relevant information, such as medical examinations, diagnoses, treatments and personal data from patients’ medical records.

Detecting and quantifying causal associations in large nonlinear time series datasets

Identifying causal relationships and quantifying their strength from observational time series data are key problems in disciplines dealing with complex dynamical systems such as the Earth system or the human body. Data-driven causal inference in such systems is challenging since datasets are often high dimensional and nonlinear with limited sample sizes. Here, we introduce a novel method that flexibly combines linear or nonlinear conditional independence tests with a causal discovery algorithm to estimate causal networks from large-scale time series datasets. We validate the method on time series of well-understood physical mechanisms in the climate system and the human heart and using large-scale synthetic datasets mimicking the typical properties of real-world data. The experiments demonstrate that our method outperforms state-of-the-art techniques in detection power, which opens up entirely new possibilities to discover and quantify causal networks from time series across a range of research fields.

Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing

Network inference algorithms are valuable tools for the study of large-scale neuroimaging datasets. Multivariate transfer entropy is well suited for this task, being a model-free measure that captures nonlinear and lagged dependencies between time series to infer a minimal directed network model. Greedy algorithms have been proposed to efficiently deal with high-dimensional datasets while avoiding redundant inferences and capturing synergistic effects. However, multiple statistical comparisons may inflate the false positive rate and are computationally demanding, which limited the size of previous validation studies. The algorithm we present-as implemented in the IDTxl open-source software-addresses these challenges by employing hierarchical statistical tests to control the family-wise error rate and to allow for efficient parallelization. The method was validated on synthetic datasets involving random networks of increasing size (up to 100 nodes), for both linear and nonlinear dynamics. The performance increased with the length of the time series, reaching consistently high precision, recall, and specificity (>98% on average) for 10,000 time samples. Varying the statistical significance threshold showed a more favorable precision-recall trade-off for longer time series. Both the network size and the sample size are one order of magnitude larger than previously demonstrated, showing feasibility for typical EEG and magnetoencephalography experiments.

Transfer entropy on symbolic recurrences

Recurrence quantification analysis offers a powerful framework to investigate complexity in dynamical systems. While several studies have demonstrated the possibility of multivariate recurrence quantification analysis, information-theoretic tools for the discovery of causal links remain elusive. Particularly enticing is to formulate information-theoretic tools on symbolic recurrence plots, which alleviate some of the methodological challenges of traditional recurrence plots and offer a richer representation of recurrences. Toward this aim, we establish a probability space in which we ground a theory of information that encodes information in the recurrences of the symbols. We introduce transfer entropy on symbolic recurrences as a tool to guide the inference of the strength and direction of the interaction between dynamical systems. We demonstrate statistically reliable discovery of causal links on synthetic and experimental time series, from only two time series or a larger dataset with multiple realizations. The proposed approach brings together recurrence plots, information theory, and symbolic dynamics to empower researchers and practitioners with effective means to visualize and quantify interactions in dynamical systems.

A study for multiscale information transfer measures based on conditional mutual information

As the big data science develops, efficient methods are demanded for various data analysis. Granger causality provides the prime model for quantifying causal interactions. However, this theoretic model does not meet the requirement for real-world data analysis, because real-world time series are diverse whose models are usually unknown. Therefore, model-free measures such as information transfer measures are strongly desired. Here, we propose the multi-scale extension of conditional mutual information measures using MORLET wavelet, which are named the WM and WPM. The proposed measures are computational efficient and interpret information transfer by multi-scales. We use both synthetic data and real-world examples to demonstrate the efficiency of the new methods. The results of the new methods are robust and reliable. Via the simulation studies, we found the new methods outperform the wavelet extension of transfer entropy (WTE) in both computational efficiency and accuracy. The features and properties of the proposed measures are also discussed.

Inference of time-varying networks through transfer entropy, the case of a Boolean network model

Inferring network topologies from the time series of individual units is of paramount importance in the study of biological and social networks. Despite considerable progress, our success in network inference is largely limited to static networks and autonomous node dynamics, which are often inadequate to describe complex systems. Here, we explore the possibility of reconstructing time-varying weighted topologies through the information-theoretic notion of transfer entropy. We focus on a Boolean network model in which the weight of the links and the spontaneous activity periodically vary in time. For slowly-varying dynamics, we establish closed-form expressions for the stationary periodic distribution and transfer entropy between each pair of nodes. Our results indicate that the instantaneous weight of each link is mapped into a corresponding transfer entropy value, thereby affording the possibility of pinpointing the dominant weights at each time. However, comparing transfer entropy readings at different times may provide erroneous estimates of the strength of the links in time, due to a counterintuitive modulation of the information flow by the non-autonomous dynamics. In fact, this time variation should be used to scale transfer entropy values toward the correct inference of the time evolution of the network weights. This study constitutes a necessary step toward a mathematically-principled use of transfer entropy to reconstruct time-varying networks.

Detecting intermittent switching leadership in coupled dynamical systems

Leader-follower relationships are commonly hypothesized as a fundamental mechanism underlying collective behaviour in many biological and physical systems. Understanding the emergence of such behaviour is relevant in science and engineering to control the dynamics of complex systems toward a desired state. In prior works, due in part to the limitations of existing methods for dissecting intermittent causal relationships, leadership is assumed to be consistent in time and space. This assumption has been contradicted by recent progress in the study of animal behaviour. In this work, we leverage information theory and time series analysis to propose a novel and simple method for dissecting changes in causal influence. Our approach computes the cumulative influence function of a given individual on the rest of the group in consecutive time intervals and identify change in the monotonicity of the function as a change in its leadership status. We demonstrate the effectiveness of our approach to dissect potential changes in leadership on self-propelled particles where the emergence of leader-follower relationship can be controlled and on tandem flights of birds recorded in their natural environment. Our method is expected to provide a novel methodological tool to further our understanding of collective behaviour.

Causal network reconstruction from time series: From theoretical assumptions to practical estimation

Causal network reconstruction from time series is an emerging topic in many fields of science. Beyond inferring directionality between two time series, the goal of causal network reconstruction or causal discovery is to distinguish direct from indirect dependencies and common drivers among multiple time series. Here, the problem of inferring causal networks including time lags from multivariate time series is recapitulated from the underlying causal assumptions to practical estimation problems. Each aspect is illustrated with simple examples including unobserved variables, sampling issues, determinism, stationarity, nonlinearity, measurement error, and significance testing. The effects of dynamical noise, autocorrelation, and high dimensionality are highlighted in comparison studies of common causal reconstruction methods. Finally, method performance evaluation approaches and criteria are suggested. The article is intended to briefly review and accessibly illustrate the foundations and practical problems of time series-based causal discovery and stimulate further methodological developments.

Directionality indices: Testing information transfer with surrogate correction

Directionality indices can be used as an indicator of the asymmetry in coupling between systems and have found particular application in relation to neurological systems. The directionality index between two systems is a function of measures of information transfer in both directions. Here we illustrate that before inferring the directionality of coupling it is first necessary to consider the use of appropriate tests of significance. We propose a surrogate corrected directionality index which incorporates such testing. We also highlight the differences between testing the significance of the directionality index itself versus testing the individual measures of information transfer in each direction. To validate the approach we compared two different methods of estimating coupling, both of which have previously been used to estimate directionality indices. These were the modeling-based evolution map approach and a conditional mutual information (CMI) method for calculating dynamic information rates. For the CMI-based approach we also compared two different methods for estimating the CMI, an equiquantization-based estimator and a k-nearest neighbors estimator.

Detection of coupling delay: A problem not yet solved

Nonparametric detection of coupling delay in unidirectionally and bidirectionally coupled nonlinear dynamical systems is examined. Both continuous and discrete-time systems are considered. Two methods of detection are assessed-the method based on conditional mutual information-the CMI method (also known as the transfer entropy method) and the method of convergent cross mapping-the CCM method. Computer simulations show that neither method is generally reliable in the detection of coupling delays. For continuous-time chaotic systems, the CMI method appears to be more sensitive and applicable in a broader range of coupling parameters than the CCM method. In the case of tested discrete-time dynamical systems, the CCM method has been found to be more sensitive, while the CMI method required much stronger coupling strength in order to bring correct results. However, when studied systems contain a strong oscillatory component in their dynamics, results of both methods become ambiguous. The presented study suggests that results of the tested algorithms should be interpreted with utmost care and the nonparametric detection of coupling delay, in general, is a problem not yet solved.

Neuronal networks in the developing brain are adversely modulated by early psychosocial neglect

The brain's neural circuitry plays a ubiquitous role across domains in cognitive processing and undergoes extensive reorganization during the course of development in part as a result of experience. In this study we investigated the effects of profound early psychosocial neglect associated with institutional rearing on the development of task-independent brain networks, estimated from longitudinally acquired electroencephalographic (EEG) data from <30 to 96 mo, in three cohorts of children from the Bucharest Early Intervention Project (BEIP), including abandoned children reared in institutions who were randomly assigned either to a foster care intervention or to remain in care as usual and never-institutionalized children. Two aberrantly connected brain networks were identified in children that had been reared in institutions: 1) a hyperconnected parieto-occipital network, which included cortical hubs and connections that may partially overlap with default-mode network, and 2) a hypoconnected network between left temporal and distributed bilateral regions, both of which were aberrantly connected across neural oscillations. This study provides the first evidence of the adverse effects of early psychosocial neglect on the wiring of the developing brain. Given these networks' potentially significant role in various cognitive processes, including memory, learning, social communication, and language, these findings suggest that institutionalization in early life may profoundly impact the neural correlates underlying multiple cognitive domains, in ways that may not be fully reversible in the short term.NEW & NOTEWORTHY This paper provides first evidence that early psychosocial neglect associated with institutional rearing profoundly affects the development of the brain's neural circuitry. Using longitudinally acquired electrophysiological data from the Bucharest Early Intervention Project (BEIP), the paper identifies multiple task-independent networks that are abnormally connected (hyper- or hypoconnected) in children reared in institutions compared with never-institutionalized children. These networks involve spatially distributed brain areas and their abnormal connections may adversely impact neural information processing across cognitive domains.

An information-based network approach for protein classification

Protein classification is one of the critical problems in bioinformatics. Early studies used geometric distances and polygenetic-tree to classify proteins. These methods use binary trees to present protein classification. In this paper, we propose a new protein classification method, whereby theories of information and networks are used to classify the multivariate relationships of proteins. In this study, protein universe is modeled as an undirected network, where proteins are classified according to their connections. Our method is unsupervised, multivariate, and alignment-free. It can be applied to the classification of both protein sequences and structures. Nine examples are used to demonstrate the efficiency of our new method.

Population dynamics, information transfer, and spatial organization in a chemical reaction network under spatial confinement and crowding conditions

We investigate, via Brownian dynamics simulations, the reaction dynamics of a generic, nonlinear chemical network under spatial confinement and crowding conditions. In detail, the Willamowski-Rossler chemical reaction system has been "extended" and considered as a prototype reaction-diffusion system. Our results are potentially relevant to a number of open problems in biophysics and biochemistry, such as the synthesis of primitive cellular units (protocells) and the definition of their role in the chemical origin of life and the characterization of vesicle-mediated drug delivery processes. More generally, the computational approach presented in this work makes the case for the use of spatial stochastic simulation methods for the study of biochemical networks in vivo where the "well-mixed" approximation is invalid and both thermal and intrinsic fluctuations linked to the possible presence of molecular species in low number copies cannot be averaged out.

Interaction between Thalamus and Hippocampus in Termination of Amygdala-Kindled Seizures in Mice

The thalamus and hippocampus have been found both involved in the initiation, propagation, and termination of temporal lobe epilepsy. However, the interaction of these regions during seizures is not clear. The present study is to explore whether some regular patterns exist in their interaction during the termination of seizures. Multichannel in vivo recording techniques were used to record the neural activities from the cornu ammonis 1 (CA1) of hippocampus and mediodorsal thalamus (MDT) in mice. The mice were kindled by electrically stimulating basolateral amygdala neurons, and Racine's rank standard was employed to classify the stage of behavioral responses (stage 1~5). The coupling index and directionality index were used to investigate the synchronization and information flow direction between CA1 and MDT. Two main results were found in this study. (1) High levels of synchronization between the thalamus and hippocampus were observed before the termination of seizures at stage 4~5 but after the termination of seizures at stage 1~2. (2) In the end of seizures at stage 4~5, the information tended to flow from MDT to CA1. Those results indicate that the synchronization and information flow direction between the thalamus and the hippocampus may participate in the termination of seizures.

Difficulties in auditory organization as a cause of reading backwardness? An auditory neuroscience perspective

Over 30 years ago, it was suggested that difficulties in the 'auditory organization' of word forms in the mental lexicon might cause reading difficulties. It was proposed that children used parameters such as rhyme and alliteration to organize word forms in the mental lexicon by acoustic similarity, and that such organization was impaired in developmental dyslexia. This literature was based on an 'oddity' measure of children's sensitivity to rhyme (e.g. wood, book, good) and alliteration (e.g. sun, sock, rag). The 'oddity' task revealed that children with dyslexia were significantly poorer at identifying the 'odd word out' than younger children without reading difficulties. Here we apply a novel modelling approach drawn from auditory neuroscience to study the possible sensory basis of the auditory organization of rhyming and non-rhyming words by children. We utilize a novel Spectral-Amplitude Modulation Phase Hierarchy (S-AMPH) approach to analysing the spectro-temporal structure of rhyming and non-rhyming words, aiming to illuminate the potential acoustic cues used by children as a basis for phonological organization. The S-AMPH model assumes that speech encoding depends on neuronal oscillatory entrainment to the amplitude modulation (AM) hierarchy in speech. Our results suggest that phonological similarity between rhyming words in the oddity task depends crucially on slow (delta band) modulations in the speech envelope. Contrary to linguistic assumptions, therefore, auditory organization by children may not depend on phonemic information for this task. Linguistically, it is assumed that 'book' does not rhyme with 'wood' and 'good' because the final phoneme differs. However, our auditory analysis suggests that the acoustic cues to this phonological dissimilarity depend primarily on the slower amplitude modulations in the speech envelope, thought to carry prosodic information. Therefore, the oddity task may help in detecting reading difficulties because phonological similarity judgements about rhyme reflect sensitivity to slow amplitude modulation patterns. Slower amplitude modulations are known to be detected less efficiently by children with dyslexia.

Acute Sleep Deprivation Induces a Local Brain Transfer Information Increase in the Frontal Cortex in a Widespread Decrease Context

Sleep deprivation (SD) has adverse effects on mental and physical health, affecting the cognitive abilities and emotional states. Specifically, cognitive functions and alertness are known to decrease after SD. The aim of this work was to identify the directional information transfer after SD on scalp EEG signals using transfer entropy (TE). Using a robust methodology based on EEG recordings of 18 volunteers deprived from sleep for 36 h, TE and spectral analysis were performed to characterize EEG data acquired every 2 h. Correlation between connectivity measures and subjective somnolence was assessed. In general, TE showed medium- and long-range significant decreases originated at the occipital areas and directed towards different regions, which could be interpreted as the transfer of predictive information from parieto-occipital activity to the rest of the head. Simultaneously, short-range increases were obtained for the frontal areas, following a consistent and robust time course with significant maps after 20 h of sleep deprivation. Changes during sleep deprivation in brain network were measured effectively by TE, which showed increased local connectivity and diminished global integration. TE is an objective measure that could be used as a potential measure of sleep pressure and somnolence with the additional property of directed relationships.

Model-free information-theoretic approach to infer leadership in pairs of zebrafish

Collective behavior affords several advantages to fish in avoiding predators, foraging, mating, and swimming. Although fish schools have been traditionally considered egalitarian superorganisms, a number of empirical observations suggest the emergence of leadership in gregarious groups. Detecting and classifying leader-follower relationships is central to elucidate the behavioral and physiological causes of leadership and understand its consequences. Here, we demonstrate an information-theoretic approach to infer leadership from positional data of fish swimming. In this framework, we measure social interactions between fish pairs through the mathematical construct of transfer entropy, which quantifies the predictive power of a time series to anticipate another, possibly coupled, time series. We focus on the zebrafish model organism, which is rapidly emerging as a species of choice in preclinical research for its genetic similarity to humans and reduced neurobiological complexity with respect to mammals. To overcome experimental confounds and generate test data sets on which we can thoroughly assess our approach, we adapt and calibrate a data-driven stochastic model of zebrafish motion for the simulation of a coupled dynamical system of zebrafish pairs. In this synthetic data set, the extent and direction of the coupling between the fish are systematically varied across a wide parameter range to demonstrate the accuracy and reliability of transfer entropy in inferring leadership. Our approach is expected to aid in the analysis of collective behavior, providing a data-driven perspective to understand social interactions.

Oscillatory Diffusion and Second-Order Cyclostationarity in Alanine Tripeptide from Molecular Dynamics Simulation

Molecular dynamics (MD) simulation distinctly describes motions of biomolecules at high resolution and can potentially be used to explain allosteric mechanism in subcellular processes. Statistical methods are necessary to realize this potential because MD simulations generate a large volume of data and because the analysis is never efficient, objective, or thorough without using appropriate statistical approaches. Tracing the flow of information within a biomolecule requires not only a description of an overall mechanism but also a multiscale statistical description from atomic interactions to the overall mechanism. The foundation of this multiscale description, in general, is a measure of correlation between motions of atoms or residues, as reflected by dynamic cross-correlation, Pearson correlation, or mutual information. However, these correlations can be inadequate because they assume wide sense stationarity, which means that the instantaneous average and correlation of a particular property are time-independent. Consequently, these measures of correlation cannot account for correlation between motions of different frequencies, since frequency implies oscillation and variation over time. Here, we characterize the nonstationarity in the form of pure oscillatory instantaneous variance in the signed dihedral angular accelerations (SDAA) along the main chain of alanine tripeptide in MD simulations by power spectrum, corrected squared envelope spectrum (CSES), and cross-CSES. This oscillation has a physical interpretation of an oscillatory diffusion. The fraction of this oscillation in all motions is as high as about 40% at some frequencies. This shows that oscillatory instantaneous variance exists in the SDAA and that significant correlation may not be accounted for in current correlation analysis. This oscillation is also found to transmit between dihedral angles. These results could have implications in the understanding of the dynamics of biomolecules.

Conditional Self-Entropy and Conditional Joint Transfer Entropy in Heart Period Variability during Graded Postural Challenge

Self-entropy (SE) and transfer entropy (TE) are widely utilized in biomedical signal processing to assess the information stored into a system and transferred from a source to a destination respectively. The study proposes a more specific definition of the SE, namely the conditional SE (CSE), and a more flexible definition of the TE based on joint TE (JTE), namely the conditional JTE (CJTE), for the analysis of information dynamics in multivariate time series. In a protocol evoking a gradual sympathetic activation and vagal withdrawal proportional to the magnitude of the orthostatic stimulus, such as the graded head-up tilt, we extracted the beat-to-beat spontaneous variability of heart period (HP), systolic arterial pressure (SAP) and respiratory activity (R) in 19 healthy subjects and we computed SE of HP, CSE of HP given SAP and R, JTE from SAP and R to HP, CJTE from SAP and R to HP given SAP and CJTE from SAP and R to HP given R. CSE of HP given SAP and R was significantly smaller than SE of HP and increased progressively with the amplitude of the stimulus, thus suggesting that dynamics internal to HP and unrelated to SAP and R, possibly linked to sympathetic activation evoked by head-up tilt, might play a role during the orthostatic challenge. While JTE from SAP and R to HP was independent of tilt table angle, CJTE from SAP and R to HP given R and from SAP and R to HP given SAP showed opposite trends with tilt table inclination, thus suggesting that the importance of the cardiac baroreflex increases and the relevance of the cardiopulmonary pathway decreases during head-up tilt. The study demonstrates the high specificity of CSE and the high flexibility of CJTE over real data and proves that they are particularly helpful in disentangling physiological mechanisms and in assessing their different contributions to the overall cardiovascular regulation.