Symbolic transfer entropy

Revealing the structure of information flows discriminates similar animal social behaviors

Behavioral correlations stretching over time are an essential but often neglected aspect of interactions among animals. These correlations pose a challenge to current behavioral-analysis methods that lack effective means to analyze complex series of interactions. Here we show that non-invasive information-theoretic tools can be used to reveal communication protocols that guide complex social interactions by measuring simultaneous flows of different types of information between subjects. We demonstrate this approach by showing that the tandem-running behavior of the ant Temnothorax rugatulus and that of the termites Coptotermes formosanus and Reticulitermes speratus are governed by different communication protocols. Our discovery reconciles the diverse ultimate causes of tandem running across these two taxa with their apparently similar signaling mechanisms. We show that bidirectional flow of information is present only in ants and is consistent with the use of acknowledgement signals to regulate the flow of directional information.

Multifractal and Entropy-Based Analysis of Delta Band Neural Activity Reveals Altered Functional Connectivity Dynamics in Schizophrenia

Dynamic functional connectivity (DFC) was established in the past decade as a potent approach to reveal non-trivial, time-varying properties of neural interactions – such as their multifractality or information content –, that otherwise remain hidden from conventional static methods. Several neuropsychiatric disorders were shown to be associated with altered DFC, with schizophrenia (SZ) being one of the most intensely studied among such conditions. Here we analyzed resting-state electroencephalography recordings of 14 SZ patients and 14 age- and gender-matched healthy controls (HC). We reconstructed dynamic functional networks from delta band (0.5–4 Hz) neural activity and captured their spatiotemporal dynamics in various global network topological measures. The acquired network measure time series were made subject to dynamic analyses including multifractal analysis and entropy estimation. Besides group-level comparisons, we built a classifier to explore the potential of DFC features in classifying individual cases. We found stronger delta-band connectivity, as well as increased variance of DFC in SZ patients. Surrogate data testing verified the true multifractal nature of DFC in SZ, with patients expressing stronger long-range autocorrelation and degree of multifractality when compared to controls. Entropy analysis indicated reduced temporal complexity of DFC in SZ. When using these indices as features, an overall cross-validation accuracy surpassing 89% could be achieved in classifying individual cases. Our results imply that dynamic features of DFC such as its multifractal properties and entropy are potent markers of altered neural dynamics in SZ and carry significant potential not only in better understanding its pathophysiology but also in improving its diagnosis. The proposed framework is readily applicable for neuropsychiatric disorders other than schizophrenia.

Spontaneous Feedforward Connectivity in Electrically Stimulated Retinal Degeneration Mice. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020;2020:3513-3516. [PMID: 33018761 DOI: 10.1109/embc44109.2020.9175231]

Retinal degeneration (Rd) is a neurodegenerative disorder primarily associated with the degeneration of the retina neurons and culminates in the eventual loss of visual perception or blindness. Decrease in fronto-, parietal and occipital brain connectivity have been reported in a number of neurodegeneration diseases involving cognitive decline. However, cortical communication in the brain of retinal degeneration patients remains largely unknown and strategies to remediate observed dysfunctional brain connectivity in such instance have not be thoroughly investigated. We used rd10 mice as a model to study brain connectivity in the human retinal degeneration disease, retinitis pigmentosa. Rd10 mice with sham matched controls were electrically stimulated at varying stimulation frequencies and the consequent perturbations in feedforward brain connectivity were studied in the visual cortex and pre-frontal cortex using electrocorticography (ECoG) and normalized symbolic transfer entropy (NSTE). Contra Vcx - contra PFx feed forward connectivity significantly (p<0.05) increased in theta, alpha and beta oscillatory bands of 2 Hz and 10 Hz stimulated rd10 respectively in comparison with sham group. Also, this increase was significantly maintained even after the end of the stimulation period.

Four-Types of IIT-Induced Group Integrity of Plecoglossus altivelis

Integrated information theory (IIT) was initially proposed to describe human consciousness in terms of intrinsic-causal brain network structures. Particularly, IIT 3.0 targets the system's cause-effect structure from spatio-temporal grain and reveals the system's irreducibility. In a previous study, we tried to apply IIT 3.0 to an actual collective behaviour in Plecoglossus altivelis. We found that IIT 3.0 exhibits qualitative discontinuity between three and four schools of fish in terms of Φ value distributions. Other measures did not show similar characteristics. In this study, we followed up on our previous findings and introduced two new factors. First, we defined the global parameter settings to determine a different kind of group integrity. Second, we set several timescales (from Δ t = 5 / 120 to Δ t = 120 / 120 s). The results showed that we succeeded in classifying fish schools according to their group sizes and the degree of group integrity around the reaction time scale of the fish, despite the small group sizes. Compared with the short time scale, the interaction heterogeneity observed in the long time scale seems to diminish. Finally, we discuss one of the longstanding paradoxes in collective behaviour, known as the heap paradox, for which two tentative answers could be provided through our IIT 3.0 analysis.

Transfer entropy rate through Lempel-Ziv complexity

The transfer entropy and the transfer entropy rate are closely related concepts that measure information exchange between two dynamical systems. These measures allow us to study linear and nonlinear causality relations and can be estimated through the use of different methodologies. However, some of them assume a data model and/or are computationally expensive. This article depicts a methodology to estimate the transfer entropy rate between two systems through the Lempel-Ziv complexity. This methodology offers a set of advantages: It estimates the transfer entropy rate from two single discrete series of measures, it is not computationally expensive, and it does not assume any data model. The simulation results over three different unidirectional coupled dynamical systems suggest that this methodology can be used to assess the direction and strength of the information flow between systems. Moreover, it provides good estimations for short-length time series.

Sleep Electroencephalographic Response to Respiratory Events in Patients With Moderate Sleep Apnea-Hypopnea Syndrome

Sleep apnea–hypopnea syndrome is a common breathing disorder that can lead to organic brain injury, prevent memory consolidation, and cause other adverse mental-related complications. Brain activity while sleeping during respiratory events is related to these dysfunctions. In this study, we analyzed variations in electroencephalography (EEG) signals before, during, and after such events. Absolute and relative powers, as well as symbolic transfer entropy (STE) of scalp EEG signals, were calculated to unveil the activity of brain regions and information interactions between them, respectively. During the respiratory events, only low-frequency power increased during rapid eye movement (REM) stage (δ-band absolute and relative power) and N1 (δ- and θ-band absolute power, δ-band relative power) sleep. But absolute power increased in low- and medium-frequency bands (δ, θ, α, and σ bands), and relative power increased mainly in the medium-frequency band (α and σ bands) during stage N2 sleep. After the respiratory events, absolute power increased in all frequency bands and sleep stages, but relative power increased in medium and high frequencies. Regarding information interactions, the β-band STE decreased during and after events. In the γ band, the intrahemispheric STE increased during events and decreased afterward. Moreover, the interhemisphere STE increased after events during REM and stage N1 sleep. The EEG changes throughout respiratory events are supporting evidence for previous EEG knowledge of the impact of sleep apnea on the brain. These findings may provide insights into the influence of the sleep apnea–hypopnea syndrome on cognitive function and neuropsychiatric defects.

Symbolic transfer entropy reveals the age structure of pandemic influenza transmission from high-volume influenza-like illness data

Existing methods to infer the relative roles of age groups in epidemic transmission can normally only accommodate a few age classes, and/or require data that are highly specific for the disease being studied. Here, symbolic transfer entropy (STE), a measure developed to identify asymmetric transfer of information between stochastic processes, is presented as a way to reveal asymmetric transmission patterns between age groups in an epidemic. STE provides a ranking of which age groups may dominate transmission, rather than a reconstruction of the explicit between-age-group transmission matrix. Using simulations, we establish that STE can identify which age groups dominate transmission even when there are differences in reporting rates between age groups and even if the data are noisy. Then, the pairwise STE is calculated between time series of influenza-like illness for 12 age groups in 884 US cities during the autumn of 2009. Elevated STE from 5 to 19 year-olds indicates that school-aged children were likely the most important transmitters of infection during the autumn wave of the 2009 pandemic in the USA. The results may be partially confounded by higher rates of physician-seeking behaviour in children compared to adults, but it is unlikely that differences in reporting rates can explain the observed differences in STE.

Characterization of surface electromyography signals of biceps brachii muscle in fatigue using symbolic motif features

Exercise-induced muscle damage is a condition which results in the loss of muscle function due to overexertion. Muscle fatigue is a precursor of this phenomenon. The characterization of muscle fatigue plays a crucial role in preventing muscle damage. In this work, an attempt is made to develop signal processing methods to understand the dynamics of the muscle’s electrical properties. Surface electromyography signals are recorded from 50 healthy adult volunteers under dynamic curl exercise. The signals are preprocessed, and the first difference signal is computed. Furthermore, ascending and descending slopes are used to generate a binary sequence. The binary sequence of various motif lengths is analyzed using features such as the average symbolic occurrence, modified Shannon entropy, chi-square value, time irreversibility, maximum probability of pattern and forbidden pattern ratio. The progression of muscle fatigue is assessed using trend analysis techniques. The motif length is optimized to maximize the rho value of features. In addition, the first and the last zones of the signal are compared with standard statistical tests. The results indicate that the recorded signals differ in both frequency and amplitude in both inter- and intra-subjects along the period of the experiment. The binary sequence generated has information related to the complexity of the signal. The presence of more repetitive patterns across the motif lengths in the case of fatigue indicates that the signal has lower complexity. In most cases, larger motif length resulted in better rho values. In a comparison of the first and the last zones, most of the extracted features are statistically significant with p < 0.05. It is observed that at the motif length of 13 all the extracted features are significant. This analysis method can be extended to diagnose other neuromuscular conditions.

The impact of financial contagion on real economy-An empirical research based on combination of complex network technology and spatial econometrics model

This study presents financial network indicators that can be applied to inspect the financial contagion on real economy, as well as the spatial spillover and industry aggregation effects. We propose to design both a directed and undirected networks of financial sectors of top 20 countries in GDP based on symbolized transfer entropy and Pearson correlation coefficients. We examine the effect and usefulness of the network indicators by newly using them instead of the original Dow Jones financial sector as explanatory variables to construct the higher-order information spatial econometric models. The results demonstrate that the estimated accuracies obtained from both the two networks are improved significantly compared with the spatial econometric model using the original data. It indicates that the network indictors are more effective to capture the dynamic information of financial systems. And meanwhile, the accuracy based on the directed network is a little higher than the undirected network, which indicates the symbolized transfer entropy, i.e. the directed and weighted network, is more suitable and effective to reflect relationships in the financial field. In addition, the results also show that under the global financial crisis, the co-movement between financial sectors of a country/region and the global financial sector as well as between financial sectors and real economy sectors is increased. However, some sectors in particular Utilities and Healthcare are impacted slightly. This study tries to use the financial network indicators in modeling to study contagion channels on the real economy and the industry aggregation effects and suggest how network indicators can be practically used in financial fields.

Finding continuity and discontinuity in fish schools via integrated information theory

Collective behaviours are known to be the result of diverse dynamics and are sometimes likened to living systems. Although many studies have revealed the dynamics of various collective behaviours, their main focus has been on the information processing performed by the collective, not on interactions within the collective. For example, the qualitative difference between three and four elements in a system has rarely been investigated. Tononi et al. proposed integrated information theory (IIT) to measure the degree of consciousness Φ. IIT postulates that the amount of information loss caused by the minimum information partition is equivalent to the degree of information integration in the system. This measure is not only useful for estimating the degree of consciousness but can also be applied to more general network systems. Here, we obtained two main results from the application of IIT (in particular, IIT 3.0) to the analysis of real fish schools (Plecoglossus altivelis). First, we observed that the discontinuity on 〈Φ(N)〉 distributions emerges for a school of four or more fish. This transition was not observed by measuring the mutual information or the sum of the transfer entropy. We also analysed the IIT on Boids simulations with respect to different coupling strengths; however, the results of the Boids model were found to be quite different from those of real fish. Second, we found a correlation between this discontinuity and the emergence of leadership. We discriminate leadership in this paper from its traditional meaning (e.g. defined by transfer entropy) because IIT-induced leadership refers not to group behaviour, as in other methods, but the degree of autonomy (i.e. group integrity). These results suggest that integrated information Φ can reveal the emergence of a new type of leadership which cannot be observed using other measures.

Information Transfer between Stock Market Sectors: A Comparison between the USA and China

Information diffusion within financial markets plays a crucial role in the process of price formation and the propagation of sentiment and risk. We perform a comparative analysis of information transfer between industry sectors of the Chinese and the USA stock markets, using daily sector indices for the period from 2000 to 2017. The information flow from one sector to another is measured by the transfer entropy of the daily returns of the two sector indices. We find that the most active sector in information exchange (i.e., the largest total information inflow and outflow) is the non-bank financial sector in the Chinese market and the technology sector in the USA market. This is consistent with the role of the non-bank sector in corporate financing in China and the impact of technological innovation in the USA. In each market, the most active sector is also the largest information sink that has the largest information inflow (i.e., inflow minus outflow). In contrast, we identify that the main information source is the bank sector in the Chinese market and the energy sector in the USA market. In the case of China, this is due to the importance of net bank lending as a signal of corporate activity and the role of energy pricing in affecting corporate profitability. There are sectors such as the real estate sector that could be an information sink in one market but an information source in the other, showing the complex behavior of different markets. Overall, these findings show that stock markets are more synchronized, or ordered, during periods of turmoil than during periods of stability.

Transfer entropy as a variable selection methodology of cryptocurrencies in the framework of a high dimensional predictive model

We determine the number of statistically significant factors in a high dimensional predictive model of cryptocurrencies using a random matrix test. The applied predictive model is of the reduced rank regression (RRR) type; in particular, we choose a flavor that can be regarded as canonical correlation analysis (CCA). A variable selection of hourly cryptocurrencies is performed using the Symbolic estimation of Transfer Entropy (STE) measure from information theory. In simulated studies, STE shows better performance compared to the Granger causality approach when considering a nonlinear system and a linear system with many drivers. In the application to cryptocurrencies, the directed graph associated to the variable selection shows a robust pattern of predictor and response clusters, where the community detection was contrasted with the modularity approach. Also, the centralities of the network discriminate between the two main types of cryptocurrencies, i.e., coins and tokens. On the factor determination of the predictive model, the result supports retaining more factors contrary to the usual visual inspection, with the additional advantage that the subjective element is avoided. In particular, it is observed that the dynamic behavior of the number of factors is moderately anticorrelated with the dynamics of the constructed composite index of predictor and response cryptocurrencies. This finding opens up new insights for anticipating possible declines in cryptocurrency prices on exchanges. Furthermore, our study suggests the existence of specific-predictor and specific-response factors, where only a small number of currencies are predominant.

Typical and Aberrant Functional Brain Flexibility: Lifespan Development and Aberrant Organization in Traumatic Brain Injury and Dyslexia

Intrinsic functional connectivity networks derived from different neuroimaging methods and connectivity estimators have revealed robust developmental trends linked to behavioural and cognitive maturation. The present study employed a dynamic functional connectivity approach to determine dominant intrinsic coupling modes in resting-state neuromagnetic data from 178 healthy participants aged 8-60 years. Results revealed significant developmental trends in three types of dominant intra- and inter-hemispheric neuronal population interactions (amplitude envelope, phase coupling, and phase-amplitude synchronization) involving frontal, temporal, and parieto-occipital regions. Multi-class support vector machines achieved 89% correct classification of participants according to their chronological age using dynamic functional connectivity indices. Moreover, systematic temporal variability in functional connectivity profiles, which was used to empirically derive a composite flexibility index, displayed an inverse U-shaped curve among healthy participants. Lower flexibility values were found among age-matched children with reading disability and adults who had suffered mild traumatic brain injury. The importance of these results for normal and abnormal brain development are discussed in light of the recently proposed role of cross-frequency interactions in the fine-grained coordination of neuronal population activity.

Synchronization transitions caused by time-varying coupling functions

Interacting dynamical systems are widespread in nature. The influence that one such system exerts on another is described by a coupling function; and the coupling functions extracted from the time-series of interacting dynamical systems are often found to be time-varying. Although much effort has been devoted to the analysis of coupling functions, the influence of time-variability on the associated dynamics remains largely unexplored. Motivated especially by coupling functions in biology, including the cardiorespiratory and neural delta-alpha coupling functions, this paper offers a contribution to the understanding of effects due to time-varying interactions. Through both numerics and mathematically rigorous theoretical consideration, we show that for time-variable coupling functions with time-independent net coupling strength, transitions into and out of phase- synchronization can occur, even though the frozen coupling functions determine phase-synchronization solely by virtue of their net coupling strength. Thus the information about interactions provided by the shape of coupling functions plays a greater role in determining behaviour when these coupling functions are time-variable. This article is part of the theme issue 'Coupling functions: dynamical interaction mechanisms in the physical, biological and social sciences'.

Information Flow between Bitcoin and Other Investment Assets

This paper studies the causal relationship between Bitcoin and other investment assets. We first test Granger causality and then calculate transfer entropy as an information-theoretic approach. Unlike the Granger causality test, we discover that transfer entropy clearly identifies causal interdependency between Bitcoin and other assets, including gold, stocks, and the U.S. dollar. However, for symbolic transfer entropy, the dynamic rise–fall pattern in return series shows an asymmetric information flow from other assets to Bitcoin. Our results imply that the Bitcoin market actively interacts with major asset markets, and its long-term equilibrium, as a nascent market, gradually synchronizes with that of other investment assets.

Evaluation of Granger Causality Measures for Constructing Networks from Multivariate Time Series

Granger causality and variants of this concept allow the study of complex dynamical systems as networks constructed from multivariate time series. In this work, a large number of Granger causality measures used to form causality networks from multivariate time series are assessed. These measures are in the time domain, such as model-based and information measures, the frequency domain, and the phase domain. The study aims also to compare bivariate and multivariate measures, linear and nonlinear measures, as well as the use of dimension reduction in linear model-based measures and information measures. The latter is particular relevant in the study of high-dimensional time series. For the performance of the multivariate causality measures, low and high dimensional coupled dynamical systems are considered in discrete and continuous time, as well as deterministic and stochastic. The measures are evaluated and ranked according to their ability to provide causality networks that match the original coupling structure. The simulation study concludes that the Granger causality measures using dimension reduction are superior and should be preferred particularly in studies involving many observed variables, such as multi-channel electroencephalograms and financial markets.

Teaching Ordinal Patterns to a Computer: Efficient Encoding Algorithms Based on the Lehmer Code

Ordinal patterns are the common basis of various techniques used in the study of dynamical systems and nonlinear time series analysis. The present article focusses on the computational problem of turning time series into sequences of ordinal patterns. In a first step, a numerical encoding scheme for ordinal patterns is proposed. Utilising the classical Lehmer code, it enumerates ordinal patterns by consecutive non-negative integers, starting from zero. This compact representation considerably simplifies working with ordinal patterns in the digital domain. Subsequently, three algorithms for the efficient extraction of ordinal patterns from time series are discussed, including previously published approaches that can be adapted to the Lehmer code. The respective strengths and weaknesses of those algorithms are discussed, and further substantiated by benchmark results. One of the algorithms stands out in terms of scalability: its run-time increases linearly with both the pattern order and the sequence length, while its memory footprint is practically negligible. These properties enable the study of high-dimensional pattern spaces at low computational cost. In summary, the tools described herein may improve the efficiency of virtually any ordinal pattern-based analysis method, among them quantitative measures like permutation entropy and symbolic transfer entropy, but also techniques like forbidden pattern identification. Moreover, the concepts presented may allow for putting ideas into practice that up to now had been hindered by computational burden. To enable smooth evaluation, a function library written in the C programming language, as well as language bindings and native implementations for various numerical computation environments are provided in the supplements.

Multifractal and entropy analysis of resting-state electroencephalography reveals spatial organization in local dynamic functional connectivity

Functional connectivity of the brain fluctuates even in resting-state condition. It has been reported recently that fluctuations of global functional network topology and those of individual connections between brain regions expressed multifractal scaling. To expand on these findings, in this study we investigated if multifractality was indeed an inherent property of dynamic functional connectivity (DFC) on the regional level as well. Furthermore, we explored if local DFC showed region-specific differences in its multifractal and entropy-related features. DFC analyses were performed on 62-channel, resting-state electroencephalography recordings of twelve young, healthy subjects. Surrogate data testing verified the true multifractal nature of regional DFC that could be attributed to the presumed nonlinear nature of the underlying processes. Moreover, we found a characteristic spatial distribution of local connectivity dynamics, in that frontal and occipital regions showed stronger long-range correlation and higher degree of multifractality, whereas the highest values of entropy were found over the central and temporal regions. The revealed topology reflected well the underlying resting-state network organization of the brain. The presented results and the proposed analysis framework could improve our understanding on how resting-state brain activity is spatio-temporally organized and may provide potential biomarkers for future clinical research.

Information-theoretic approach to detect directional information flow in EEG signals induced by TMS

Effective connectivity analysis has been widely applied to noninvasive recordings such as functional magnetic resonance imaging and electroencephalograms (EEGs). Previous studies have aimed to extract the causal relations between brain regions, but the validity of the derived connectivity has not yet been fully determined. This is because it is generally difficult to identify causality in the usual experimental framework based on observations alone. Transcranial magnetic stimulation (TMS) provides a framework in which a controllable perturbation is applied to a local brain region and the effect is examined by comparing the neural activity with and without this stimulation. This study evaluates two methods for effective connectivity analysis, symbolic transfer entropy (STE) and vector autoregression (VAR), by applying them to TMS-EEG data. In terms of the consistency of results from different experimental sessions, STE is found to yield robust results irrespective of sessions, whereas VAR produces less correlation between sessions. Furthermore, STE preferentially detects the directional information flow from the TMS target. Taken together, our results suggest that STE is a reliable method for detecting the effect of TMS, implying that it would also be useful for identifying neural activity during cognitive tasks and resting states.

Adaptive symbolic transfer entropy and its applications in modeling for complex industrial systems

Directed coupling between variables is the foundation of studying the dynamical behavior of complex systems. We propose an adaptive symbolic transfer entropy (ASTE) method based on the principle of equal probability division. First, the adaptive kernel density method is used to obtain an accurate probability density function for an observation series. Second, the complete phase space of the system can be obtained by using the multivariable phase space reconstruction method. This provides common parameters for symbolizing a time series, including delay time and embedding dimension. Third, an optimization strategy is used to select the appropriate symbolic parameters of a time series, such as the symbol set and partition intervals, which can be used to convert the time series to a symbol sequence. Then the transfer entropy between the symbolic sequences can be carried out. Finally, the proposed method is analyzed and validated using the chaotic Lorenz system and typical complex industrial systems. The results show that the ASTE method is superior to the existing transfer entropy and symbolic transfer entropy methods in terms of measurement accuracy and noise resistance, and it can be applied to the network modeling and performance safety analysis of complex industrial systems.

Causal stability and synchronization

Synchronization of chaos arises between coupled dynamical systems and is very well understood as a temporal phenomenon, which leads the coupled systems to converge or develop a dependence with time. In this work, we provide a complementary spatial perspective to this phenomenon by introducing the novel idea of causal stability. We then propose and prove a causal stability synchronization theorem as a necessary and sufficient condition for complete synchronization. We also provide an empirical criterion to identify synchronizing variables in coupled identical chaotic dynamical systems based on intrasystem causal influences estimated using time series data of the driving system alone. For this, a recently proposed measure, Compression-Complexity Causality (CCC), is used. The sign and magnitude of the estimated CCC value capture the nature of dynamical influences from each variable to rest of the subsystem and are thus able to determine whether or not the variable, when used to couple another system, will drive that system to synchronization.

Interpretation of Entropy Algorithms in the Context of Biomedical Signal Analysis and Their Application to EEG Analysis in Epilepsy

Biomedical signals are measurable time series that describe a physiological state of a biological system. Entropy algorithms have been previously used to quantify the complexity of biomedical signals, but there is a need to understand the relationship of entropy to signal processing concepts. In this study, ten synthetic signals that represent widely encountered signal structures in the field of signal processing were created to interpret permutation, modified permutation, sample, quadratic sample and fuzzy entropies. Subsequently, the entropy algorithms were applied to two different databases containing electroencephalogram (EEG) signals from epilepsy studies. Transitions from randomness to periodicity were successfully detected in the synthetic signals, while significant differences in EEG signals were observed based on different regions and states of the brain. In addition, using results from one entropy algorithm as features and the k-nearest neighbours algorithm, maximum classification accuracies in the first EEG database ranged from 63% to 73.5%, while these values increased by approximately 20% when using two different entropies as features. For the second database, maximum classification accuracy reached 62.5% using one entropy algorithm, while using two algorithms as features further increased that by 10%. Embedding entropies (sample, quadratic sample and fuzzy entropies) are found to outperform the rest of the algorithms in terms of sensitivity and show greater potential by considering the fine-tuning possibilities they offer. On the other hand, permutation and modified permutation entropies are more consistent across different input parameter values and considerably faster to calculate.

Transfer entropy for synchronized behavior estimation of interpersonal relationships in human communication: identifying leaders or followers

A person’s behavioral rhythms are synchronized spontaneously and unconsciously with those of other people, which often have positive effects, such as facilitating cooperation on tasks and promoting empathy for others. Although synchronization is induced by mutual interaction, it is unclear whether both individuals have the same influence. Is there a division of roles, in which some people are leaders and some followers? To address this, we calculated the transfer entropy (TE) of behavioral rhythms in a two-person cooperative tapping task, which provides an estimate of the direction of information propagation between two systems. We used TE to identify the causal relationship between two people (leader and follower); that is, the significant differences in the TE from one partner to another and vice versa. In this study, if there was a high TE from one individual (e.g., participant A) to the other individual (e.g., participant B), we defined participant A as the leader group and B as the follower group. First, using computer simulations, the programs which tapping intervals were almost independent with or were almost same with those of the partner programs were identified as the leader or follower, respectively, thereby confirming our hypothesis. Second, based on the results of the human experiment, we identified the leader and follower in some groups. Interestingly, the leader group showed a high systemizing quotient, which is related to communication deficits in developmental disorders such as autism. The results are consistent with participants’ subjective impressions of their partners. Our methods can be used to estimate the interpersonal division of roles in complex human communications.

Surrogate Data Preserving All the Properties of Ordinal Patterns up to a Certain Length

We propose a method for generating surrogate data that preserves all the properties of ordinal patterns up to a certain length, such as the numbers of allowed/forbidden ordinal patterns and transition likelihoods from ordinal patterns into others. The null hypothesis is that the details of the underlying dynamics do not matter beyond the refinements of ordinal patterns finer than a predefined length. The proposed surrogate data help construct a test of determinism that is free from the common linearity assumption for a null-hypothesis.

Media coverage and firearm acquisition in the aftermath of a mass shooting

With an alarming frequency, the United States is experiencing mass shooting events, which often result in heated public debates on firearm control. Whether such events play any role in recent dramatic increases in firearm prevalence remains an open question. This study adopts an information-theoretic framework to analyse the complex interplay between the occurrence of a mass shooting, media coverage on firearm control policies and firearm acquisition at both national and state levels. Through the analysis of time series from 1999 to 2017, we identify a correlation between the occurrence of a mass shooting and the rate of growth in firearm acquisition. More importantly, a transfer entropy analysis pinpoints media coverage on firearm control policies as a potential causal link in a Wiener-Granger sense that establishes this correlation. Our results demonstrate that media coverage may increase public worry about more stringent firearm control and partially drive increases in firearm prevalence.

Transfer entropy on symbolic recurrences

Recurrence quantification analysis offers a powerful framework to investigate complexity in dynamical systems. While several studies have demonstrated the possibility of multivariate recurrence quantification analysis, information-theoretic tools for the discovery of causal links remain elusive. Particularly enticing is to formulate information-theoretic tools on symbolic recurrence plots, which alleviate some of the methodological challenges of traditional recurrence plots and offer a richer representation of recurrences. Toward this aim, we establish a probability space in which we ground a theory of information that encodes information in the recurrences of the symbols. We introduce transfer entropy on symbolic recurrences as a tool to guide the inference of the strength and direction of the interaction between dynamical systems. We demonstrate statistically reliable discovery of causal links on synthetic and experimental time series, from only two time series or a larger dataset with multiple realizations. The proposed approach brings together recurrence plots, information theory, and symbolic dynamics to empower researchers and practitioners with effective means to visualize and quantify interactions in dynamical systems.

Transfer entropy computation using the Perron-Frobenius operator

We propose a method for computing the transfer entropy between time series using Ulam's approximation of the Perron-Frobenius (transfer) operator associated with the map generating the dynamics. Our method differs from standard transfer entropy estimators in that the invariant measure is estimated not directly from the data points, but from the invariant distribution of the transfer operator approximated from the data points. For sparse time series and low embedding dimension, the transfer operator is approximated using a triangulation of the attractor, whereas for data-rich time series or higher embedding dimension, we use a faster grid approach. We compare the performance of our methods with existing estimators such as the k nearest neighbors method and kernel density estimation method, using coupled instances of well known chaotic systems: coupled logistic maps and a coupled Rössler-Lorenz system. We find that our estimators are robust against moderate levels of noise. For sparse time series with less than 100 observations and low embedding dimension, our triangulation estimator shows improved ability to detect coupling directionality, relative to standard transfer entropy estimators.

Ordinal partition transition network based complexity measures for inferring coupling direction and delay from time series

It has been demonstrated that the construction of ordinal partition transition networks (OPTNs) from time series provides a prospective approach to improve our understanding of the underlying dynamical system. In this work, we introduce a suite of OPTN based complexity measures to infer the coupling direction between two dynamical systems from pairs of time series. For several examples of coupled stochastic processes, we demonstrate that our approach is able to successfully identify interaction delays of both unidirectional and bidirectional coupling configurations. Moreover, we show that the causal interaction between two coupled chaotic Hénon maps can be captured by the OPTN based complexity measures for a broad range of coupling strengths before the onset of synchronization. Finally, we apply our method to two real-world observational climate time series, disclosing the interaction delays underlying the temperature records from two distinct stations in Oxford and Vienna. Our results suggest that ordinal partition transition networks can be used as complementary tools for causal inference tasks and provide insights into the potentials and theoretical foundations of time series networks.

Reconstruction of dynamic networks with time-delayed interactions in the presence of fast-varying noises

Most complex social, biological and technological systems can be described by dynamic networks. Reconstructing network structures from measurable data is a fundamental problem in almost all interdisciplinary fields. Network nodes interact with each other and those interactions often have diversely distributed time delays. Accurate reconstruction of any targeted interaction to a node requires measured data of all its neighboring nodes together with information on the time delays of interactions from these neighbors. When networks are large, these data are often not available and time-delay factors are deeply hidden. Here we show that fast-varying noise can be of great help in solving these challenging problems. By computing suitable correlations, we can infer the intensity and time delay of any targeted interaction with the data of two related nodes (driving and driven nodes) only while all other nodes in the network are hidden. This method is analytically derived and fully justified by extensive numerical simulations.

Information flow reveals prediction limits in online social activity

Modern society depends on the flow of information over online social networks, and users of popular platforms generate substantial behavioural data about themselves and their social ties^1-5. However, it remains unclear what fundamental limits exist when using these data to predict the activities and interests of individuals, and to what accuracy such predictions can be made using an individual's social ties. Here, we show that 95% of the potential predictive accuracy for an individual is achievable using their social ties only, without requiring that individual's data. We used information theoretic tools to estimate the predictive information in the writings of Twitter users, providing an upper bound on the available predictive information that holds for any predictive or machine learning methods. As few as 8-9 of an individual's contacts are sufficient to obtain predictability compared with that of the individual alone. Distinct temporal and social effects are visible by measuring information flow along social ties, allowing us to better study the dynamics of online activity. Our results have distinct privacy implications: information is so strongly embedded in a social network that, in principle, one can profile an individual from their available social ties even when the individual forgoes the platform completely.

A study for multiscale information transfer measures based on conditional mutual information

As the big data science develops, efficient methods are demanded for various data analysis. Granger causality provides the prime model for quantifying causal interactions. However, this theoretic model does not meet the requirement for real-world data analysis, because real-world time series are diverse whose models are usually unknown. Therefore, model-free measures such as information transfer measures are strongly desired. Here, we propose the multi-scale extension of conditional mutual information measures using MORLET wavelet, which are named the WM and WPM. The proposed measures are computational efficient and interpret information transfer by multi-scales. We use both synthetic data and real-world examples to demonstrate the efficiency of the new methods. The results of the new methods are robust and reliable. Via the simulation studies, we found the new methods outperform the wavelet extension of transfer entropy (WTE) in both computational efficiency and accuracy. The features and properties of the proposed measures are also discussed.

Role of Network Science in the Study of Anesthetic State Transitions

The heterogeneity of molecular mechanisms, target neural circuits, and neurophysiologic effects of general anesthetics makes it difficult to develop a reliable and drug-invariant index of general anesthesia. No single brain region or mechanism has been identified as the neural correlate of consciousness, suggesting that consciousness might emerge through complex interactions of spatially and temporally distributed brain functions. The goal of this review article is to introduce the basic concepts of networks and explain why the application of network science to general anesthesia could be a pathway to discover a fundamental mechanism of anesthetic-induced unconsciousness. This article reviews data suggesting that reduced network efficiency, constrained network repertoires, and changes in cortical dynamics create inhospitable conditions for information processing and transfer, which lead to unconsciousness. This review proposes that network science is not just a useful tool but a necessary theoretical framework and method to uncover common principles of anesthetic-induced unconsciousness.

Inference of time-varying networks through transfer entropy, the case of a Boolean network model

Inferring network topologies from the time series of individual units is of paramount importance in the study of biological and social networks. Despite considerable progress, our success in network inference is largely limited to static networks and autonomous node dynamics, which are often inadequate to describe complex systems. Here, we explore the possibility of reconstructing time-varying weighted topologies through the information-theoretic notion of transfer entropy. We focus on a Boolean network model in which the weight of the links and the spontaneous activity periodically vary in time. For slowly-varying dynamics, we establish closed-form expressions for the stationary periodic distribution and transfer entropy between each pair of nodes. Our results indicate that the instantaneous weight of each link is mapped into a corresponding transfer entropy value, thereby affording the possibility of pinpointing the dominant weights at each time. However, comparing transfer entropy readings at different times may provide erroneous estimates of the strength of the links in time, due to a counterintuitive modulation of the information flow by the non-autonomous dynamics. In fact, this time variation should be used to scale transfer entropy values toward the correct inference of the time evolution of the network weights. This study constitutes a necessary step toward a mathematically-principled use of transfer entropy to reconstruct time-varying networks.

Optimization of relative parameters in transfer entropy estimation and application to corticomuscular coupling in humans

BACKGROUND

As a non-modeled information theoretical measure, the transfer entropy (TE) could be applied to quantitatively analyze the linear and nonlinear coupling characteristics between two observations. However, the parameters selection of TE (the parameters used in state space reconstruction and estimating Shannon entropy) has a serious influence on the accuracy of its results.

NEW METHOD

In this study, the hybrid particle swarm optimization (HPSO) was applied to improve the accuracy of TE by optimizing its parameters. In HPSO, the TE calculation and significant analysis were integrated into the fitness function, and the optimal parameters group within the parameter space could be automatically found through an iteration process.

RESULTS

The TE results computed under the parameters optimized by HPSO (HPSO-TE), was assessed with a numerical non-linear model, the neural mass model and the recorded electroencephalogram (EEG) and electromyogram (EMG) signals. Compared with TE, HPSO-TE could reduce the 'false positive' in non-linear model, and 'spurious coupling', i.e. two nonzero TEs for unidirectionally coupled systems, especially when coupling strength was weak. The robustness against noise and long time-delay was improved. Moreover, the experimental data analysis showed HPSO-TE revealed the dominant direction (EEG → EMG) in corticomuscular coupling, and had higher values than TE which showed the same dominant direction.

COMPARISON WITH EXISTING METHOD

The implication of HPSO improved the accuracy of TE in estimating the coupling strength and direction.

CONCLUSIONS

The efficiency of TE could be improved by HPSO for estimating coupling relationships, especially for weakly coupled, strong noisy and long time-delay series.

Allosterism and signal transfer in DNA

We analysed the basic mechanisms of signal transmission in DNA and the origins of the allostery exhibited by systems such as the ternary complex BAMHI-DNA-GRDBD. We found that perturbation information generated by a primary protein binding event travels as a wave to distant regions of DNA following a hopping mechanism. However, such a structural perturbation is transient and does not lead to permanent changes in the DNA geometry and interaction properties at the secondary binding site. The BAMHI-DNA-GRDBD allosteric mechanism does not occur through any traditional models: direct (protein-protein), indirect (reorganization of the secondary site) readout or solvent-release. On the contrary, it is generated by a subtle and less common entropy-mediated mechanism, which might have an important role to explain other DNA-mediated cooperative effects.

Symbolic Entropy Analysis and Its Applications

This editorial explains the scope of the special issue and provides a thematic introduction to the contributed papers.

Impact of type of intracranial EEG sensors on link strengths of evolving functional brain networks

Objective and Approach: Investigating properties of evolving functional brain networks has become a valuable tool to characterize the complex dynamics of the epileptic brain. Such networks are usually derived from electroencephalograms (EEG) recorded with sensors implanted chronically into deeper structures of the brain and/or placed onto the cortex. It is still unclear, however, whether the use of different sensors for an identification of network nodes affects properties of functional brain networks. We address this question by investigating properties of links of such networks that we characterize by assessing interactions in multi-sensor, multi-day EEG data recorded from 49 epilepsy patients during presurgical evaluation. These data allow us to study the impact of different types of sensors together with the impact of various physiologic and pathophysiologic activities on the properties of links.

MAIN RESULTS

We observe that different types of sensors differently impact on spatial means and temporal fluctuations of link strengths. Moreover, the impact depends on the relative anatomical location of sensors with respect to location and extent of sources of the prevailing activities.

SIGNIFICANCE

Type and location of sensors should be considered when constructing networks.

Joint dyadic action: Error correction by two persons works better than by one alone

We investigated how two people learn to coordinate their movement to achieve a joint goal. Pairs of participants oscillated a joystick with their dominant hand whilst looking at a common feedback, a Lissajous figure, where each participant controlled either the vertical or horizontal coordinate of a moving dot. In the absence of specific instructions, inter-personal coordination was highly variable, punctuated by intermittent phase locking. When participants were required to produce a circular Lissajous figure, coordination variability decreased while accuracy, transfer entropy and the incidence of stable coordinative solutions (fixed points, including bi-stability) increased as a function of practice trials. When one partner closed his/her eyes, so that the other one received the full control of error correction, the stability and accuracy of coordination decreased. A questionnaire showed that partners experienced the feeling of we-control. The results were interpreted in terms of a disturbance ∼ correction challenge: joint action is enhanced by having a flexibly adjusting co-actor rather than a more predictable, but not adjusting, partner. At transfer, partners were able to produce a new, never-practiced Lissajous pattern, evidencing the generalisability of joint learning.

Causal inference in nonlinear systems: Granger causality versus time-delayed mutual information

The Granger causality (GC) analysis has been extensively applied to infer causal interactions in dynamical systems arising from economy and finance, physics, bioinformatics, neuroscience, social science, and many other fields. In the presence of potential nonlinearity in these systems, the validity of the GC analysis in general is questionable. To illustrate this, here we first construct minimal nonlinear systems and show that the GC analysis fails to infer causal relations in these systems-it gives rise to all types of incorrect causal directions. In contrast, we show that the time-delayed mutual information (TDMI) analysis is able to successfully identify the direction of interactions underlying these nonlinear systems. We then apply both methods to neuroscience data collected from experiments and demonstrate that the TDMI analysis but not the GC analysis can identify the direction of interactions among neuronal signals. Our work exemplifies inference hazards in the GC analysis in nonlinear systems and suggests that the TDMI analysis can be an appropriate tool in such a case.

Complexity Changes in Brain Activity in Healthy Ageing: A Permutation Lempel-Ziv Complexity Study of Magnetoencephalograms

Maturation and ageing, which can be characterised by the dynamic changes in brain morphology, can have an impact on the physiology of the brain. As such, it is possible that these changes can have an impact on the magnetic activity of the brain recorded using magnetoencephalography. In this study changes in the resting state brain (magnetic) activity due to healthy ageing were investigated by estimating the complexity of magnetoencephalogram (MEG) signals. The main aim of this study was to identify if the complexity of background MEG signals changed significantly across the human lifespan for both males and females. A sample of 177 healthy participants (79 males and 98 females aged between 21 and 80 and grouped into 3 categories i.e., early-, mid- and late-adulthood) was used in this investigation. This investigation also extended to evaluating if complexity values remained relatively stable during the 5 min recording. Complexity was estimated using permutation Lempel-Ziv complexity, a recently introduced complexity metric, with a motif length of 5 and a lag of 1. Effects of age and gender were investigated in the MEG channels over 5 brain regions, i.e., anterior, central, left lateral, posterior, and, right lateral, with highest complexity values observed in the signals recorded by the channels over the anterior and central regions of the brain. Results showed that while changes due to age had a significant effect on the complexity of the MEG signals recorded over 5 brain regions, gender did not have a significant effect on complexity values in all age groups investigated. Moreover, although some changes in complexity were observed between the different minutes of recording, due to the small magnitude of the changes it was concluded that practical significance might outweigh statistical significance in this instance. The results from this study can contribute to form a fingerprint of the characteristics of healthy ageing in MEGs that could be useful when investigating changes to the resting state activity due to pathology.

Transient and equilibrium causal effects in coupled oscillators

Two quite different types of causal effects are given by (i) changes in near future states of a driven system under changes in a current state of a driving system and (ii) changes in statistical characteristics of a driven system dynamics under changes in coupling parameters, e.g., under switching the coupling off. The former can be called transient causal effects and can be estimated from a time series within the well established framework of the Wiener-Granger causality, while the latter represent equilibrium (or stationary) causal effects which are often most interesting but generally inaccessible to estimation from an observed time series recorded at fixed coupling parameters. In this work, relationships between the two kinds of causal effects are found for unidirectionally coupled stochastic linear oscillators depending on their frequencies and damping factors. Approximate closed-form expressions for these relationships are derived. Their limitations and possible extensions are discussed, and their practical applicability to extracting equilibrium causal effects from time series is argued.

Causal network reconstruction from time series: From theoretical assumptions to practical estimation

Causal network reconstruction from time series is an emerging topic in many fields of science. Beyond inferring directionality between two time series, the goal of causal network reconstruction or causal discovery is to distinguish direct from indirect dependencies and common drivers among multiple time series. Here, the problem of inferring causal networks including time lags from multivariate time series is recapitulated from the underlying causal assumptions to practical estimation problems. Each aspect is illustrated with simple examples including unobserved variables, sampling issues, determinism, stationarity, nonlinearity, measurement error, and significance testing. The effects of dynamical noise, autocorrelation, and high dimensionality are highlighted in comparison studies of common causal reconstruction methods. Finally, method performance evaluation approaches and criteria are suggested. The article is intended to briefly review and accessibly illustrate the foundations and practical problems of time series-based causal discovery and stimulate further methodological developments.

Causation and information flow with respect to relative entropy

Recently, a rigorous formalism has been established for information flow and causality within dynamical systems with respect to Shannon entropy. In this study, we re-establish the formalism with respect to relative entropy, or Kullback-Leiber divergence, a well-accepted measure of predictability because of its appealing properties such as invariance upon nonlinear transformation and consistency with the second law of thermodynamics. Different from previous studies (which yield consistent results only for 2D systems), the resulting information flow, say T, is precisely the same as that with respect to Shannon entropy for systems of arbitrary dimensionality, except for a minus sign (reflecting the opposite notion of predictability vs. uncertainty). As before, T possesses a property called principle of nil causality, a fact that classical formalisms fail to verify in many situation. Besides, it proves to be invariant upon nonlinear transformation, indicating that the so-obtained information flow should be an intrinsic physical property. This formalism has been validated with the stochastic gradient system, a nonlinear system that admits an analytical equilibrium solution of the Boltzmann type.

The quoter model: A paradigmatic model of the social flow of written information

We propose a model for the social flow of information in the form of text data, which simulates the posting and sharing of short social media posts. Nodes in a graph representing a social network take turns generating words, leading to a symbolic time series associated with each node. Information propagates over the graph via a quoting mechanism, where nodes randomly copy short segments of text from each other. We characterize information flows from these text via information-theoretic estimators, and we derive analytic relationships between model parameters and the values of these estimators. We explore and validate the model with simulations on small network motifs and larger random graphs. Tractable models such as ours that generate symbolic data while controlling the information flow allow us to test and compare measures of information flow applicable to real social media data. In particular, by choosing different network structures, we can develop test scenarios to determine whether or not measures of information flow can distinguish between true and spurious interactions, and how topological network properties relate to information flow.

Open or closed? Information flow decided by transfer operators and forecastability quality metric

A basic systems question concerns the concept of closure, meaning autonomy (closed) in the sense of describing the (sub)system as fully consistent within itself. Alternatively, the system may be nonautonomous (open), meaning it receives influence from an outside subsystem. We assert here that the concept of information flow and the related concept of causation inference are summarized by this simple question of closure as we define herein. We take the forecasting perspective of Weiner-Granger causality that describes a causal relationship exists if a subsystem's forecast quality depends on considering states of another subsystem. Here, we develop a new direct analytic discussion, rather than a data oriented approach. That is, we refer to the underlying Frobenius-Perron (FP) transfer operator that moderates evolution of densities of ensembles of orbits, and two alternative forms of the restricted Frobenius-Perron operator, interpreted as if either closed (deterministic FP) or not closed (the unaccounted outside influence seems stochastic and we show correspondingly requires the stochastic FP operator). Thus follows contrasting the kernels of the variants of the operators, as if densities in their own rights. However, the corresponding differential entropy comparison by Kullback-Leibler divergence, as one would typically use when developing transfer entropy, becomes ill-defined. Instead, we build our Forecastability Quality Metric (FQM) upon the "symmetrized" variant known as Jensen-Shannon divergence, and we are also able to point out several useful resulting properties. We illustrate the FQM by a simple coupled chaotic system. Our analysis represents a new theoretical direction, but we do describe data oriented directions for the future.