A Tutorial for Information Theory in Neuroscience

Limitations to Estimating Mutual Information in Large Neural Populations

Information theory provides a powerful framework to analyse the representation of sensory stimuli in neural population activity. However, estimating the quantities involved such as entropy and mutual information from finite samples is notoriously hard and any direct estimate is known to be heavily biased. This is especially true when considering large neural populations. We study a simple model of sensory processing and show through a combinatorial argument that, with high probability, for large neural populations any finite number of samples of neural activity in response to a set of stimuli is mutually distinct. As a consequence, the mutual information when estimated directly from empirical histograms will be equal to the stimulus entropy. Importantly, this is the case irrespective of the precise relation between stimulus and neural activity and corresponds to a maximal bias. This argument is general and applies to any application of information theory, where the state space is large and one relies on empirical histograms. Overall, this work highlights the need for alternative approaches for an information theoretic analysis when dealing with large neural populations.

Deriving pairwise transfer entropy from network structure and motifs

Transfer entropy (TE) is an established method for quantifying directed statistical dependencies in neuroimaging and complex systems datasets. The pairwise (or bivariate) TE from a source to a target node in a network does not depend solely on the local source-target link weight, but on the wider network structure that the link is embedded in. This relationship is studied using a discrete-time linearly coupled Gaussian model, which allows us to derive the TE for each link from the network topology. It is shown analytically that the dependence on the directed link weight is only a first approximation, valid for weak coupling. More generally, the TE increases with the in-degree of the source and decreases with the in-degree of the target, indicating an asymmetry of information transfer between hubs and low-degree nodes. In addition, the TE is directly proportional to weighted motif counts involving common parents or multiple walks from the source to the target, which are more abundant in networks with a high clustering coefficient than in random networks. Our findings also apply to Granger causality, which is equivalent to TE for Gaussian variables. Moreover, similar empirical results on random Boolean networks suggest that the dependence of the TE on the in-degree extends to nonlinear dynamics.

Statistical Approaches for the Analysis of Dependency Among Neurons Under Noise

Neuronal noise is a major factor affecting the communication between coupled neurons. In this work, we propose a statistical toolset to infer the coupling between two neurons under noise. We estimate these statistical dependencies from data which are generated by a coupled Hodgkin-Huxley (HH) model with additive noise. To infer the coupling using observation data, we employ copulas and information-theoretic quantities, such as the mutual information (MI) and the transfer entropy (TE). Copulas and MI between two variables are symmetric quantities, whereas TE is asymmetric. We demonstrate the performances of copulas and MI as functions of different noise levels and show that they are effective in the identification of the interactions due to coupling and noise. Moreover, we analyze the inference of TE values between neurons as a function of noise and conclude that TE is an effective tool for finding out the direction of coupling between neurons under the effects of noise.

Application of information theory in systems biology

Over recent years, new light has been shed on aspects of information processing in cells. The quantification of information, as described by Shannon’s information theory, is a basic and powerful tool that can be applied to various fields, such as communication, statistics, and computer science, as well as to information processing within cells. It has also been used to infer the network structure of molecular species. However, the difficulty of obtaining sufficient sample sizes and the computational burden associated with the high-dimensional data often encountered in biology can result in bottlenecks in the application of information theory to systems biology. This article provides an overview of the application of information theory to systems biology, discussing the associated bottlenecks and reviewing recent work.

Decoding collective communications using information theory tools

Organisms have evolved sensory mechanisms to extract pertinent information from their environment, enabling them to assess their situation and act accordingly. For social organisms travelling in groups, like the fish in a school or the birds in a flock, sharing information can further improve their situational awareness and reaction times. Data on the benefits and costs of social coordination, however, have largely allowed our understanding of why collective behaviours have evolved to outpace our mechanistic knowledge of how they arise. Recent studies have begun to correct this imbalance through fine-scale analyses of group movement data. One approach that has received renewed attention is the use of information theoretic (IT) tools like mutual information, transfer entropy and causation entropy, which can help identify causal interactions in the type of complex, dynamical patterns often on display when organisms act collectively. Yet, there is a communications gap between studies focused on the ecological constraints and solutions of collective action with those demonstrating the promise of IT tools in this arena. We attempt to bridge this divide through a series of ecologically motivated examples designed to illustrate the benefits and challenges of using IT tools to extract deeper insights into the interaction patterns governing group-level dynamics. We summarize some of the approaches taken thus far to circumvent existing challenges in this area and we conclude with an optimistic, yet cautionary perspective.

Coherent feed-forward loop acts as an efficient information transmitting motif

We present a theoretical formalism to study steady-state information transmission in a coherent type-1 feed-forward loop motif with an additive signal integration mechanism. Our construct allows a two-step cascade to be slowly transformed into a bifurcation network via a feed-forward loop, which is a prominent network motif. Using a Gaussian framework, we show that among these three network patterns, the feed-forward loop motif harnesses the maximum amount of Shannon mutual information fractions constructed between the final gene-product and each of the master and coregulators of the target gene. We also show that this feed-forward loop motif provides a substantially lower amount of noise in target gene expression, compared with the other two network structures. Our theoretical predictions, which remain invariant for a couple of parametric transformations, point out that the coherent type-1 feed-forward loop motif may qualify as a better decoder of environmental signals when compared with the other two network patterns in perspective.

Information-Theoretic Model and Analysis of Molecular Signaling in Targeted Drug Delivery

Targeted drug delivery (TDD) modality promises a smart localization of appropriate dose of therapeutic drugs to the targeted part of the body at reduced system toxicity. To achieve the desired goals of TDD, accurate analysis of the system is important. Recent advances in molecular communication (MC) present prospects to analyzing the TDD process using engineering concepts and tools. Specifically, the MC platform supports the abstraction of TDD process as a communication engineering problem in which the injection and transportation of drug particles in the human body and the delivery to a specific tissue or organ can be analyzed using communication engineering tools. In this paper we stand on the MC platform to present the information-theoretic model and analysis of the TDD systems. We present a modular structure of the TDD system and the probabilistic models of the MC-abstracted modules in an intuitive manner. Simulated results of information-theoretic measures such as the mutual information are employed to analyze the performance of the TDD system. Results indicate that uncertainties in drug injection/release systems, nanoparticles propagation channel and nanoreceiver systems influence the mutual information of the system, which is relative to the system's bioequivalence measure.

A Data-Driven Measure of Effective Connectivity Based on Renyi's α-Entropy

Transfer entropy (TE) is a model-free effective connectivity measure based on information theory. It has been increasingly used in neuroscience because of its ability to detect unknown non-linear interactions, which makes it well suited for exploratory brain effective connectivity analyses. Like all information theoretic quantities, TE is defined regarding the probability distributions of the system under study, which in practice are unknown and must be estimated from data. Commonly used methods for TE estimation rely on a local approximation of the probability distributions from nearest neighbor distances, or on symbolization schemes that then allow the probabilities to be estimated from the symbols' relative frequencies. However, probability estimation is a challenging problem, and avoiding this intermediate step in TE computation is desirable. In this work, we propose a novel TE estimator using functionals defined on positive definite and infinitely divisible kernels matrices that approximate Renyi's entropy measures of order α. Our data-driven approach estimates TE directly from data, sidestepping the need for probability distribution estimation. Also, the proposed estimator encompasses the well-known definition of TE as a sum of Shannon entropies in the limiting case when α → 1. We tested our proposal on a simulation framework consisting of two linear models, based on autoregressive approaches and a linear coupling function, respectively, and on the public electroencephalogram (EEG) database BCI Competition IV, obtained under a motor imagery paradigm. For the synthetic data, the proposed kernel-based TE estimation method satisfactorily identifies the causal interactions present in the data. Also, it displays robustness to varying noise levels and data sizes, and to the presence of multiple interaction delays in the same connected network. Obtained results for the motor imagery task show that our approach codes discriminant spatiotemporal patterns for the left and right-hand motor imagination tasks, with classification performances that compare favorably to the state-of-the-art.

Restricted information in a two-step cascade

A cell must sense extracellular and intracellular fluctuations and respond appropriately to survive for optimal cellular functioning. Accordingly, a cell builds up biochemical networks which can transduce information of extracellular and intracellular fluctuations accurately. We consider a generic two-step cascade as a model gene regulatory network containing three regulatory proteins S, X, and Y connected as S→X→Y. The intermediate node X is a stochastic variable, acts as an obstacle, and impedes the information flow from S to Y. We quantify the information that is restricted by X using the tools of information theory and term this as restricted information. In this context, we further propose two measurable quantities, restricted efficiency and information transfer efficiency. The former determines how efficiently X restricts the upstream information coming from S, while the latter computes the efficiency of X to pass the upstream information toward Y. We also quantify the information that is being uniquely transferred from X to Y, which determines the extent of the ability of X to act as a source of information. Our analysis shows that when the signal strength (or mean population of S, 〈s〉) is low, the intermediate X can carry forward the upstream information reliably as well, as it acts as a better source of information, thereby increasing the fidelity of the network. But at the high signal strength, X restricts most of the upstream information, and its ability to act as a source of information gets reduced. This leads to a loss of fidelity of the network.

Causal Composition: Structural Differences among Dynamically Equivalent Systems

The dynamical evolution of a system of interacting elements can be predicted in terms of its elementary constituents and their interactions, or in terms of the system’s global state transitions. For this reason, systems with equivalent global dynamics are often taken to be equivalent for all relevant purposes. Nevertheless, such systems may still vary in their causal composition—the way mechanisms within the system specify causes and effects over different subsets of system elements. We demonstrate this point based on a set of small discrete dynamical systems with reversible dynamics that cycle through all their possible states. Our analysis elucidates the role of composition within the formal framework of integrated information theory. We show that the global dynamical and information-theoretic capacities of reversible systems can be maximal even though they may differ, quantitatively and qualitatively, in the information that their various subsets specify about each other (intrinsic information). This can be the case even for a system and its time-reversed equivalent. Due to differences in their causal composition, two systems with equivalent global dynamics may still differ in their capacity for autonomy, agency, and phenomenology.

Thermodynamics Beyond Molecules: Statistical Thermodynamics of Probability Distributions

Statistical thermodynamics has a universal appeal that extends beyond molecular systems, and yet, as its tools are being transplanted to fields outside physics, the fundamental question, what is thermodynamics, has remained unanswered. We answer this question here. Generalized statistical thermodynamics is a variational calculus of probability distributions. It is independent of physical hypotheses but provides the means to incorporate our knowledge, assumptions and physical models about a stochastic processes that gives rise to the probability in question. We derive the familiar calculus of thermodynamics via a probabilistic argument that makes no reference to physics. At the heart of the theory is a space of distributions and a special functional that assigns probabilities to this space. The maximization of this functional generates the mathematical network of thermodynamic relationship. We obtain statistical mechanics as a special case and make contact with Information Theory and Bayesian inference.

Encoding of the Intent to Drink Alcohol by the Prefrontal Cortex Is Blunted in Rats with a Family History of Excessive Drinking

The prefrontal cortex (PFC) plays a central role in guiding decision making, and its function is altered by alcohol use and an individual's innate risk for excessive alcohol drinking. The primary goal of this work was to determine how neural activity in the PFC guides the decision to drink. Towards this goal, the within-session changes in neural activity were measured from medial PFC (mPFC) of rats performing a drinking procedure that allowed them to consume or abstain from alcohol in a self-paced manner. Recordings were obtained from rats that either lacked or expressed an innate risk for excessive alcohol intake, Wistar or alcohol-preferring (P) rats, respectively. Wistar rats exhibited patterns of neural activity consistent with the intention to drink or abstain from drinking, whereas these patterns were blunted or absent in P rats. Collectively, these data indicate that neural activity patterns in mPFC associated with the intention to drink alcohol are influenced by innate risk for excessive alcohol drinking. This observation may indicate a lack of control over the decision to drink by this otherwise well-validated supervisory brain region.

Genetic absence epilepsy: Effective connectivity from piriform cortex to mediodorsal thalamus

OBJECTIVE

The objective of the study was to quantify effective connectivity from the piriform cortex to mediodorsal thalamus, in Genetic Absence Epilepsy Rats from Strasbourg (GAERS).

METHODS

Local field potentials (LFPs) were recorded using microelectrode arrays implanted in the mediodorsal thalamus and piriform cortex, in three urethane anesthetized GAERS and three control rats. Screw electrodes were placed in the primary motor cortex to identify epileptiform discharges. We used transfer entropy to measure effective connectivity from piriform cortex to mediodorsal thalamus prior to and during generalized epileptiform discharges.

RESULTS

We observed increased theta band effective connectivity from piriform cortex to mediodorsal thalamus, prior to and during epileptiform discharges in GAERS compared with controls. Increased effective connectivity was also observed in beta and gamma bands from the piriform cortex to mediodorsal thalamus, but only during epileptiform discharges.

CONCLUSIONS

This preliminary study suggests that increased effective theta connectivity from the piriform cortex to the mediodorsal thalamus may be a feature of the 'epileptic network' associated with genetic absence epilepsy. Our findings indicate an underlying predisposition of this direct pathway to propagate epileptiform discharges in genetic absence epilepsy.

Comparing Open-Source Toolboxes for Processing and Analysis of Spike and Local Field Potentials Data

Analysis of spike and local field potential (LFP) data is an essential part of neuroscientific research. Today there exist many open-source toolboxes for spike and LFP data analysis implementing various functionality. Here we aim to provide a practical guidance for neuroscientists in the choice of an open-source toolbox best satisfying their needs. We overview major open-source toolboxes for spike and LFP data analysis as well as toolboxes with tools for connectivity analysis, dimensionality reduction and generalized linear modeling. We focus on comparing toolboxes functionality, statistical and visualization tools, documentation and support quality. To give a better insight, we compare and illustrate functionality of the toolboxes on open-access dataset or simulated data and make corresponding MATLAB scripts publicly available.

Effects of Inducing Gamma Oscillations in Hippocampal Subregions DG, CA3, and CA1 on the Potential Alleviation of Alzheimer's Disease-Related Pathology: Computer Modeling and Simulations

The aim of this study was to evaluate the possibility of the gamma oscillation function (40–130 Hz) to reduce Alzheimer’s disease related pathology in a computer model of the hippocampal network dentate gyrus, CA3, and CA1 (DG-CA3-CA1) regions. Methods: Computer simulations were made for a pathological model in which Alzheimer’s disease was simulated by synaptic degradation in the hippocampus. Pathology modeling was based on sequentially turning off the connections with entorhinal cortex layer 2 (EC2) and the dentate gyrus on CA3 pyramidal neurons. Gamma induction modeling consisted of simulating the oscillation provided by the septo-hippocampal pathway with band frequencies from 40–130 Hz. Pathological models with and without gamma induction were compared with a control. Results: In the hippocampal regions of DG, CA3, and CA1, and jointly DG-CA3-CA1 and CA3-CA1, gamma induction resulted in a statistically significant improvement in terms of increased numbers of spikes, spikes per burst, and burst duration as compared with the model simulating Alzheimer’s disease (AD). The positive maximal Lyapunov exponent was negative in both the control model and the one with gamma induction as opposed to the pathological model where it was positive within the DG-CA3-CA1 region. Gamma induction resulted in decreased transfer entropy in accordance with the information flow in DG → CA3 and CA3 → CA1. Conclusions: The results of simulation studies show that inducing gamma oscillations in the hippocampus may reduce Alzheimer’s disease related pathology. Pathologically higher transfer entropy values after gamma induction returned to values comparable to the control model.

Computer Model of Synapse Loss During an Alzheimer's Disease-Like Pathology in Hippocampal Subregions DG, CA3 and CA1-The Way to Chaos and Information Transfer

The aim of the study was to compare the computer model of synaptic breakdown in an Alzheimer’s disease-like pathology in the dentate gyrus (DG), CA3 and CA1 regions of the hippocampus with a control model using neuronal parameters and methods describing the complexity of the system, such as the correlative dimension, Shannon entropy and positive maximal Lyapunov exponent. The model of synaptic breakdown (from 13% to 50%) in the hippocampus modeling the dynamics of an Alzheimer’s disease-like pathology was simulated. Modeling consisted in turning off one after the other EC2 connections and connections from the dentate gyrus on the CA3 pyramidal neurons. The pathological model of synaptic disintegration was compared to a control. The larger synaptic breakdown was associated with a statistically significant decrease in the number of spikes (R = −0.79, P < 0.001), spikes per burst (R = −0.76, P < 0.001) and burst duration (R = −0.83, P < 0.001) and an increase in the inter-burst interval (R = 0.85, P < 0.001) in DG-CA3-CA1. The positive maximal Lyapunov exponent in the control model was negative, but in the pathological model had a positive value of DG-CA3-CA1. A statistically significant decrease of Shannon entropy with the direction of information flow DG->CA3->CA1 (R = −0.79, P < 0.001) in the pathological model and a statistically significant increase with greater synaptic breakdown (R = 0.24, P < 0.05) of the CA3-CA1 region was obtained. The reduction of entropy transfer for DG->CA3 at the level of synaptic breakdown of 35% was 35%, compared with the control. Entropy transfer for CA3->CA1 at the level of synaptic breakdown of 35% increased to 95% relative to the control. The synaptic breakdown model in an Alzheimer’s disease-like pathology in DG-CA3-CA1 exhibits chaotic features as opposed to the control. Synaptic breakdown in which an increase of Shannon entropy is observed indicates an irreversible process of Alzheimer’s disease. The increase in synapse loss resulted in decreased information flow and entropy transfer in DG->CA3, and at the same time a strong increase in CA3->CA1.