Synchronization in random balanced networks

Nonorthogonal Eigenvectors, Fluctuation-Dissipation Relations, and Entropy Production

Celebrated fluctuation-dissipation theorem (FDT) linking the response function to time dependent correlations of observables measured in the reference unperturbed state is one of the central results in equilibrium statistical mechanics. In this Letter we discuss an extension of the standard FDT to the case when multidimensional matrix representing transition probabilities is strictly non-normal. This feature dramatically modifies the dynamics, by incorporating the effect of eigenvector nonorthogonality via the associated overlap matrix of Chalker-Mehlig type. In particular, the rate of entropy production per unit time is strongly enhanced by that matrix. We suggest, that this mechanism has an impact on the studies of collective phenomena in neural matrix models, leading, via transient behavior, to such phenomena as synchronization and emergence of the memory. We also expect, that the described mechanism generating the entropy production is generic for wide class of phenomena, where dynamics is driven by non-normal operators. For the case of driving by a large Ginibre matrix the entropy production rate is evaluated analytically, as well as for the Rajan-Abbott model for neural networks.

Effect in the spectra of eigenvalues and dynamics of RNNs trained with excitatory-inhibitory constraint

In order to comprehend and enhance models that describes various brain regions it is important to study the dynamics of trained recurrent neural networks. Including Dale's law in such models usually presents several challenges. However, this is an important aspect that allows computational models to better capture the characteristics of the brain. Here we present a framework to train networks using such constraint. Then we have used it to train them in simple decision making tasks. We characterized the eigenvalue distributions of the recurrent weight matrices of such networks. Interestingly, we discovered that the non-dominant eigenvalues of the recurrent weight matrix are distributed in a circle with a radius less than 1 for those whose initial condition before training was random normal and in a ring for those whose initial condition was random orthogonal. In both cases, the radius does not depend on the fraction of excitatory and inhibitory units nor the size of the network. Diminution of the radius, compared to networks trained without the constraint, has implications on the activity and dynamics that we discussed here.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-023-09956-w.

Exploring weight initialization, diversity of solutions, and degradation in recurrent neural networks trained for temporal and decision-making tasks

Recurrent Neural Networks (RNNs) are frequently used to model aspects of brain function and structure. In this work, we trained small fully-connected RNNs to perform temporal and flow control tasks with time-varying stimuli. Our results show that different RNNs can solve the same task by converging to different underlying dynamics and also how the performance gracefully degrades as either network size is decreased, interval duration is increased, or connectivity damage is induced. For the considered tasks, we explored how robust the network obtained after training can be according to task parameterization. In the process, we developed a framework that can be useful to parameterize other tasks of interest in computational neuroscience. Our results are useful to quantify different aspects of the models, which are normally used as black boxes and need to be understood in order to model the biological response of cerebral cortex areas.

Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks

Different brain areas, such as the cortex and, more specifically, the prefrontal cortex, show great recurrence in their connections, even in early sensory areas. Several approaches and methods based on trained networks have been proposed to model and describe these regions. It is essential to understand the dynamics behind the models because they are used to build different hypotheses about the functioning of brain areas and to explain experimental results. The main contribution here is the description of the dynamics through the classification and interpretation carried out with a set of numerical simulations. This study sheds light on the multiplicity of solutions obtained for the same tasks and shows the link between the spectra of linearized trained networks and the dynamics of the counterparts. The patterns in the distribution of the eigenvalues of the recurrent weight matrix were studied and properly related to the dynamics in each task.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-022-09802-5.

Evaluating the statistical similarity of neural network activity and connectivity via eigenvector angles

Neural systems are networks, and strategic comparisons between multiple networks are a prevalent task in many research scenarios. In this study, we construct a statistical test for the comparison of matrices representing pairwise aspects of neural networks, in particular, the correlation between spiking activity and connectivity. The "eigenangle test" quantifies the similarity of two matrices by the angles between their ranked eigenvectors. We calibrate the behavior of the test for use with correlation matrices using stochastic models of correlated spiking activity and demonstrate how it compares to classical two-sample tests, such as the Kolmogorov-Smirnov distance, in the sense that it is able to evaluate also structural aspects of pairwise measures. Furthermore, the principle of the eigenangle test can be applied to compare the similarity of adjacency matrices of certain types of networks. Thus, the approach can be used to quantitatively explore the relationship between connectivity and activity with the same metric. By applying the eigenangle test to the comparison of connectivity matrices and correlation matrices of a random balanced network model before and after a specific synaptic rewiring intervention, we gauge the influence of connectivity features on the correlated activity. Potential applications of the eigenangle test include simulation experiments, model validation, and data analysis.

Time-convergent random matrices from mean-field pinned interacting eigenvalues

We study a multivariate system over a finite lifespan represented by a Hermitian-valued random matrix process whose eigenvalues (i) interact in a mean-field way and (ii) converge to their weighted ensemble average at their terminal time. We prove that such a system is guaranteed to converge in time to the identity matrix that is scaled by a Gaussian random variable whose variance is inversely proportional to the dimension of the matrix. As the size of the system grows asymptotically, the eigenvalues tend to mutually independent diffusions that converge to zero at their terminal time, a Brownian bridge being the archetypal example. Unlike commonly studied random matrices that have non-colliding eigenvalues, the proposed eigenvalues of the given system here may collide. We provide the dynamics of the eigenvalue gap matrix, which is a random skew-symmetric matrix that converges in time to the $\textbf{0}$ matrix. Our framework can be applied in producing mean-field interacting counterparts of stochastic quantum reduction models for which the convergence points are determined with respect to the average state of the entire composite system.

Dynamical systems on large networks with predator-prey interactions are stable and exhibit oscillations

We analyze the stability of linear dynamical systems defined on sparse, random graphs with predator-prey, competitive, and mutualistic interactions. These systems are aimed at modeling the stability of fixed points in large systems defined on complex networks, such as ecosystems consisting of a large number of species that interact through a food web. We develop an exact theory for the spectral distribution and the leading eigenvalue of the corresponding sparse Jacobian matrices. This theory reveals that the nature of local interactions has a strong influence on a system's stability. We show that, in general, linear dynamical systems defined on random graphs with a prescribed degree distribution of unbounded support are unstable if they are large enough, implying a tradeoff between stability and diversity. Remarkably, in contrast to the generic case, antagonistic systems that contain only interactions of the predator-prey type can be stable in the infinite size limit. This feature for antagonistic systems is accompanied by a peculiar oscillatory behavior of the dynamical response of the system after a perturbation, when the mean degree of the graph is small enough. Moreover, for antagonistic systems we also find that there exist a dynamical phase transition and critical mean degree above which the response becomes nonoscillatory.

Localization and Universality of Eigenvectors in Directed Random Graphs

Although the spectral properties of random graphs have been a long-standing focus of network theory, the properties of right eigenvectors of directed graphs have so far eluded an exact analytic treatment. We present a general theory for the statistics of the right eigenvector components in directed random graphs with a prescribed degree distribution and with randomly weighted links. We obtain exact analytic expressions for the inverse participation ratio and show that right eigenvectors of directed random graphs with a small average degree are localized. Remarkably, if the fourth moment of the degree distribution is finite, then the critical mean degree of the localization transition is independent of the degree fluctuations, which is different from localization in undirected graphs that is governed by degree fluctuations. We also show that in the high connectivity limit the distribution of the right eigenvector components is solely determined by the degree distribution. For delocalized eigenvectors, we recover in this limit the universal results from standard random matrix theory that are independent of the degree distribution, while for localized eigenvectors the eigenvector distribution depends on the degree distribution.

From Synaptic Interactions to Collective Dynamics in Random Neuronal Networks Models: Critical Role of Eigenvectors and Transient Behavior

The study of neuronal interactions is at the center of several big collaborative neuroscience projects (including the Human Connectome Project, the Blue Brain Project, and the Brainome) that attempt to obtain a detailed map of the entire brain. Under certain constraints, mathematical theory can advance predictions of the expected neural dynamics based solely on the statistical properties of the synaptic interaction matrix. This work explores the application of free random variables to the study of large synaptic interaction matrices. Besides recovering in a straightforward way known results on eigenspectra in types of models of neural networks proposed by Rajan and Abbott (2006), we extend them to heavy-tailed distributions of interactions. More important, we analytically derive the behavior of eigenvector overlaps, which determine the stability of the spectra. We observe that on imposing the neuronal excitation/inhibition balance, despite the eigenvalues remaining unchanged, their stability dramatically decreases due to the strong nonorthogonality of associated eigenvectors. This leads us to the conclusion that understanding the temporal evolution of asymmetric neural networks requires considering the entangled dynamics of both eigenvectors and eigenvalues, which might bear consequences for learning and memory processes in these models. Considering the success of free random variables theory in a wide variety of disciplines, we hope that the results presented here foster the additional application of these ideas in the area of brain sciences.

Training dynamically balanced excitatory-inhibitory networks

The construction of biologically plausible models of neural circuits is crucial for understanding the computational properties of the nervous system. Constructing functional networks composed of separate excitatory and inhibitory neurons obeying Dale’s law presents a number of challenges. We show how a target-based approach, when combined with a fast online constrained optimization technique, is capable of building functional models of rate and spiking recurrent neural networks in which excitation and inhibition are balanced. Balanced networks can be trained to produce complicated temporal patterns and to solve input-output tasks while retaining biologically desirable features such as Dale’s law and response variability.

Coherent chaos in a recurrent neural network with structured connectivity

We present a simple model for coherent, spatially correlated chaos in a recurrent neural network. Networks of randomly connected neurons exhibit chaotic fluctuations and have been studied as a model for capturing the temporal variability of cortical activity. The dynamics generated by such networks, however, are spatially uncorrelated and do not generate coherent fluctuations, which are commonly observed across spatial scales of the neocortex. In our model we introduce a structured component of connectivity, in addition to random connections, which effectively embeds a feedforward structure via unidirectional coupling between a pair of orthogonal modes. Local fluctuations driven by the random connectivity are summed by an output mode and drive coherent activity along an input mode. The orthogonality between input and output mode preserves chaotic fluctuations by preventing feedback loops. In the regime of weak structured connectivity we apply a perturbative approach to solve the dynamic mean-field equations, showing that in this regime coherent fluctuations are driven passively by the chaos of local residual fluctuations. When we introduce a row balance constraint on the random connectivity, stronger structured connectivity puts the network in a distinct dynamical regime of self-tuned coherent chaos. In this regime the coherent component of the dynamics self-adjusts intermittently to yield periods of slow, highly coherent chaos. The dynamics display longer time-scales and switching-like activity. We show how in this regime the dynamics depend qualitatively on the particular realization of the connectivity matrix: a complex leading eigenvalue can yield coherent oscillatory chaos while a real leading eigenvalue can yield chaos with broken symmetry. The level of coherence grows with increasing strength of structured connectivity until the dynamics are almost entirely constrained to a single spatial mode. We examine the effects of network-size scaling and show that these results are not finite-size effects. Finally, we show that in the regime of weak structured connectivity, coherent chaos emerges also for a generalized structured connectivity with multiple input-output modes.

Symmetries Constrain Dynamics in a Family of Balanced Neural Networks

We examine a family of random firing-rate neural networks in which we enforce the neurobiological constraint of Dale's Law-each neuron makes either excitatory or inhibitory connections onto its post-synaptic targets. We find that this constrained system may be described as a perturbation from a system with nontrivial symmetries. We analyze the symmetric system using the tools of equivariant bifurcation theory and demonstrate that the symmetry-implied structures remain evident in the perturbed system. In comparison, spectral characteristics of the network coupling matrix are relatively uninformative about the behavior of the constrained system.

Instability to a heterogeneous oscillatory state in randomly connected recurrent networks with delayed interactions

Oscillatory dynamics are ubiquitous in biological networks. Possible sources of oscillations are well understood in low-dimensional systems but have not been fully explored in high-dimensional networks. Here we study large networks consisting of randomly coupled rate units. We identify a type of bifurcation in which a continuous part of the eigenvalue spectrum of the linear stability matrix crosses the instability line at nonzero frequency. This bifurcation occurs when the interactions are delayed and partially antisymmetric and leads to a heterogeneous oscillatory state in which oscillations are apparent in the activity of individual units but not on the population-average level.

Limitations and tradeoffs in synchronization of large-scale networks with uncertain links

The synchronization of nonlinear systems connected over large-scale networks has gained popularity in a variety of applications, such as power grids, sensor networks, and biology. Stochastic uncertainty in the interconnections is a ubiquitous phenomenon observed in these physical and biological networks. We provide a size-independent network sufficient condition for the synchronization of scalar nonlinear systems with stochastic linear interactions over large-scale networks. This sufficient condition, expressed in terms of nonlinear dynamics, the Laplacian eigenvalues of the nominal interconnections, and the variance and location of the stochastic uncertainty, allows us to define a synchronization margin. We provide an analytical characterization of important trade-offs between the internal nonlinear dynamics, network topology, and uncertainty in synchronization. For nearest neighbour networks, the existence of an optimal number of neighbours with a maximum synchronization margin is demonstrated. An analytical formula for the optimal gain that produces the maximum synchronization margin allows us to compare the synchronization properties of various complex network topologies.

Asynchronous Rate Chaos in Spiking Neuronal Circuits

The brain exhibits temporally complex patterns of activity with features similar to those of chaotic systems. Theoretical studies over the last twenty years have described various computational advantages for such regimes in neuronal systems. Nevertheless, it still remains unclear whether chaos requires specific cellular properties or network architectures, or whether it is a generic property of neuronal circuits. We investigate the dynamics of networks of excitatory-inhibitory (EI) spiking neurons with random sparse connectivity operating in the regime of balance of excitation and inhibition. Combining Dynamical Mean-Field Theory with numerical simulations, we show that chaotic, asynchronous firing rate fluctuations emerge generically for sufficiently strong synapses. Two different mechanisms can lead to these chaotic fluctuations. One mechanism relies on slow I-I inhibition which gives rise to slow subthreshold voltage and rate fluctuations. The decorrelation time of these fluctuations is proportional to the time constant of the inhibition. The second mechanism relies on the recurrent E-I-E feedback loop. It requires slow excitation but the inhibition can be fast. In the corresponding dynamical regime all neurons exhibit rate fluctuations on the time scale of the excitation. Another feature of this regime is that the population-averaged firing rate is substantially smaller in the excitatory population than in the inhibitory population. This is not necessarily the case in the I-I mechanism. Finally, we discuss the neurophysiological and computational significance of our results.