From Synaptic Interactions to Collective Dynamics in Random Neuronal Networks Models: Critical Role of Eigenvectors and Transient Behavior

Exploring Flip Flop memories and beyond: training Recurrent Neural Networks with key insights

Training neural networks to perform different tasks is relevant across various disciplines. In particular, Recurrent Neural Networks (RNNs) are of great interest in Computational Neuroscience. Open-source frameworks dedicated to Machine Learning, such as Tensorflow and Keras have produced significant changes in the development of technologies that we currently use. This work contributes by comprehensively investigating and describing the application of RNNs for temporal processing through a study of a 3-bit Flip Flop memory implementation. We delve into the entire modeling process, encompassing equations, task parametrization, and software development. The obtained networks are meticulously analyzed to elucidate dynamics, aided by an array of visualization and analysis tools. Moreover, the provided code is versatile enough to facilitate the modeling of diverse tasks and systems. Furthermore, we present how memory states can be efficiently stored in the vertices of a cube in the dimensionally reduced space, supplementing previous results with a distinct approach.

Unconventional criticality, scaling breakdown, and diverse universality classes in the Wilson-Cowan model of neural dynamics

The Wilson-Cowan model constitutes a paradigmatic approach to understanding the collective dynamics of networks of excitatory and inhibitory units. It has been profusely used in the literature to analyze the possible phases of neural networks at a mean-field level, e.g., assuming large fully connected networks. Moreover, its stochastic counterpart allows one to study fluctuation-induced phenomena, such as avalanches. Here we revisit the stochastic Wilson-Cowan model paying special attention to the possible phase transitions between quiescent and active phases. We unveil eight possible types of such transitions, including continuous ones with scaling behavior belonging to known universality classes-such as directed percolation and tricritical directed percolation-as well as six distinct ones. In particular, we show that under some special circumstances, at a so-called "Hopf tricritical directed percolation" transition, rather unconventional behavior is observed, including the emergence of scaling breakdown. Other transitions are discontinuous and show different types of anomalies in scaling and/or exhibit mixed features of continuous and discontinuous transitions. These results broaden our knowledge of the possible types of critical behavior in networks of excitatory and inhibitory units and are, thus, of relevance to understanding avalanche dynamics in actual neuronal recordings. From a more general perspective, these results help extend the theory of nonequilibrium phase transitions into quiescent or absorbing states.

Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks

Different brain areas, such as the cortex and, more specifically, the prefrontal cortex, show great recurrence in their connections, even in early sensory areas. Several approaches and methods based on trained networks have been proposed to model and describe these regions. It is essential to understand the dynamics behind the models because they are used to build different hypotheses about the functioning of brain areas and to explain experimental results. The main contribution here is the description of the dynamics through the classification and interpretation carried out with a set of numerical simulations. This study sheds light on the multiplicity of solutions obtained for the same tasks and shows the link between the spectra of linearized trained networks and the dynamics of the counterparts. The patterns in the distribution of the eigenvalues of the recurrent weight matrix were studied and properly related to the dynamics in each task.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-022-09802-5.

Network structural origin of instabilities in large complex systems

A central issue in the study of large complex network systems, such as power grids, financial networks, and ecological systems, is to understand their response to dynamical perturbations. Recent studies recognize that many real networks show nonnormality and that nonnormality can give rise to reactivity-the capacity of a linearly stable system to amplify its response to perturbations, oftentimes exciting nonlinear instabilities. Here, we identify network structural properties underlying the pervasiveness of nonnormality and reactivity in real directed networks, which we establish using the most extensive dataset of such networks studied in this context to date. The identified properties are imbalances between incoming and outgoing network links and paths at each node. On the basis of this characterization, we develop a theory that quantitatively predicts nonnormality and reactivity and explains the observed pervasiveness. We suggest that these results can be used to design, upgrade, control, and manage networks to avoid or promote network instabilities.

Real spectra of large real asymmetric random matrices

When a randomness is introduced at the level of real matrix elements, depending on its particular realization, a pair of eigenvalues can appear as real or form a complex conjugate pair. We show that in the limit of large matrix size the density of such real eigenvalues is proportional to the square root of the asymptotic density of complex eigenvalues continuated to the real line. This relation allows one to calculate the real densities up to a normalization constant, which is then applied to various examples, including heavy-tailed ensembles and adjacency matrices of sparse random regular graphs.

Edge of Chaos and Avalanches in Neural Networks with Heavy-Tailed Synaptic Weight Distribution

We propose an analytically tractable neural connectivity model with power-law distributed synaptic strengths. When threshold neurons with biologically plausible number of incoming connections are considered, our model features a continuous transition to chaos and can reproduce biologically relevant low activity levels and scale-free avalanches, i.e., bursts of activity with power-law distributions of sizes and lifetimes. In contrast, the Gaussian counterpart exhibits a discontinuous transition to chaos and thus cannot be poised near the edge of chaos. We validate our predictions in simulations of networks of binary as well as leaky integrate-and-fire neurons. Our results suggest that heavy-tailed synaptic distribution may form a weakly informative sparse-connectivity prior that can be useful in biological and artificial adaptive systems.