A bio-inspired bistable recurrent cell allows for long-lasting memory

Warming up recurrent neural networks to maximise reachable multistability greatly improves learning

Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called "warmup" to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.

A Novel Traffic Prediction Method Using Machine Learning for Energy Efficiency in Service Provider Networks

This paper presents a systematic approach for solving complex prediction problems with a focus on energy efficiency. The approach involves using neural networks, specifically recurrent and sequential networks, as the main tool for prediction. In order to test the methodology, a case study was conducted in the telecommunications industry to address the problem of energy efficiency in data centers. The case study involved comparing four recurrent and sequential neural networks, including recurrent neural networks (RNNs), long short-term memory (LSTM), gated recurrent units (GRUs), and online sequential extreme learning machine (OS-ELM), to determine the best network in terms of prediction accuracy and computational time. The results show that OS-ELM outperformed the other networks in both accuracy and computational efficiency. The simulation was applied to real traffic data and showed potential energy savings of up to 12.2% in a single day. This highlights the importance of energy efficiency and the potential for the methodology to be applied to other industries. The methodology can be further developed as technology and data continue to advance, making it a promising solution for a wide range of prediction problems.

Multiplexing-based control of stochastic resonance

We show that multiplexing (Here, the term "multiplexing" means a special network topology where a one-layer network is connected to another one-layer networks through coupling between replica nodes. In the present paper, this term does not refer to the signal processing issues and telecommunications.) allows us to control noise-induced dynamics of multilayer networks in the regime of stochastic resonance. We illustrate this effect on an example of two- and multi-layer networks of bistable overdamped oscillators. In particular, we demonstrate that multiplexing suppresses the effect of stochastic resonance if the periodic forcing is present in only one layer. In contrast, multiplexing allows us to enhance the stochastic resonance if the periodic forcing and noise are present in all the interacting layers. In such a case, the impact of multiplexing has a resonant character: the most pronounced effect of stochastic resonance is achieved for an appropriate intermediate value of coupling strength between the layers. Moreover, multiplexing-induced enhancement of the stochastic resonance can become more pronounced for the increasing number of coupled layers. To visualize the revealed phenomena, we use the evolution of the dependence of the signal-to-noise ratio on the noise intensity for varying strength of coupling between the layers.

Two-color optically addressed spatial light modulator as a generic spatiotemporal system

Nonlinear spatiotemporal systems are the basis for countless physical phenomena in such diverse fields as ecology, optics, electronics, and neuroscience. The canonical approach to unify models originating from different fields is the normal form description, which determines the generic dynamical aspects and different bifurcation scenarios. Realizing different types of dynamical systems via one experimental platform that enables continuous transition between normal forms through tuning accessible system parameters is, therefore, highly relevant. Here, we show that a transmissive, optically addressed spatial light modulator under coherent optical illumination and optical feedback coupling allows tuning between pitchfork, transcritical, and saddle-node bifurcations of steady states. We demonstrate this by analytically deriving the system's dynamical equations in correspondence to the normal forms of the associated bifurcations and confirm these results via extensive numerical simulations. Our model describes a nematic liquid crystal device using nano-dimensional dichalcogenide (a-As ₂S ₃) glassy thin films as photo sensors and alignment layers, and we use device parameters obtained from experimental characterization. Optical coupling, for example, using diffraction, holography, or integrated unitary maps allows implementing a variety of system topologies of technological relevance for neural networks and potentially Ising or XY-Hamiltonian models with ultralow energy consumption.