Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

Optimal Sampling of Dynamical Large Deviations in Two Dimensions via Tensor Networks

We use projected entangled-pair states (PEPS) to calculate the large deviation statistics of the dynamical activity of the two-dimensional East model, and the two-dimensional symmetric simple exclusion process (SSEP) with open boundaries, in lattices of up to 40×40 sites. We show that at long times both models have phase transitions between active and inactive dynamical phases. For the 2D East model we find that this trajectory transition is of the first order, while for the SSEP we find indications of a second order transition. We then show how the PEPS can be used to implement a trajectory sampling scheme capable of directly accessing rare trajectories. We also discuss how the methods described here can be extended to study rare events at finite times.

Adaptive power method for estimating large deviations in Markov chains

We study the performance of a stochastic algorithm based on the power method that adaptively learns the large deviation functions characterizing the fluctuations of additive functionals of Markov processes, used in physics to model nonequilibrium systems. This algorithm was introduced in the context of risk-sensitive control of Markov chains and was recently adapted to diffusions evolving continuously in time. Here we provide an in-depth study of the convergence of this algorithm close to dynamical phase transitions, exploring the speed of convergence as a function of the learning rate and the effect of including transfer learning. We use as a test example the mean degree of a random walk on an Erdős-Rényi random graph, which shows a transition between high-degree trajectories of the random walk evolving in the bulk of the graph and low-degree trajectories evolving in dangling edges of the graph. The results show that the adaptive power method is efficient close to dynamical phase transitions, while having many advantages in terms of performance and complexity compared to other algorithms used to compute large deviation functions.

Finite Time Large Deviations via Matrix Product States

Recent work has shown the effectiveness of tensor network methods for computing large deviation functions in constrained stochastic models in the infinite time limit. Here we show that these methods can also be used to study the statistics of dynamical observables at arbitrary finite time. This is a harder problem because, in contrast to the infinite time case, where only the extremal eigenstate of a tilted Markov generator is relevant, for finite time the whole spectrum plays a role. We show that finite time dynamical partition sums can be computed efficiently and accurately in one dimension using matrix product states and describe how to use such results to generate rare event trajectories on demand. We apply our methods to the Fredrickson-Andersen and East kinetically constrained models and to the symmetric simple exclusion process, unveiling dynamical phase diagrams in terms of counting field and trajectory time. We also discuss extensions of this method to higher dimensions.

Exact solution of the "Rule 150" reversible cellular automaton

We study the dynamics and statistics of the Rule 150 reversible cellular automaton (RCA). This is a one-dimensional lattice system of binary variables with synchronous (Floquet) dynamics that corresponds to a bulk deterministic and reversible discretized version of the kinetically constrained "exclusive one-spin facilitated" (XOR) Fredrickson-Andersen (FA) model, where the local dynamics is restricted: A site flips if and only if its adjacent sites are in different states from each other. Similar to other RCA that have been recently studied, such as Rule 54 and Rule 201, the Rule 150 RCA is integrable, however, in contrast is noninteracting: The emergent quasiparticles, which are identified by the domain walls, behave as free fermions. This property allows us to solve the model by means of matrix product ansatz. In particular, we find the exact equilibrium and nonequilibrium stationary states for systems with closed (periodic) and open (stochastic) boundaries, respectively, resolve the full spectrum of the time evolution operator and, therefore, gain access to the relaxation dynamics, and obtain the exact large deviation statistics of dynamical observables in the long-time limit.

Reinforcement learning of rare diffusive dynamics

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, such as those relevant in the study of reactive events, and trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, such as those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

Evolutionary reinforcement learning of dynamical large deviations

We show how to bound and calculate the likelihood of dynamical large deviations using evolutionary reinforcement learning. An agent, a stochastic model, propagates a continuous-time Monte Carlo trajectory and receives a reward conditioned upon the values of certain path-extensive quantities. Evolution produces progressively fitter agents, potentially allowing the calculation of a piece of a large-deviation rate function for a particular model and path-extensive quantity. For models with small state spaces, the evolutionary process acts directly on rates, and for models with large state spaces, the process acts on the weights of a neural network that parameterizes the model's rates. This approach shows how path-extensive physics problems can be considered within a framework widely used in machine learning.

Exact large deviation statistics and trajectory phase transition of a deterministic boundary driven cellular automaton

We study the statistical properties of the long-time dynamics of the rule 54 reversible cellular automaton (CA), driven stochastically at its boundaries. This CA can be considered as a discrete-time and deterministic version of the Fredrickson-Andersen kinetically constrained model (KCM). By means of a matrix product ansatz, we compute the exact large deviation cumulant generating functions for a wide range of time-extensive observables of the dynamics, together with their associated rate functions and conditioned long-time distributions over configurations. We show that for all instances of boundary driving the CA dynamics occurs at the point of phase coexistence between competing active and inactive dynamical phases, similar to what happens in more standard KCMs. We also find the exact finite size scaling behavior of these trajectory transitions, and provide the explicit "Doob-transformed" dynamics that optimally realizes rare dynamical events.