Actor-Critic Learning Control Based on -Regularized Temporal-Difference Prediction With Gradient Correction

Payoff Control in Multichannel Games: Influencing Opponent Learning Evolution

In this article, we introduce a new theory for payoff control in multichannel learning environments, where agents interact with each other over multiple channels and each channel is a repeated normal form game. We propose two payoff control strategies-partial control and full control-that allow a single agent to set an upper bound to the opponent's expected payoffs summed across all channels, even if the opponent is a reinforcement learning agent. We prove that a partial (or full) control strategy can be obtained by solving a system of inequalities, and characterize the conditions under which such a partial (or full) control strategy exists. We show that by utilizing these control strategies, the agent can influence the opponent's learning evolution and direct it toward a desired viable equilibrium. Our experiments confirm the effectiveness of our theory for payoff control in a wide range of multichannel learning environments.

Actor-Critic Learning Control With Regularization and Feature Selection in Policy Gradient Estimation

Actor-critic (AC) learning control architecture has been regarded as an important framework for reinforcement learning (RL) with continuous states and actions. In order to improve learning efficiency and convergence property, previous works have been mainly devoted to solve regularization and feature learning problem in the policy evaluation. In this article, we propose a novel AC learning control method with regularization and feature selection for policy gradient estimation in the actor network. The main contribution is that l₁ -regularization is used on the actor network to achieve the function of feature selection. In each iteration, policy parameters are updated by the regularized dual-averaging (RDA) technique, which solves a minimization problem that involves two terms: one is the running average of the past policy gradients and the other is the l₁ -regularization term of policy parameters. Our algorithm can efficiently calculate the solution of the minimization problem, and we call the new adaptation of policy gradient RDA-policy gradient (RDA-PG). The proposed RDA-PG can learn stochastic and deterministic near-optimal policies. The convergence of the proposed algorithm is established based on the theory of two-timescale stochastic approximation. The simulation and experimental results show that RDA-PG performs feature selection successfully in the actor and learns sparse representations of the actor both in stochastic and deterministic cases. RDA-PG performs better than existing AC algorithms on standard RL benchmark problems with irrelevant features or redundant features.

A3C-GS: Adaptive Moment Gradient Sharing With Locks for Asynchronous Actor-Critic Agents

We propose an asynchronous gradient sharing mechanism for the parallel actor-critic algorithms with improved exploration characteristics. The proposed algorithm (A3C-GS) has the property of automatically diversifying worker policies in the short term for exploration, thereby reducing the need for entropy loss terms. Despite policy diversification, the algorithm converges to the optimal policy in the long term. We show in our analysis that the gradient sharing operation is a composition of two contractions. The first contraction performs gradient computation, while the second contraction is a gradient sharing operation coordinated by locks. From these two contractions, certain short- and long-term properties result. For the short term, gradient sharing induces temporary heterogeneity in policies for performing needed exploration. In the long term, under a suitably small learning rate and gradient clipping, convergence to the optimal policy is theoretically guaranteed. We verify our results with several high-dimensional experiments and compare A3C-GS against other on-policy policy-gradient algorithms. Our proposed algorithm achieved the highest weighted score. Despite lower entropy weights, it performed well in high-dimensional environments that require exploration due to sparse rewards and those that need navigation in 3-D environments for long survival tasks. It consistently performed better than the base asynchronous advantage actor-critic (A3C) algorithm.