Robust Off-Policy Deep Reinforcement Learning

Abstract

Significant progress in Deep Learning, has helped Deep Reinforcement Learning (RL) algorithms achieve human like performance across applications ranging from games like Go and chess to simple robotic tasks. Off-policy Deep RL algorithms in particular have shown promising results on a wide range of simulated tasks. However, they are encumbered by stability concerns thus preventing their real-world deployment. This thesis makes several contributions toward developing off-policy deep RL algorithms that are robust, scalable, sample efficient and suitable for safety critical applications.

Our first contribution is Bagged Critic for Continuous Control (BC3). BC3 mitigates overestimation bias in off-policy actor-critic algorithms by employing an ensemble of state-value functions. Our second contribution is Spctral Normalized Actor Critic (SNAC). SNAC bounds the Lipschitz constants of the actor-critic networks in off-policy algorithms which in return bound the gradients flowing the network. Bounded gradients help RL agorithms learn more robust and sample efficient policies. Our last contribution is orthogonality constrained actor critic algorithms. Enforcing orthogonality on the weight matrices of actor critic networks helps preserve the norm of the gradients thus preventing vanishing gradients and avoiding convergence to suboptimal polices.