text: Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire NVIDIA DGX-1 to learn successful strategies in Atari games in single-digit minutes, using both synchronous and asynchronous algorithms.
  @cite_15: We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at this https URL
  @cite_6: 
  @cite_11: In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time, which is already a problem in single task learning. We have developed a new distributed agent IMPALA (Importance-Weighted Actor Learner Architecture) that can scale to thousands of machines and achieve a throughput rate of 250,000 frames per second. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace, which was critical for achieving learning stability. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (, 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (, 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents, use less data and crucially exhibits positive transfer between tasks as a result of its multi-task approach.
summary: The policy gradient method A3C is itself a parallelized algorithm. In GA3C @cite_6 , a speedup over CPU-only A3C was achieved by using a GPU. It was employed asynchronously, with predictor'' and trainer'' threads queuing observations and rewards for batched inferences and training updates. GA3C induced a policy lag'' between generation and consumption of training data, worsening sample complexity. In independent work simultaneous to ours, @cite_11 extended policy gradient methods to a distributed setting, enabling an alternative approach to multi-GPU training called IMPALA. They introduced a more heavily modified algorithm, , to mitigate policy lag--which we avoid--and did not employ GPU inference. In PAAC @cite_15 , the authors explored the use of many simulators and increased batch sizes learning rates in (single-GPU) batched A2C--ideas central to our studies. Our contributions to actor-critic methods exceed this work in a number of ways, chiefly: improved sampling organization, tremendously enhanced scale and speed using multiple GPUs, and inclusion of asynchronous optimization.