  The ever-growing complexity of reinforcement learning (RL) tasks demands a
distributed RL system to efficiently generate and process a massive amount of
data to train intelligent agents. However, existing open-source libraries
suffer from various limitations, which impede their practical use in
challenging scenarios where large-scale training is necessary. While industrial
systems from OpenAI and DeepMind have achieved successful large-scale RL
training, their system architecture and implementation details remain
undisclosed to the community. In this paper, we present a novel abstraction on
the dataflows of RL training, which unifies practical RL training across
diverse applications into a general framework and enables fine-grained
optimizations. Following this abstraction, we develop a scalable, efficient,
and extensible distributed RL system called ReaLly Scalable RL (SRL). The
system architecture of SRL separates major RL computation components and allows
massively parallelized training. Moreover, SRL offers user-friendly and
extensible interfaces for customized algorithms. Our evaluation shows that SRL
outperforms existing academic libraries in both a single machine and a
medium-sized cluster. In a large-scale cluster, the novel architecture of SRL
leads to up to 3.7x speedup compared to the design choices adopted by the
existing libraries. We also conduct a direct benchmark comparison to OpenAI's
industrial system, Rapid, in the challenging hide-and-seek environment. SRL
reproduces the same solution as reported by OpenAI with up to 5x speedup in
wall-clock time. Furthermore, we also examine the performance of SRL in a much
harder variant of the hide-and-seek environment and achieve substantial
learning speedup by scaling SRL to over 15k CPU cores and 32 A100 GPUs.
Notably, SRL is the first in the academic community to perform RL experiments
at such a large scale.
