  Deep Reinforcement Learning (RL) has been demonstrated to yield capable
agents and control policies in several domains but is commonly plagued by
prohibitively long training times. Additionally, in the case of continuous
control problems, the applicability of learned policies on real-world embedded
devices is limited due to the lack of real-time guarantees and portability of
existing deep learning libraries. To address these challenges, we present
RLtools, a dependency-free, header-only, pure C++ library for deep supervised
and reinforcement learning. Leveraging the template meta-programming
capabilities of recent C++ standards, we provide composable components that can
be tightly integrated by the compiler. Its novel architecture allows RLtools to
be used seamlessly on a heterogeneous set of platforms, from HPC clusters over
workstations and laptops to smartphones, smartwatches, and microcontrollers.
Specifically, due to the tight integration of the RL algorithms with simulation
environments, RLtools can solve popular RL problems like the Pendulum-v1
swing-up about 7 to 15 times faster in terms of wall-clock training time
compared to other popular RL frameworks when using TD3. We also provide a
low-overhead and parallelized interface to the MuJoCo simulator, showing that
our PPO implementation achieves state of the art returns in the Ant-v4
environment while being 25%-30% faster in terms of wall-clock training time.
Finally, we also benchmark the policy inference on a diverse set of
microcontrollers and show that in most cases our optimized inference
implementation is much faster than even the manufacturer's DSP libraries. To
the best of our knowledge, RLtools enables the first-ever demonstration of
training a deep RL algorithm directly on a microcontroller, giving rise to the
field of TinyRL. The source code is available through our project page at
https://rl.tools.
