  We present Variable Experience Rollout (VER), a technique for efficiently
scaling batched on-policy reinforcement learning in heterogenous environments
(where different environments take vastly different times to generate rollouts)
to many GPUs residing on, potentially, many machines. VER combines the
strengths of and blurs the line between synchronous and asynchronous on-policy
RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy
experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL).
  VER leads to significant and consistent speed-ups across a broad range of
embodied navigation and mobile manipulation tasks in photorealistic 3D
simulation environments. Specifically, for PointGoal navigation and ObjectGoal
navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO,
the current state of art distributed SyncOnRL, with similar sample efficiency.
For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in
Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x
speedup) on 8 GPUs than DD-PPO. Compared to SampleFactory (the current
state-of-the-art AsyncOnRL), VER matches its speed on 1 GPU, and is 70% faster
(1.7x speedup) on 8 GPUs with better sample efficiency.
  We leverage these speed-ups to train chained skills for GeometricGoal
rearrangement tasks in the Home Assistant Benchmark (HAB). We find a surprising
emergence of navigation in skills that do not ostensible require any
navigation. Specifically, the Pick skill involves a robot picking an object
from a table. During training the robot was always spawned close to the table
and never needed to navigate. However, we find that if base movement is part of
the action space, the robot learns to navigate then pick an object in new
environments with 50% success, demonstrating surprisingly high
out-of-distribution generalization.
