  Robots will experience non-stationary environment dynamics throughout their
lifetime: the robot dynamics can change due to wear and tear, or its
surroundings may change over time. Eventually, the robots should perform well
in all of the environment variations it has encountered. At the same time, it
should still be able to learn fast in a new environment. We identify two
challenges in Reinforcement Learning (RL) under such a lifelong learning
setting with off-policy data: first, existing off-policy algorithms struggle
with the trade-off between being conservative to maintain good performance in
the old environment and learning efficiently in the new environment, despite
keeping all the data in the replay buffer. We propose the Offline Distillation
Pipeline to break this trade-off by separating the training procedure into an
online interaction phase and an offline distillation phase.Second, we find that
training with the imbalanced off-policy data from multiple environments across
the lifetime creates a significant performance drop. We identify that this
performance drop is caused by the combination of the imbalanced quality and
size among the datasets which exacerbate the extrapolation error of the
Q-function. During the distillation phase, we apply a simple fix to the issue
by keeping the policy closer to the behavior policy that generated the data. In
the experiments, we demonstrate these two challenges and the proposed solutions
with a simulated bipedal robot walk-ing task across various environment
changes. We show that the Offline Distillation Pipeline achieves better
performance across all the encountered environments without affecting data
collection. We also provide a comprehensive empirical study to support our
hypothesis on the data imbalance issue.
