  Progress in deep learning highlights the tremendous potential of utilizing
diverse robotic datasets for attaining effective generalization and makes it
enticing to consider leveraging broad datasets for attaining robust
generalization in robotic learning as well. However, in practice, we often want
to learn a new skill in a new environment that is unlikely to be contained in
the prior data. Therefore we ask: how can we leverage existing diverse offline
datasets in combination with small amounts of task-specific data to solve new
tasks, while still enjoying the generalization benefits of training on large
amounts of data? In this paper, we demonstrate that end-to-end offline RL can
be an effective approach for doing this, without the need for any
representation learning or vision-based pre-training. We present pre-training
for robots (PTR), a framework based on offline RL that attempts to effectively
learn new tasks by combining pre-training on existing robotic datasets with
rapid fine-tuning on a new task, with as few as 10 demonstrations. PTR utilizes
an existing offline RL method, conservative Q-learning (CQL), but extends it to
include several crucial design decisions that enable PTR to actually work and
outperform a variety of prior methods. To our knowledge, PTR is the first RL
method that succeeds at learning new tasks in a new domain on a real WidowX
robot with as few as 10 task demonstrations, by effectively leveraging an
existing dataset of diverse multi-task robot data collected in a variety of toy
kitchens. We also demonstrate that PTR can enable effective autonomous
fine-tuning and improvement in a handful of trials, without needing any
demonstrations. An accompanying overview video can be found in the
supplementary material and at thi URL: https://sites.google.com/view/ptr-final/
