  Reward function design and exploration time are arguably the biggest
obstacles to the deployment of reinforcement learning (RL) agents in the real
world. In many real-world tasks, designing a reward function takes considerable
hand engineering and often requires additional sensors to be installed just to
measure whether the task has been executed successfully. Furthermore, many
interesting tasks consist of multiple implicit intermediate steps that must be
executed in sequence. Even when the final outcome can be measured, it does not
necessarily provide feedback on these intermediate steps. To address these
issues, we propose leveraging the abstraction power of intermediate visual
representations learned by deep models to quickly infer perceptual reward
functions from small numbers of demonstrations. We present a method that is
able to identify key intermediate steps of a task from only a handful of
demonstration sequences, and automatically identify the most discriminative
features for identifying these steps. This method makes use of the features in
a pre-trained deep model, but does not require any explicit specification of
sub-goals. The resulting reward functions can then be used by an RL agent to
learn to perform the task in real-world settings. To evaluate the learned
reward, we present qualitative results on two real-world tasks and a
quantitative evaluation against a human-designed reward function. We also show
that our method can be used to learn a real-world door opening skill using a
real robot, even when the demonstration used for reward learning is provided by
a human using their own hand. To our knowledge, these are the first results
showing that complex robotic manipulation skills can be learned directly and
without supervised labels from a video of a human performing the task.
Supplementary material and data are available at
https://sermanet.github.io/rewards
