  Imitation from observation (IfO) is a learning paradigm that consists of
training autonomous agents in a Markov Decision Process (MDP) by observing
expert demonstrations without access to its actions. These demonstrations could
be sequences of environment states or raw visual observations of the
environment. Recent work in IfO has focused on this problem in the case of
observations of low-dimensional environment states, however, access to these
highly-specific observations is unlikely in practice. In this paper, we adopt a
challenging, but more realistic problem formulation, learning control policies
that operate on a learned latent space with access only to visual
demonstrations of an expert completing a task. We present BootIfOL, an IfO
algorithm that aims to learn a reward function that takes an agent trajectory
and compares it to an expert, providing rewards based on similarity to agent
behavior and implicit goal. We consider this reward function to be a distance
metric between trajectories of agent behavior and learn it via contrastive
learning. The contrastive learning objective aims to closely represent expert
trajectories and to distance them from non-expert trajectories. The set of
non-expert trajectories used in contrastive learning is made progressively more
complex by bootstrapping from roll-outs of the agent learned through RL using
the current reward function. We evaluate our approach on a variety of control
tasks showing that we can train effective policies using a limited number of
demonstrative trajectories, greatly improving on prior approaches that consider
raw observations.
