Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Time-Contrastive Networks: Self-Supervised Learning from Video   We propose a self-supervised approach for learning representations and
robotic behaviors entirely from unlabeled videos recorded from multiple
viewpoints, and study how this representation can be used in two robotic
imitation settings: imitating object interactions from videos of humans, and
imitating human poses. Imitation of human behavior requires a
viewpoint-invariant representation that captures the relationships between
end-effectors (hands or robot grippers) and the environment, object attributes,
and body pose. We train our representations using a metric learning loss, where
multiple simultaneous viewpoints of the same observation are attracted in the
embedding space, while being repelled from temporal neighbors which are often
visually similar but functionally different. In other words, the model
simultaneously learns to recognize what is common between different-looking
images, and what is different between similar-looking images. This signal
causes our model to discover attributes that do not change across viewpoint,
but do change across time, while ignoring nuisance variables such as
occlusions, motion blur, lighting and background. We demonstrate that this
representation can be used by a robot to directly mimic human poses without an
explicit correspondence, and that it can be used as a reward function within a
reinforcement learning algorithm. While representations are learned from an
unlabeled collection of task-related videos, robot behaviors such as pouring
are learned by watching a single 3rd-person demonstration by a human. Reward
functions obtained by following the human demonstrations under the learned
representation enable efficient reinforcement learning that is practical for
real-world robotic systems. Video results, open-source code and dataset are
available at https://sermanet.github.io/imitate

14
The categories of this paper are: math-ph
12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
7
The categories of this paper are: hep-th
23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
21
The categories of this paper are: adap-org
28
The categories of this paper are: alg-geom
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
16
The categories of this paper are: stat
11
The categories of this paper are: hep-lat
15
The categories of this paper are: nucl-ex
36
The categories of this paper are: cmp-lg