Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Layered Controllable Video Generation   We introduce layered controllable video generation, where we, without any
supervision, decompose the initial frame of a video into foreground and
background layers, with which the user can control the video generation process
by simply manipulating the foreground mask. The key challenges are the
unsupervised foreground-background separation, which is ambiguous, and ability
to anticipate user manipulations with access to only raw video sequences. We
address these challenges by proposing a two-stage learning procedure. In the
first stage, with the rich set of losses and dynamic foreground size prior, we
learn how to separate the frame into foreground and background layers and,
conditioned on these layers, how to generate the next frame using VQ-VAE
generator. In the second stage, we fine-tune this network to anticipate edits
to the mask, by fitting (parameterized) control to the mask from future frame.
We demonstrate the effectiveness of this learning and the more granular control
mechanism, while illustrating state-of-the-art performance on two benchmark
datasets. We provide a video abstract as well as some video results on
https://gabriel-huang.github.io/layered_controllable_video_generation

5
The categories of this paper are: gr-qc
23
The categories of this paper are: solv-int
29
The categories of this paper are: funct-an
21
The categories of this paper are: adap-org
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph
26
The categories of this paper are: bayes-an
4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
3
The categories of this paper are: physics
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
13
The categories of this paper are: quant-ph
31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess