Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: NWT: Towards natural audio-to-video generation with representation
  learning   In this work we introduce NWT, an expressive speech-to-video model. Unlike
approaches that use domain-specific intermediate representations such as pose
keypoints, NWT learns its own latent representations, with minimal assumptions
about the audio and video content. To this end, we propose a novel discrete
variational autoencoder with adversarial loss, dVAE-Adv, which learns a new
discrete latent representation we call Memcodes. Memcodes are straightforward
to implement, require no additional loss terms, are stable to train compared
with other approaches, and show evidence of interpretability. To predict on the
Memcode space, we use an autoregressive encoder-decoder model conditioned on
audio. Additionally, our model can control latent attributes in the generated
video that are not annotated in the data. We train NWT on clips from HBO's Last
Week Tonight with John Oliver. NWT consistently scores above other approaches
in Mean Opinion Score (MOS) on tests of overall video naturalness, facial
naturalness and expressiveness, and lipsync quality. This work sets a strong
baseline for generalized audio-to-video synthesis. Samples are available at
https://next-week-tonight.github.io/NWT/.

7
The categories of this paper are: hep-th
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
8
The categories of this paper are: hep-ex
9
The categories of this paper are: nlin
33
The categories of this paper are: plasm-ph
28
The categories of this paper are: alg-geom
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
3
The categories of this paper are: physics
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th