Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Taming Visually Guided Sound Generation   Recent advances in visually-induced audio generation are based on sampling
short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio
from the state-of-the-art model takes minutes on a high-end GPU. In this work,
we propose a single model capable of generating visually relevant,
high-fidelity sounds prompted with a set of frames from open-domain videos in
less time than it takes to play it on a single GPU.
  We train a transformer to sample a new spectrogram from the pre-trained
spectrogram codebook given the set of video features. The codebook is obtained
using a variant of VQGAN trained to produce a compact sampling space with a
novel spectrogram-based perceptual loss. The generated spectrogram is
transformed into a waveform using a window-based GAN that significantly speeds
up generation. Considering the lack of metrics for automatic evaluation of
generated spectrograms, we also build a family of metrics called FID and MKL.
These metrics are based on a novel sound classifier, called Melception, and
designed to evaluate the fidelity and relevance of open-domain samples.
  Both qualitative and quantitative studies are conducted on small- and
large-scale datasets to evaluate the fidelity and relevance of generated
samples. We also compare our model to the state-of-the-art and observe a
substantial improvement in quality, size, and computation time. Code, demo, and
samples: v-iashin.github.io/SpecVQGAN

16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph
29
The categories of this paper are: funct-an
25
The categories of this paper are: patt-sol
6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci
34
The categories of this paper are: chem-ph
24
The categories of this paper are: dg-ga
35
The categories of this paper are: mtrl-th
20
The categories of this paper are: acc-phys
22
The categories of this paper are: chao-dyn
14
The categories of this paper are: math-ph
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin