Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Deep Cross-Modal Audio-Visual Generation   Cross-modal audio-visual perception has been a long-lasting topic in
psychology and neurology, and various studies have discovered strong
correlations in human perception of auditory and visual stimuli. Despite works
in computational multimodal modeling, the problem of cross-modal audio-visual
generation has not been systematically studied in the literature. In this
paper, we make the first attempt to solve this cross-modal generation problem
leveraging the power of deep generative adversarial training. Specifically, we
use conditional generative adversarial networks to achieve cross-modal
audio-visual generation of musical performances. We explore different encoding
methods for audio and visual signals, and work on two scenarios:
instrument-oriented generation and pose-oriented generation. Being the first to
explore this new problem, we compose two new datasets with pairs of images and
sounds of musical performances of different instruments. Our experiments using
both classification and human evaluations demonstrate that our model has the
ability to generate one modality, i.e., audio/visual, from the other modality,
i.e., visual/audio, to a good extent. Our experiments on various design choices
along with the datasets will facilitate future research in this new problem
space.

21
The categories of this paper are: adap-org
27
The categories of this paper are: comp-gas
4
The categories of this paper are: cond-mat
7
The categories of this paper are: hep-th
19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol
5
The categories of this paper are: gr-qc
12
The categories of this paper are: nucl-th
6
The categories of this paper are: astro-ph
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio