Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: VATT: Transformers for Multimodal Self-Supervised Learning from Raw
  Video, Audio and Text   We present a framework for learning multimodal representations from unlabeled
data using convolution-free Transformer architectures. Specifically, our
Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts
multimodal representations that are rich enough to benefit a variety of
downstream tasks. We train VATT end-to-end from scratch using multimodal
contrastive losses and evaluate its performance by the downstream tasks of
video action recognition, audio event classification, image classification, and
text-to-video retrieval. Furthermore, we study a modality-agnostic,
single-backbone Transformer by sharing weights among the three modalities. We
show that the convolution-free VATT outperforms state-of-the-art ConvNet-based
architectures in the downstream tasks. Especially, VATT's vision Transformer
achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600,
72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding
supervised pre-training. Transferring to image classification leads to 78.7%
top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer
from scratch, showing the generalizability of our model despite the domain gap
between videos and images. VATT's audio Transformer also sets a new record on
waveform-based audio event recognition by achieving the mAP of 39.4% on
AudioSet without any supervised pre-training. VATT's source code is publicly
available.

37
The categories of this paper are: supr-con
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
28
The categories of this paper are: alg-geom
26
The categories of this paper are: bayes-an
33
The categories of this paper are: plasm-ph
29
The categories of this paper are: funct-an
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
34
The categories of this paper are: chem-ph
11
The categories of this paper are: hep-lat
13
The categories of this paper are: quant-ph
27
The categories of this paper are: comp-gas