Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language
  Representation Learning   We study joint learning of Convolutional Neural Network (CNN) and Transformer
for vision-language pre-training (VLPT) which aims to learn cross-modal
alignments from millions of image-text pairs. State-of-the-art approaches
extract salient image regions and align regions with words step-by-step. As
region-based visual features usually represent parts of an image, it is
challenging for existing vision-language models to fully understand the
semantics from paired natural languages. In this paper, we propose SOHO to "See
Out of tHe bOx" that takes a whole image as input, and learns vision-language
representation in an end-to-end manner. SOHO does not require bounding box
annotations which enables inference 10 times faster than region-based
approaches. In particular, SOHO learns to extract comprehensive yet compact
image features through a visual dictionary (VD) that facilitates cross-modal
understanding. VD is designed to represent consistent visual abstractions of
similar semantics. It is updated on-the-fly and utilized in our proposed
pre-training task Masked Visual Modeling (MVM). We conduct experiments on four
well-established vision-language tasks by following standard VLPT settings. In
particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text
retrieval 5k test split, 1.5% accuracy on NLVR$^2$ test-P split, 6.7% accuracy
on SNLI-VE test split, respectively.

4
The categories of this paper are: cond-mat
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
6
The categories of this paper are: astro-ph
30
The categories of this paper are: q-alg
24
The categories of this paper are: dg-ga
11
The categories of this paper are: hep-lat
16
The categories of this paper are: stat
2
The categories of this paper are: math
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph