Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Supervision Exists Everywhere: A Data Efficient Contrastive
  Language-Image Pre-training Paradigm   Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has
attracted unprecedented attention for its impressive zero-shot recognition
ability and excellent transferability to downstream tasks. However, CLIP is
quite data-hungry and requires 400M image-text pairs for pre-training, thereby
restricting its adoption. This work proposes a novel training paradigm, Data
efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by
carefully utilizing the widespread supervision among the image-text pairs, our
De-CLIP can learn generic visual features more efficiently. Instead of using
the single image-text contrastive supervision, we fully exploit data potential
through the use of (1) self-supervision within each modality; (2) multi-view
supervision across modalities; (3) nearest-neighbor supervision from other
similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can
achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the
CLIP-ResNet50 while using 7.1 x fewer data. Our DeCLIP-ResNet50 outperforms its
counterpart in 8 out of 11 visual datasets when transferred to downstream
tasks. Moreover, Scaling up the model and computing also works well in our
framework.Our code, dataset and models are released at:
https://github.com/Sense-GVT/DeCLIP

5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
35
The categories of this paper are: mtrl-th
14
The categories of this paper are: math-ph
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
26
The categories of this paper are: bayes-an
8
The categories of this paper are: hep-ex
34
The categories of this paper are: chem-ph
30
The categories of this paper are: q-alg
36
The categories of this paper are: cmp-lg