Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Corrupted Image Modeling for Self-Supervised Visual Pre-Training   We introduce Corrupted Image Modeling (CIM) for self-supervised visual
pre-training. CIM uses an auxiliary generator with a small trainable BEiT to
corrupt the input image instead of using artificial mask tokens, where some
patches are randomly selected and replaced with plausible alternatives sampled
from the BEiT output distribution. Given this corrupted image, an enhancer
network learns to either recover all the original image pixels, or predict
whether each visual token is replaced by a generator sample or not. The
generator and the enhancer are simultaneously trained and synergistically
updated. After pre-training, the enhancer can be used as a high-capacity visual
encoder for downstream tasks. CIM is a general and flexible visual pre-training
framework that is suitable for various network architectures. For the first
time, CIM demonstrates that both ViT and CNN can learn rich visual
representations using a unified, non-Siamese framework. Experimental results
show that our approach achieves compelling results in vision benchmarks, such
as ImageNet classification and ADE20K semantic segmentation. For example,
300-epoch CIM pre-trained vanilla ViT-Base/16 and ResNet-50 obtain 83.3 and
80.6 Top-1 fine-tuning accuracy on ImageNet-1K image classification
respectively.

10
The categories of this paper are: q-bio
0
The categories of this paper are: hep-ph
33
The categories of this paper are: plasm-ph
11
The categories of this paper are: hep-lat
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
16
The categories of this paper are: stat
26
The categories of this paper are: bayes-an
24
The categories of this paper are: dg-ga
32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
27
The categories of this paper are: comp-gas
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
29
The categories of this paper are: funct-an
5
The categories of this paper are: gr-qc