Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers   This paper explores a better codebook for BERT pre-training of vision
transformers. The recent work BEiT successfully transfers BERT pre-training
from NLP to the vision field. It directly adopts one simple discrete VAE as the
visual tokenizer, but has not considered the semantic level of the resulting
visual tokens. By contrast, the discrete tokens in NLP field are naturally
highly semantic. This difference motivates us to learn a perceptual codebook.
And we surprisingly find one simple yet effective idea: enforcing perceptual
similarity during the dVAE training. We demonstrate that the visual tokens
generated by the proposed perceptual codebook do exhibit better semantic
meanings, and subsequently help pre-training achieve superior transfer
performance in various downstream tasks. For example, we achieve 84.5% Top-1
accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive
method BEiT by +1.3 with the same pre-training epochs. It can also improve the
performance of object detection and segmentation tasks on COCO val by +1.3 box
AP and +1.0 mask AP, semantic segmentation on ADE20k by +1.0 mIoU. Equipped
with a larger backbone ViT-H, we achieve the state-of-the-art performance
(88.3% Top-1 accuracy) among the methods using only ImageNet-1K data. The code
and models will be available at https://github.com/microsoft/PeCo.

36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
28
The categories of this paper are: alg-geom
0
The categories of this paper are: hep-ph
8
The categories of this paper are: hep-ex
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
2
The categories of this paper are: math
3
The categories of this paper are: physics
35
The categories of this paper are: mtrl-th
15
The categories of this paper are: nucl-ex