Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Rethinking and Improving Relative Position Encoding for Vision
  Transformer   Relative position encoding (RPE) is important for transformer to capture
sequence ordering of input tokens. General efficacy has been proven in natural
language processing. However, in computer vision, its efficacy is not well
studied and even remains controversial, e.g., whether relative position
encoding can work equally well as absolute position? In order to clarify this,
we first review existing relative position encoding methods and analyze their
pros and cons when applied in vision transformers. We then propose new relative
position encoding methods dedicated to 2D images, called image RPE (iRPE). Our
methods consider directional relative distance modeling as well as the
interactions between queries and relative position embeddings in self-attention
mechanism. The proposed iRPE methods are simple and lightweight. They can be
easily plugged into transformer blocks. Experiments demonstrate that solely due
to the proposed encoding methods, DeiT and DETR obtain up to 1.5% (top-1 Acc)
and 1.3% (mAP) stable improvements over their original versions on ImageNet and
COCO respectively, without tuning any extra hyperparameters such as learning
rate and weight decay. Our ablation and analysis also yield interesting
findings, some of which run counter to previous understanding. Code and models
are open-sourced at https://github.com/microsoft/Cream/tree/main/iRPE.

7
The categories of this paper are: hep-th
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
10
The categories of this paper are: q-bio
3
The categories of this paper are: physics
23
The categories of this paper are: solv-int
31
The categories of this paper are: ao-sci
18
The categories of this paper are: econ
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
37
The categories of this paper are: supr-con
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat
19
The categories of this paper are: eess