Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Relative Position Prediction as Pre-training for Text Encoders   Meaning is defined by the company it keeps. However, company is two-fold:
It's based on the identity of tokens and also on their position (topology). We
argue that a position-centric perspective is more general and useful. The
classic MLM and CLM objectives in NLP are easily phrased as position
predictions over the whole vocabulary. Adapting the relative position encoding
paradigm in NLP to create relative labels for self-supervised learning, we seek
to show superior pre-training judged by performance on downstream tasks.

36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
29
The categories of this paper are: funct-an
9
The categories of this paper are: nlin
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
8
The categories of this paper are: hep-ex
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
27
The categories of this paper are: comp-gas
6
The categories of this paper are: astro-ph
16
The categories of this paper are: stat
22
The categories of this paper are: chao-dyn