Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: DA-Transformer: Distance-aware Transformer   Transformer has achieved great success in the NLP field by composing various
advanced models like BERT and GPT. However, Transformer and its existing
variants may not be optimal in capturing token distances because the position
or distance embeddings used by these methods usually cannot keep the precise
information of real distances, which may not be beneficial for modeling the
orders and relations of contexts. In this paper, we propose DA-Transformer,
which is a distance-aware Transformer that can exploit the real distance. We
propose to incorporate the real distances between tokens to re-scale the raw
self-attention weights, which are computed by the relevance between attention
query and key. Concretely, in different self-attention heads the relative
distance between each pair of tokens is weighted by different learnable
parameters, which control the different preferences on long- or short-term
information of these heads. Since the raw weighted real distances may not be
optimal for adjusting self-attention weights, we propose a learnable sigmoid
function to map them into re-scaled coefficients that have proper ranges. We
first clip the raw self-attention weights via the ReLU function to keep
non-negativity and introduce sparsity, and then multiply them with the
re-scaled coefficients to encode real distance information into self-attention.
Extensive experiments on five benchmark datasets show that DA-Transformer can
effectively improve the performance of many tasks and outperform the vanilla
Transformer and its several variants.

25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom
36
The categories of this paper are: cmp-lg
7
The categories of this paper are: hep-th
14
The categories of this paper are: math-ph
17
The categories of this paper are: q-fin
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat
10
The categories of this paper are: q-bio
21
The categories of this paper are: adap-org
20
The categories of this paper are: acc-phys
4
The categories of this paper are: cond-mat
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
19
The categories of this paper are: eess