Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Fastformer: Additive Attention Can Be All You Need   Transformer is a powerful model for text understanding. However, it is
inefficient due to its quadratic complexity to input sequence length. Although
there are many methods on Transformer acceleration, they are still either
inefficient on long sequences or not effective enough. In this paper, we
propose Fastformer, which is an efficient Transformer model based on additive
attention. In Fastformer, instead of modeling the pair-wise interactions
between tokens, we first use additive attention mechanism to model global
contexts, and then further transform each token representation based on its
interaction with global context representations. In this way, Fastformer can
achieve effective context modeling with linear complexity. Extensive
experiments on five datasets show that Fastformer is much more efficient than
many existing Transformer models and can meanwhile achieve comparable or even
better long text modeling performance.

19
The categories of this paper are: eess
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
8
The categories of this paper are: hep-ex
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
7
The categories of this paper are: hep-th
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
5
The categories of this paper are: gr-qc
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
29
The categories of this paper are: funct-an
17
The categories of this paper are: q-fin