Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Fast Transformers with Clustered Attention   Transformers have been proven a successful model for a variety of tasks in
sequence modeling. However, computing the attention matrix, which is their key
component, has quadratic complexity with respect to the sequence length, thus
making them prohibitively expensive for large sequences. To address this, we
propose clustered attention, which instead of computing the attention for every
query, groups queries into clusters and computes attention just for the
centroids. To further improve this approximation, we use the computed clusters
to identify the keys with the highest attention per query and compute the exact
key/query dot products. This results in a model with linear complexity with
respect to the sequence length for a fixed number of clusters. We evaluate our
approach on two automatic speech recognition datasets and show that our model
consistently outperforms vanilla transformers for a given computational budget.
Finally, we demonstrate that our model can approximate arbitrarily complex
attention distributions with a minimal number of clusters by approximating a
pretrained BERT model on GLUE and SQuAD benchmarks with only 25 clusters and no
loss in performance.

15
The categories of this paper are: nucl-ex
2
The categories of this paper are: math
34
The categories of this paper are: chem-ph
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
29
The categories of this paper are: funct-an
33
The categories of this paper are: plasm-ph
35
The categories of this paper are: mtrl-th
0
The categories of this paper are: hep-ph
12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
27
The categories of this paper are: comp-gas
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph