Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Cluster-Former: Clustering-based Sparse Transformer for Long-Range
  Dependency Encoding   Transformer has become ubiquitous in the deep learning field. One of the key
ingredients that destined its success is the self-attention mechanism, which
allows fully-connected contextual encoding over input tokens. However, despite
its effectiveness in modeling short sequences, self-attention suffers when
handling inputs with extreme long-range dependencies, as its complexity grows
quadratically with respect to the sequence length. Therefore, long sequences
are often encoded by Transformer in chunks using a sliding window. In this
paper, we propose Cluster-Former, a novel clustering-based sparse Transformer
to perform attention across chunked sequences. The proposed framework is
pivoted on two unique types of Transformer layer: Sliding-Window Layer and
Cluster-Former Layer, which encode local sequence information and global
context jointly and iteratively. This new design allows information integration
beyond local windows, which is especially beneficial for question answering
(QA) tasks that rely on long-range dependencies. Experiments show that
Cluster-Former achieves state-of-the-art performance on several major QA
benchmarks.

21
The categories of this paper are: adap-org
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ
20
The categories of this paper are: acc-phys
6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
22
The categories of this paper are: chao-dyn