Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: BP-Transformer: Modelling Long-Range Context via Binary Partitioning   The Transformer model is widely successful on many natural language
processing tasks. However, the quadratic complexity of self-attention limit its
application on long text. In this paper, adopting a fine-to-coarse attention
mechanism on multi-scale spans via binary partitioning (BP), we propose
BP-Transformer (BPT for short). BPT yields $O(k\cdot n\log (n/k))$ connections
where $k$ is a hyperparameter to control the density of attention. BPT has a
good balance between computation complexity and model capacity. A series of
experiments on text classification, machine translation and language modeling
shows BPT has a superior performance for long text than previous self-attention
models. Our code, hyperparameters and CUDA kernels for sparse attention are
available in PyTorch.

24
The categories of this paper are: dg-ga
10
The categories of this paper are: q-bio
22
The categories of this paper are: chao-dyn
37
The categories of this paper are: supr-con
26
The categories of this paper are: bayes-an
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
20
The categories of this paper are: acc-phys
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
15
The categories of this paper are: nucl-ex
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat
36
The categories of this paper are: cmp-lg
29
The categories of this paper are: funct-an