Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for
  Sequences   We describe an efficient hierarchical method to compute attention in the
Transformer architecture. The proposed attention mechanism exploits a matrix
structure similar to the Hierarchical Matrix (H-Matrix) developed by the
numerical analysis community, and has linear run time and memory complexity. We
perform extensive experiments to show that the inductive bias embodied by our
hierarchical attention is effective in capturing the hierarchical structure in
the sequences typical for natural language and vision tasks. Our method is
superior to alternative sub-quadratic proposals by over +6 points on average on
the Long Range Arena benchmark. It also sets a new SOTA test perplexity on
One-Billion Word dataset with 5x fewer model parameters than that of the
previous-best Transformer-based models.

20
The categories of this paper are: acc-phys
19
The categories of this paper are: eess
33
The categories of this paper are: plasm-ph
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
14
The categories of this paper are: math-ph
15
The categories of this paper are: nucl-ex
6
The categories of this paper are: astro-ph
8
The categories of this paper are: hep-ex
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin
16
The categories of this paper are: stat
32
The categories of this paper are: atom-ph