Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Sparsity and Sentence Structure in Encoder-Decoder Attention of
  Summarization Systems   Transformer models have achieved state-of-the-art results in a wide range of
NLP tasks including summarization. Training and inference using large
transformer models can be computationally expensive. Previous work has focused
on one important bottleneck, the quadratic self-attention mechanism in the
encoder. Modified encoder architectures such as LED or LoBART use local
attention patterns to address this problem for summarization. In contrast, this
work focuses on the transformer's encoder-decoder attention mechanism. The cost
of this attention becomes more significant in inference or training approaches
that require model-generated histories. First, we examine the complexity of the
encoder-decoder attention. We demonstrate empirically that there is a sparse
sentence structure in document summarization that can be exploited by
constraining the attention mechanism to a subset of input sentences, whilst
maintaining system performance. Second, we propose a modified architecture that
selects the subset of sentences to constrain the encoder-decoder attention.
Experiments are carried out on abstractive summarization tasks, including
CNN/DailyMail, XSum, Spotify Podcast, and arXiv.

33
The categories of this paper are: plasm-ph
21
The categories of this paper are: adap-org
14
The categories of this paper are: math-ph
18
The categories of this paper are: econ
24
The categories of this paper are: dg-ga
6
The categories of this paper are: astro-ph
15
The categories of this paper are: nucl-ex
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
28
The categories of this paper are: alg-geom
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
4
The categories of this paper are: cond-mat
2
The categories of this paper are: math
16
The categories of this paper are: stat
30
The categories of this paper are: q-alg