Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Sparsifying Encoder Outputs in Sequence-to-Sequence Models   Sequence-to-sequence models usually transfer all encoder outputs to the
decoder for generation. In this work, by contrast, we hypothesize that these
encoder outputs can be compressed to shorten the sequence delivered for
decoding. We take Transformer as the testbed and introduce a layer of
stochastic gates in-between the encoder and the decoder. The gates are
regularized using the expected value of the sparsity-inducing L0penalty,
resulting in completely masking-out a subset of encoder outputs. In other
words, via joint training, the L0DROP layer forces Transformer to route
information through a subset of its encoder states. We investigate the effects
of this sparsification on two machine translation and two summarization tasks.
Experiments show that, depending on the task, around 40-70% of source encodings
can be pruned without significantly compromising quality. The decrease of the
output length endows L0DROP with the potential of improving decoding
efficiency, where it yields a speedup of up to 1.65x on document summarization
tasks against the standard Transformer. We analyze the L0DROP behaviour and
observe that it exhibits systematic preferences for pruning certain word types,
e.g., function words and punctuation get pruned most. Inspired by these
observations, we explore the feasibility of specifying rule-based patterns that
mask out encoder outputs based on information such as part-of-speech tags, word
frequency and word position.

31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
13
The categories of this paper are: quant-ph
37
The categories of this paper are: supr-con
3
The categories of this paper are: physics
36
The categories of this paper are: cmp-lg
17
The categories of this paper are: q-fin
4
The categories of this paper are: cond-mat
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
7
The categories of this paper are: hep-th
34
The categories of this paper are: chem-ph
10
The categories of this paper are: q-bio
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat
23
The categories of this paper are: solv-int