Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learned Token Pruning for Transformers   Deploying transformer models in practice is challenging due to their
inference cost, which scales quadratically with input sequence length. To
address this, we present a novel Learned Token Pruning (LTP) method which
adaptively removes unimportant tokens as an input sequence passes through
transformer layers. In particular, LTP prunes tokens with an attention score
below a threshold value which is learned for each layer during training. Our
threshold-based method allows the length of the pruned sequence to vary
adaptively based on the input sequence, and avoids algorithmically expensive
operations such as top-k token selection. We extensively test the performance
of LTP on GLUE tasks and show that our method outperforms the prior
state-of-the-art token pruning methods by up to ~2.5% higher accuracy with the
same amount of FLOPs. In particular, LTP achieves up to 2.1x FLOPs reduction
with less than 1% accuracy drop, which results in up to 1.9x and 2.0x
throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs,
respectively. Furthermore, we demonstrate that LTP is more robust than prior
methods to variations on input sentence lengths. Our code has been developed in
PyTorch and has been open-sourced.

7
The categories of this paper are: hep-th
5
The categories of this paper are: gr-qc
20
The categories of this paper are: acc-phys
24
The categories of this paper are: dg-ga
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
18
The categories of this paper are: econ
4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
23
The categories of this paper are: solv-int
16
The categories of this paper are: stat
33
The categories of this paper are: plasm-ph
14
The categories of this paper are: math-ph
9
The categories of this paper are: nlin
28
The categories of this paper are: alg-geom
29
The categories of this paper are: funct-an