Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Faster Depth-Adaptive Transformers   Depth-adaptive neural networks can dynamically adjust depths according to the
hardness of input words, and thus improve efficiency. The main challenge is how
to measure such hardness and decide the required depths (i.e., layers) to
conduct. Previous works generally build a halting unit to decide whether the
computation should continue or stop at each layer. As there is no specific
supervision of depth selection, the halting unit may be under-optimized and
inaccurate, which results in suboptimal and unstable performance when modeling
sentences. In this paper, we get rid of the halting unit and estimate the
required depths in advance, which yields a faster depth-adaptive model.
Specifically, two approaches are proposed to explicitly measure the hardness of
input words and estimate corresponding adaptive depth, namely 1) mutual
information (MI) based estimation and 2) reconstruction loss based estimation.
We conduct experiments on the text classification task with 24 datasets in
various sizes and domains. Results confirm that our approaches can speed up the
vanilla Transformer (up to 7x) while preserving high accuracy. Moreover,
efficiency and robustness are significantly improved when compared with other
depth-adaptive approaches.

36
The categories of this paper are: cmp-lg
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
31
The categories of this paper are: ao-sci
17
The categories of this paper are: q-fin
33
The categories of this paper are: plasm-ph
7
The categories of this paper are: hep-th
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
25
The categories of this paper are: patt-sol
34
The categories of this paper are: chem-ph
28
The categories of this paper are: alg-geom
3
The categories of this paper are: physics
4
The categories of this paper are: cond-mat
19
The categories of this paper are: eess
6
The categories of this paper are: astro-ph