Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Transformer Growth for Progressive BERT Training   Due to the excessive cost of large-scale language model pre-training,
considerable efforts have been made to train BERT progressively -- start from
an inferior but low-cost model and gradually grow the model to increase the
computational complexity. Our objective is to advance the understanding of
Transformer growth and discover principles that guide progressive training.
First, we find that similar to network architecture search, Transformer growth
also favors compound scaling. Specifically, while existing methods only conduct
network growth in a single dimension, we observe that it is beneficial to use
compound growth operators and balance multiple dimensions (e.g., depth, width,
and input length of the model). Moreover, we explore alternative growth
operators in each dimension via controlled comparison to give operator
selection practical guidance. In light of our analyses, the proposed method
speeds up BERT pre-training by 73.6% and 82.2% for the base and large models
respectively, while achieving comparable performances

18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol
20
The categories of this paper are: acc-phys
6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess
16
The categories of this paper are: stat
29
The categories of this paper are: funct-an
17
The categories of this paper are: q-fin
5
The categories of this paper are: gr-qc
15
The categories of this paper are: nucl-ex
22
The categories of this paper are: chao-dyn
27
The categories of this paper are: comp-gas
14
The categories of this paper are: math-ph
2
The categories of this paper are: math
23
The categories of this paper are: solv-int