Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Patient Knowledge Distillation for BERT Model Compression   Pre-trained language models such as BERT have proven to be highly effective
for natural language processing (NLP) tasks. However, the high demand for
computing resources in training such models hinders their application in
practice. In order to alleviate this resource hunger in large-scale model
training, we propose a Patient Knowledge Distillation approach to compress an
original large model (teacher) into an equally-effective lightweight shallow
network (student). Different from previous knowledge distillation methods,
which only use the output from the last layer of the teacher network for
distillation, our student model patiently learns from multiple intermediate
layers of the teacher model for incremental knowledge extraction, following two
strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$)
PKD-Skip: learning from every $k$ layers. These two patient distillation
schemes enable the exploitation of rich information in the teacher's hidden
layers, and encourage the student model to patiently learn from and imitate the
teacher through a multi-layer distillation process. Empirically, this
translates into improved results on multiple NLP tasks with significant gain in
training efficiency, without sacrificing model accuracy.

8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
26
The categories of this paper are: bayes-an
25
The categories of this paper are: patt-sol
3
The categories of this paper are: physics
7
The categories of this paper are: hep-th
18
The categories of this paper are: econ
11
The categories of this paper are: hep-lat
14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg
37
The categories of this paper are: supr-con