Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Distilling Knowledge from Pre-trained Language Models via Text Smoothing   This paper studies compressing pre-trained language models, like BERT (Devlin
et al.,2019), via teacher-student knowledge distillation. Previous works
usually force the student model to strictly mimic the smoothed labels predicted
by the teacher BERT. As an alternative, we propose a new method for BERT
distillation, i.e., asking the teacher to generate smoothed word ids, rather
than labels, for teaching the student model in knowledge distillation. We call
this kind of methodTextSmoothing. Practically, we use the softmax prediction of
the Masked Language Model(MLM) in BERT to generate word distributions for given
texts and smooth those input texts using that predicted soft word ids. We
assume that both the smoothed labels and the smoothed texts can implicitly
augment the input corpus, while text smoothing is intuitively more efficient
since it can generate more instances in one neural network forward
step.Experimental results on GLUE and SQuAD demonstrate that our solution can
achieve competitive results compared with existing BERT distillation methods.

7
The categories of this paper are: hep-th
23
The categories of this paper are: solv-int
17
The categories of this paper are: q-fin
5
The categories of this paper are: gr-qc
30
The categories of this paper are: q-alg
3
The categories of this paper are: physics
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph