Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Knowledge Distillation from Internal Representations   Knowledge distillation is typically conducted by training a small model (the
student) to mimic a large and cumbersome model (the teacher). The idea is to
compress the knowledge from the teacher by using its output probabilities as
soft-labels to optimize the student. However, when the teacher is considerably
large, there is no guarantee that the internal knowledge of the teacher will be
transferred into the student; even if the student closely matches the
soft-labels, its internal representations may be considerably different. This
internal mismatch can undermine the generalization capabilities originally
intended to be transferred from the teacher to the student. In this paper, we
propose to distill the internal representations of a large model such as BERT
into a simplified version of it. We formulate two ways to distill such
representations and various algorithms to conduct the distillation. We
experiment with datasets from the GLUE benchmark and consistently show that
adding knowledge distillation from internal representations is a more powerful
method than only using soft-label distillation.

32
The categories of this paper are: atom-ph
10
The categories of this paper are: q-bio
5
The categories of this paper are: gr-qc
4
The categories of this paper are: cond-mat
23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
34
The categories of this paper are: chem-ph
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
19
The categories of this paper are: eess
30
The categories of this paper are: q-alg