Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: BERT Learns to Teach: Knowledge Distillation with Meta Learning   We present Knowledge Distillation with Meta Learning (MetaDistil), a simple
yet effective alternative to traditional knowledge distillation (KD) methods
where the teacher model is fixed during training. We show the teacher network
can learn to better transfer knowledge to the student network (i.e., learning
to teach) with the feedback from the performance of the distilled student
network in a meta learning framework. Moreover, we introduce a pilot update
mechanism to improve the alignment between the inner-learner and meta-learner
in meta learning algorithms that focus on an improved inner-learner.
Experiments on various benchmarks show that MetaDistil can yield significant
improvements compared with traditional KD algorithms and is less sensitive to
the choice of different student capacity and hyperparameters, facilitating the
use of KD on different tasks and models.

28
The categories of this paper are: alg-geom
7
The categories of this paper are: hep-th
33
The categories of this paper are: plasm-ph
18
The categories of this paper are: econ
11
The categories of this paper are: hep-lat
5
The categories of this paper are: gr-qc
21
The categories of this paper are: adap-org
8
The categories of this paper are: hep-ex
12
The categories of this paper are: nucl-th
19
The categories of this paper are: eess
6
The categories of this paper are: astro-ph
23
The categories of this paper are: solv-int
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
20
The categories of this paper are: acc-phys
22
The categories of this paper are: chao-dyn