Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Pro-KD: Progressive Distillation by Following the Footsteps of the
  Teacher   With ever growing scale of neural models, knowledge distillation (KD)
attracts more attention as a prominent tool for neural model compression.
However, there are counter intuitive observations in the literature showing
some challenging limitations of KD. A case in point is that the best performing
checkpoint of the teacher might not necessarily be the best teacher for
training the student in KD. Therefore, one important question would be how to
find the best checkpoint of the teacher for distillation? Searching through the
checkpoints of the teacher would be a very tedious and computationally
expensive process, which we refer to as the \textit{checkpoint-search problem}.
Moreover, another observation is that larger teachers might not necessarily be
better teachers in KD which is referred to as the \textit{capacity-gap}
problem. To address these challenging problems, in this work, we introduce our
progressive knowledge distillation (Pro-KD) technique which defines a smoother
training path for the student by following the training footprints of the
teacher instead of solely relying on distilling from a single mature
fully-trained teacher. We demonstrate that our technique is quite effective in
mitigating the capacity-gap problem and the checkpoint search problem. We
evaluate our technique using a comprehensive set of experiments on different
tasks such as image classification (CIFAR-10 and CIFAR-100), natural language
understanding tasks of the GLUE benchmark, and question answering (SQuAD 1.1
and 2.0) using BERT-based models and consistently got superior results over
state-of-the-art techniques.

36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
26
The categories of this paper are: bayes-an
14
The categories of this paper are: math-ph
10
The categories of this paper are: q-bio
27
The categories of this paper are: comp-gas
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
28
The categories of this paper are: alg-geom
9
The categories of this paper are: nlin
18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
33
The categories of this paper are: plasm-ph