Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Knowledge Distillation Thrives on Data Augmentation   Knowledge distillation (KD) is a general deep neural network training
framework that uses a teacher model to guide a student model. Many works have
explored the rationale for its success, however, its interplay with data
augmentation (DA) has not been well recognized so far. In this paper, we are
motivated by an interesting observation in classification: KD loss can benefit
from extended training iterations while the cross-entropy loss does not. We
show this disparity arises because of data augmentation: KD loss can tap into
the extra information from different input views brought by DA. By this
explanation, we propose to enhance KD via a stronger data augmentation scheme
(e.g., mixup, CutMix). Furthermore, an even stronger new DA approach is
developed specifically for KD based on the idea of active learning. The
findings and merits of the proposed method are validated by extensive
experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We can achieve
improved performance simply by using the original KD loss combined with
stronger augmentation schemes, compared to existing state-of-the-art methods,
which employ more advanced distillation losses. In addition, when our
approaches are combined with more advanced distillation losses, we can advance
the state-of-the-art performance even more. On top of the encouraging
performance, this paper also sheds some light on explaining the success of
knowledge distillation. The discovered interplay between KD and DA may inspire
more advanced KD algorithms.

19
The categories of this paper are: eess
21
The categories of this paper are: adap-org
17
The categories of this paper are: q-fin
16
The categories of this paper are: stat
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
14
The categories of this paper are: math-ph
10
The categories of this paper are: q-bio
34
The categories of this paper are: chem-ph
13
The categories of this paper are: quant-ph
26
The categories of this paper are: bayes-an
33
The categories of this paper are: plasm-ph
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol