Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Relational Knowledge Distillation   Knowledge distillation aims at transferring knowledge acquired in one model
(a teacher) to another model (a student) that is typically smaller. Previous
approaches can be expressed as a form of training the student to mimic output
activations of individual data examples represented by the teacher. We
introduce a novel approach, dubbed relational knowledge distillation (RKD),
that transfers mutual relations of data examples instead. For concrete
realizations of RKD, we propose distance-wise and angle-wise distillation
losses that penalize structural differences in relations. Experiments conducted
on different tasks show that the proposed method improves educated student
models with a significant margin. In particular for metric learning, it allows
students to outperform their teachers' performance, achieving the state of the
arts on standard benchmark datasets.

13
The categories of this paper are: quant-ph
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex
12
The categories of this paper are: nucl-th
11
The categories of this paper are: hep-lat
10
The categories of this paper are: q-bio
4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
24
The categories of this paper are: dg-ga
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
29
The categories of this paper are: funct-an
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
19
The categories of this paper are: eess