Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Knowledge Distillation Using Hierarchical Self-Supervision Augmented
  Distribution   Knowledge distillation (KD) is an effective framework that aims to transfer
meaningful information from a large teacher to a smaller student. Generally, KD
often involves how to define and transfer knowledge. Previous KD methods often
focus on mining various forms of knowledge, for example, feature maps and
refined information. However, the knowledge is derived from the primary
supervised task and thus is highly task-specific. Motivated by the recent
success of self-supervised representation learning, we propose an auxiliary
self-supervision augmented task to guide networks to learn more meaningful
features. Therefore, we can derive soft self-supervision augmented
distributions as richer dark knowledge from this task for KD. Unlike previous
knowledge, this distribution encodes joint knowledge from supervised and
self-supervised feature learning. Beyond knowledge exploration, another crucial
aspect is how to learn and distill our proposed knowledge effectively. To fully
take advantage of hierarchical feature maps, we propose to append several
auxiliary branches at various hidden layers. Each auxiliary branch is guided to
learn self-supervision augmented task and distill this distribution from
teacher to student. Thus we call our KD method as Hierarchical Self-Supervision
Augmented Knowledge Distillation (HSSAKD). Experiments on standard image
classification show that both offline and online HSSAKD achieves
state-of-the-art performance in the field of KD. Further transfer experiments
on object detection further verify that HSSAKD can guide the network to learn
better features, which can be attributed to learn and distill an auxiliary
self-supervision augmented task effectively.

4
The categories of this paper are: cond-mat
2
The categories of this paper are: math
9
The categories of this paper are: nlin
18
The categories of this paper are: econ
12
The categories of this paper are: nucl-th
24
The categories of this paper are: dg-ga
32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
10
The categories of this paper are: q-bio
37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
0
The categories of this paper are: hep-ph
25
The categories of this paper are: patt-sol
33
The categories of this paper are: plasm-ph
34
The categories of this paper are: chem-ph
35
The categories of this paper are: mtrl-th