Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Representation Consolidation for Training Expert Students   Traditionally, distillation has been used to train a student model to emulate
the input/output functionality of a teacher. A more useful goal than emulation,
yet under-explored, is for the student to learn feature representations that
transfer well to future tasks. However, we observe that standard distillation
of task-specific teachers actually *reduces* the transferability of student
representations to downstream tasks. We show that a multi-head, multi-task
distillation method using an unlabeled proxy dataset and a generalist teacher
is sufficient to consolidate representations from task-specific teacher(s) and
improve downstream performance, outperforming the teacher(s) and the strong
baseline of ImageNet pretrained features. Our method can also combine the
representational knowledge of multiple teachers trained on one or multiple
domains into a single model, whose representation is improved on all teachers'
domain(s).

13
The categories of this paper are: quant-ph
22
The categories of this paper are: chao-dyn
32
The categories of this paper are: atom-ph
31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
4
The categories of this paper are: cond-mat
35
The categories of this paper are: mtrl-th
26
The categories of this paper are: bayes-an
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
5
The categories of this paper are: gr-qc
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex