Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Student-Teacher Learning from Clean Inputs to Noisy Inputs   Feature-based student-teacher learning, a training method that encourages the
student's hidden features to mimic those of the teacher network, is empirically
successful in transferring the knowledge from a pre-trained teacher network to
the student network. Furthermore, recent empirical results demonstrate that,
the teacher's features can boost the student network's generalization even when
the student's input sample is corrupted by noise. However, there is a lack of
theoretical insights into why and when this method of transferring knowledge
can be successful between such heterogeneous tasks. We analyze this method
theoretically using deep linear networks, and experimentally using nonlinear
networks. We identify three vital factors to the success of the method: (1)
whether the student is trained to zero training loss; (2) how knowledgeable the
teacher is on the clean-input problem; (3) how the teacher decomposes its
knowledge in its hidden features. Lack of proper control in any of the three
factors leads to failure of the student-teacher learning method.

12
The categories of this paper are: nucl-th
21
The categories of this paper are: adap-org
29
The categories of this paper are: funct-an
28
The categories of this paper are: alg-geom
31
The categories of this paper are: ao-sci
22
The categories of this paper are: chao-dyn
26
The categories of this paper are: bayes-an
11
The categories of this paper are: hep-lat
18
The categories of this paper are: econ
36
The categories of this paper are: cmp-lg
32
The categories of this paper are: atom-ph
0
The categories of this paper are: hep-ph
34
The categories of this paper are: chem-ph
33
The categories of this paper are: plasm-ph
24
The categories of this paper are: dg-ga
10
The categories of this paper are: q-bio