Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural
  Networks   Existing knowledge distillation methods on graph neural networks (GNNs) are
almost offline, where the student model extracts knowledge from a powerful
teacher model to improve its performance. However, a pre-trained teacher model
is not always accessible due to training cost, privacy, etc. In this paper, we
propose a novel online knowledge distillation framework to resolve this
problem. Specifically, each student GNN model learns the extracted local
structure from another simultaneously trained counterpart in an alternating
training procedure. We further develop a cross-layer distillation strategy by
aligning ahead one student layer with the layer in different depth of another
student model, which theoretically makes the structure information spread over
all layers. Experimental results on five datasets including PPI,
Coauthor-CS/Physics and Amazon-Computer/Photo demonstrate that the student
performance is consistently boosted in our collaborative training framework
without the supervision of a pre-trained teacher model. In addition, we also
find that our alignahead technique can accelerate the model convergence speed
and its effectiveness can be generally improved by increasing the student
numbers in training. Code is available:
https://github.com/GuoJY-eatsTG/Alignahead

30
The categories of this paper are: q-alg
7
The categories of this paper are: hep-th
14
The categories of this paper are: math-ph
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
22
The categories of this paper are: chao-dyn
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
23
The categories of this paper are: solv-int
3
The categories of this paper are: physics
32
The categories of this paper are: atom-ph
10
The categories of this paper are: q-bio
15
The categories of this paper are: nucl-ex