Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Co-advise: Cross Inductive Bias Distillation   Transformers recently are adapted from the community of natural language
processing as a promising substitute of convolution-based neural networks for
visual learning tasks. However, its supremacy degenerates given an insufficient
amount of training data (e.g., ImageNet). To make it into practical utility, we
propose a novel distillation-based method to train vision transformers. Unlike
previous works, where merely heavy convolution-based teachers are provided, we
introduce lightweight teachers with different architectural inductive biases
(e.g., convolution and involution) to co-advise the student transformer. The
key is that teachers with different inductive biases attain different knowledge
despite that they are trained on the same dataset, and such different knowledge
compounds and boosts the student's performance during distillation. Equipped
with this cross inductive bias distillation method, our vision transformers
(termed as CivT) outperform all previous transformers of the same architecture
on ImageNet.

5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
37
The categories of this paper are: supr-con
34
The categories of this paper are: chem-ph
35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math
15
The categories of this paper are: nucl-ex
9
The categories of this paper are: nlin
8
The categories of this paper are: hep-ex
6
The categories of this paper are: astro-ph
33
The categories of this paper are: plasm-ph
20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
11
The categories of this paper are: hep-lat