Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Online Ensemble Model Compression using Knowledge Distillation   This paper presents a novel knowledge distillation based model compression
framework consisting of a student ensemble. It enables distillation of
simultaneously learnt ensemble knowledge onto each of the compressed student
models. Each model learns unique representations from the data distribution due
to its distinct architecture. This helps the ensemble generalize better by
combining every model's knowledge. The distilled students and ensemble teacher
are trained simultaneously without requiring any pretrained weights. Moreover,
our proposed method can deliver multi-compressed students with single training,
which is efficient and flexible for different scenarios. We provide
comprehensive experiments using state-of-the-art classification models to
validate our framework's effectiveness. Notably, using our framework a 97%
compressed ResNet110 student model managed to produce a 10.64% relative
accuracy gain over its individual baseline training on CIFAR100 dataset.
Similarly a 95% compressed DenseNet-BC(k=12) model managed a 8.17% relative
accuracy gain.

9
The categories of this paper are: nlin
31
The categories of this paper are: ao-sci
6
The categories of this paper are: astro-ph
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
0
The categories of this paper are: hep-ph
24
The categories of this paper are: dg-ga
15
The categories of this paper are: nucl-ex
11
The categories of this paper are: hep-lat
5
The categories of this paper are: gr-qc