Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Fine-tuning Pruned Networks with Linear Over-parameterization   Structured pruning compresses neural networks by reducing channels (filters)
for fast inference and low footprint at run-time. To restore accuracy after
pruning, fine-tuning is usually applied to pruned networks. However, too few
remaining parameters in pruned networks inevitably bring a great challenge to
fine-tuning to restore accuracy. To address this challenge, we propose a novel
method that first linearly over-parameterizes the compact layers in pruned
networks to enlarge the number of fine-tuning parameters and then
re-parameterizes them to the original layers after fine-tuning. Specifically,
we equivalently expand the convolution/linear layer with several consecutive
convolution/linear layers that do not alter the current output feature maps.
Furthermore, we utilize similarity-preserving knowledge distillation that
encourages the over-parameterized block to learn the immediate data-to-data
similarities of the corresponding dense layer to maintain its feature learning
ability. The proposed method is comprehensively evaluated on CIFAR-10 and
ImageNet which significantly outperforms the vanilla fine-tuning strategy,
especially for large pruning ratio.

25
The categories of this paper are: patt-sol
33
The categories of this paper are: plasm-ph
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
12
The categories of this paper are: nucl-th
11
The categories of this paper are: hep-lat
26
The categories of this paper are: bayes-an
5
The categories of this paper are: gr-qc
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
34
The categories of this paper are: chem-ph
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
15
The categories of this paper are: nucl-ex
30
The categories of this paper are: q-alg