Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection   Recent empirical works show that large deep neural networks are often highly
redundant and one can find much smaller subnetworks without a significant drop
of accuracy. However, most existing methods of network pruning are empirical
and heuristic, leaving it open whether good subnetworks provably exist, how to
find them efficiently, and if network pruning can be provably better than
direct training using gradient descent. We answer these problems positively by
proposing a simple greedy selection approach for finding good subnetworks,
which starts from an empty network and greedily adds important neurons from the
large network. This differs from the existing methods based on backward
elimination, which remove redundant neurons from the large network.
Theoretically, applying the greedy selection strategy on sufficiently large
{pre-trained} networks guarantees to find small subnetworks with lower loss
than networks directly trained with gradient descent. Our results also apply to
pruning randomly weighted networks. Practically, we improve prior arts of
network pruning on learning compact neural architectures on ImageNet, including
ResNet, MobilenetV2/V3, and ProxylessNet. Our theory and empirical results on
MobileNet suggest that we should fine-tune the pruned subnetworks to leverage
the information from the large model, instead of re-training from new random
initialization as suggested in \citet{liu2018rethinking}.

31
The categories of this paper are: ao-sci
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
26
The categories of this paper are: bayes-an
13
The categories of this paper are: quant-ph
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
27
The categories of this paper are: comp-gas
3
The categories of this paper are: physics
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin