Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls
  for Network Pruning   We report, for the first time, on the cascade weight shedding phenomenon in
deep neural networks where in response to pruning a small percentage of a
network's weights, a large percentage of the remaining is shed over a few
epochs during the ensuing fine-tuning phase. We show that cascade weight
shedding, when present, can significantly improve the performance of an
otherwise sub-optimal scheme such as random pruning. This explains why some
pruning methods may perform well under certain circumstances, but poorly under
others, e.g., ResNet50 vs. MobileNetV3. We provide insight into why the global
magnitude-based pruning, i.e., GMP, despite its simplicity, provides a
competitive performance for a wide range of scenarios. We also demonstrate
cascade weight shedding's potential for improving GMP's accuracy, and reduce
its computational complexity. In doing so, we highlight the importance of
pruning and learning-rate schedules. We shed light on weight and learning-rate
rewinding methods of re-training, showing their possible connections to the
cascade weight shedding and reason for their advantage over fine-tuning. We
also investigate cascade weight shedding's effect on the set of kept weights,
and its implications for semi-structured pruning. Finally, we give directions
for future research.

2
The categories of this paper are: math
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
33
The categories of this paper are: plasm-ph
35
The categories of this paper are: mtrl-th
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
30
The categories of this paper are: q-alg
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
32
The categories of this paper are: atom-ph
19
The categories of this paper are: eess