Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch   Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress
and accelerate the models on resource-constrained environments. It can be
generally categorized into unstructured fine-grained sparsity that zeroes out
multiple individual weights distributed across the neural network, and
structured coarse-grained sparsity which prunes blocks of sub-networks of a
neural network. Fine-grained sparsity can achieve a high compression ratio but
is not hardware friendly and hence receives limited speed gains. On the other
hand, coarse-grained sparsity cannot concurrently achieve both apparent
acceleration on modern GPUs and decent performance. In this paper, we are the
first to study training from scratch an N:M fine-grained structured sparse
network, which can maintain the advantages of both unstructured fine-grained
sparsity and structured coarse-grained sparsity simultaneously on specifically
designed GPUs. Specifically, a 2:4 sparse network could achieve 2x speed-up
without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel
and effective ingredient, sparse-refined straight-through estimator (SR-STE),
to alleviate the negative influence of the approximated gradients computed by
vanilla STE during optimization. We also define a metric, Sparse Architecture
Divergence (SAD), to measure the sparse network's topology change during the
training process. Finally, We justify SR-STE's advantages with SAD and
demonstrate the effectiveness of SR-STE by performing comprehensive experiments
on various tasks. Source codes and models are available at
https://github.com/NM-sparsity/NM-sparsity.

28
The categories of this paper are: alg-geom
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
24
The categories of this paper are: dg-ga
33
The categories of this paper are: plasm-ph
27
The categories of this paper are: comp-gas
10
The categories of this paper are: q-bio
14
The categories of this paper are: math-ph
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
21
The categories of this paper are: adap-org
4
The categories of this paper are: cond-mat
16
The categories of this paper are: stat
20
The categories of this paper are: acc-phys
15
The categories of this paper are: nucl-ex
6
The categories of this paper are: astro-ph