Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Block-Sparse Recurrent Neural Networks   Recurrent Neural Networks (RNNs) are used in state-of-the-art models in
domains such as speech recognition, machine translation, and language
modelling. Sparsity is a technique to reduce compute and memory requirements of
deep learning models. Sparse RNNs are easier to deploy on devices and high-end
server processors. Even though sparse operations need less compute and memory
relative to their dense counterparts, the speed-up observed by using sparse
operations is less than expected on different hardware platforms. In order to
address this issue, we investigate two different approaches to induce block
sparsity in RNNs: pruning blocks of weights in a layer and using group lasso
regularization to create blocks of weights with zeros. Using these techniques,
we demonstrate that we can create block-sparse RNNs with sparsity ranging from
80% to 90% with small loss in accuracy. This allows us to reduce the model size
by roughly 10x. Additionally, we can prune a larger dense network to recover
this loss in accuracy while maintaining high block sparsity and reducing the
overall parameter count. Our technique works with a variety of block sizes up
to 32x32. Block-sparse RNNs eliminate overheads related to data storage and
irregular memory accesses while increasing hardware efficiency compared to
unstructured sparsity.

26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
23
The categories of this paper are: solv-int
13
The categories of this paper are: quant-ph
10
The categories of this paper are: q-bio
27
The categories of this paper are: comp-gas
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
25
The categories of this paper are: patt-sol
18
The categories of this paper are: econ
36
The categories of this paper are: cmp-lg
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess