Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Compressing Language Models using Doped Kronecker Products   Kronecker Products (KP) have been used to compress IoT RNN Applications by
15-38x compression factors, achieving better results than traditional
compression methods. However when KP is applied to large Natural Language
Processing tasks, it leads to significant accuracy loss (approx 26%). This
paper proposes a way to recover accuracy otherwise lost when applying KP to
large NLP tasks, by allowing additional degrees of freedom in the KP matrix.
More formally, we propose doping, a process of adding an extremely sparse
overlay matrix on top of the pre-defined KP structure. We call this compression
method doped kronecker product compression. To train these models, we present a
new solution to the phenomenon of co-matrix adaption (CMA), which uses a new
regularization scheme called co matrix dropout regularization (CMR). We present
experimental results that demonstrate compression of a large language model
with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score. At
25x compression, an equivalent pruned network leads to 7.9% loss in perplexity
score, while HMD and LMF lead to 15% and 27% loss in perplexity score
respectively.

8
The categories of this paper are: hep-ex
32
The categories of this paper are: atom-ph
6
The categories of this paper are: astro-ph
35
The categories of this paper are: mtrl-th
19
The categories of this paper are: eess
21
The categories of this paper are: adap-org
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
30
The categories of this paper are: q-alg
37
The categories of this paper are: supr-con
5
The categories of this paper are: gr-qc
36
The categories of this paper are: cmp-lg
17
The categories of this paper are: q-fin
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
12
The categories of this paper are: nucl-th