Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Information-Theoretic Understanding of Population Risk Improvement with
  Model Compression   We show that model compression can improve the population risk of a
pre-trained model, by studying the tradeoff between the decrease in the
generalization error and the increase in the empirical risk with model
compression. We first prove that model compression reduces an
information-theoretic bound on the generalization error; this allows for an
interpretation of model compression as a regularization technique to avoid
overfitting. We then characterize the increase in empirical risk with model
compression using rate distortion theory. These results imply that the
population risk could be improved by model compression if the decrease in
generalization error exceeds the increase in empirical risk. We show through a
linear regression example that such a decrease in population risk due to model
compression is indeed possible. Our theoretical results further suggest that
the Hessian-weighted $K$-means clustering compression approach can be improved
by regularizing the distance between the clustering centers. We provide
experiments with neural networks to support our theoretical assertions.

19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
21
The categories of this paper are: adap-org
20
The categories of this paper are: acc-phys
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
34
The categories of this paper are: chem-ph
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
25
The categories of this paper are: patt-sol
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci