Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learnability for the Information Bottleneck   The Information Bottleneck (IB) method (\cite{tishby2000information})
provides an insightful and principled approach for balancing compression and
prediction for representation learning. The IB objective $I(X;Z)-\beta I(Y;Z)$
employs a Lagrange multiplier $\beta$ to tune this trade-off. However, in
practice, not only is $\beta$ chosen empirically without theoretical guidance,
there is also a lack of theoretical understanding between $\beta$,
learnability, the intrinsic nature of the dataset and model capacity. In this
paper, we show that if $\beta$ is improperly chosen, learning cannot happen --
the trivial representation $P(Z|X)=P(Z)$ becomes the global minimum of the IB
objective. We show how this can be avoided, by identifying a sharp phase
transition between the unlearnable and the learnable which arises as $\beta$ is
varied. This phase transition defines the concept of IB-Learnability. We prove
several sufficient conditions for IB-Learnability, which provides theoretical
guidance for choosing a good $\beta$. We further show that IB-learnability is
determined by the largest confident, typical, and imbalanced subset of the
examples (the conspicuous subset), and discuss its relation with model
capacity. We give practical algorithms to estimate the minimum $\beta$ for a
given dataset. We also empirically demonstrate our theoretical conditions with
analyses of synthetic datasets, MNIST, and CIFAR10.

24
The categories of this paper are: dg-ga
27
The categories of this paper are: comp-gas
21
The categories of this paper are: adap-org
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
12
The categories of this paper are: nucl-th
8
The categories of this paper are: hep-ex
18
The categories of this paper are: econ
28
The categories of this paper are: alg-geom
7
The categories of this paper are: hep-th
31
The categories of this paper are: ao-sci
20
The categories of this paper are: acc-phys
29
The categories of this paper are: funct-an
26
The categories of this paper are: bayes-an