Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-label Contrastive Predictive Coding   Variational mutual information (MI) estimators are widely used in
unsupervised representation learning methods such as contrastive predictive
coding (CPC). A lower bound on MI can be obtained from a multi-class
classification problem, where a critic attempts to distinguish a positive
sample drawn from the underlying joint distribution from $(m-1)$ negative
samples drawn from a suitable proposal distribution. Using this approach, MI
estimates are bounded above by $\log m$, and could thus severely underestimate
unless $m$ is very large. To overcome this limitation, we introduce a novel
estimator based on a multi-label classification problem, where the critic needs
to jointly identify multiple positive samples at the same time. We show that
using the same amount of negative samples, multi-label CPC is able to exceed
the $\log m$ bound, while still being a valid lower bound of mutual
information. We demonstrate that the proposed approach is able to lead to
better mutual information estimation, gain empirical improvements in
unsupervised representation learning, and beat a current state-of-the-art
knowledge distillation method over 10 out of 13 tasks.

27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
7
The categories of this paper are: hep-th
0
The categories of this paper are: hep-ph
24
The categories of this paper are: dg-ga
18
The categories of this paper are: econ
26
The categories of this paper are: bayes-an
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
12
The categories of this paper are: nucl-th
11
The categories of this paper are: hep-lat
37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
8
The categories of this paper are: hep-ex
13
The categories of this paper are: quant-ph