Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: KNIFE: Kernelized-Neural Differential Entropy Estimation   Mutual Information (MI) has been widely used as a loss regularizer for
training neural networks. This has been particularly effective when learn
disentangled or compressed representations of high dimensional data. However,
differential entropy (DE), another fundamental measure of information, has not
found widespread use in neural network training. Although DE offers a
potentially wider range of applications than MI, off-the-shelf DE estimators
are either non differentiable, computationally intractable or fail to adapt to
changes in the underlying distribution. These drawbacks prevent them from being
used as regularizers in neural networks training. To address shortcomings in
previously proposed estimators for DE, here we introduce KNIFE, a fully
parameterized, differentiable kernel-based estimator of DE. The flexibility of
our approach also allows us to construct KNIFE-based estimators for conditional
(on either discrete or continuous variables) DE, as well as MI. We empirically
validate our method on high-dimensional synthetic data and further apply it to
guide the training of neural networks for real-world tasks. Our experiments on
a large variety of tasks, including visual domain adaptation, textual fair
classification, and textual fine-tuning demonstrate the effectiveness of
KNIFE-based estimation. Code can be found at
https://github.com/g-pichler/knife.

27
The categories of this paper are: comp-gas
26
The categories of this paper are: bayes-an
22
The categories of this paper are: chao-dyn
33
The categories of this paper are: plasm-ph
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol
32
The categories of this paper are: atom-ph
10
The categories of this paper are: q-bio
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
4
The categories of this paper are: cond-mat
7
The categories of this paper are: hep-th
3
The categories of this paper are: physics
14
The categories of this paper are: math-ph
30
The categories of this paper are: q-alg