text: Estimating mutual information from i.i.d. samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications. The most popular estimator is one proposed by Kraskov and St "ogbauer and Grassberger (KSG) in 2004, and is nonparametric and based on the distances of each sample to its @math nearest neighboring sample, where @math is a fixed small integer. Despite its widespread use (part of scientific software packages), theoretical properties of this estimator have been largely unexplored. In this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the bias as a function of number of samples. We argue that the superior performance benefits of the KSG estimator stems from a curious "correlation boosting" effect and build on this intuition to modify the KSG estimator in novel ways to construct a superior estimator. As a byproduct of our investigations, we obtain nearly tight rates of convergence of the @math error of the well known fixed @math nearest neighbor estimator of differential entropy by Kozachenko and Leonenko.
  @cite_0: We propose and analyse estimators for statistical functionals of one or more distributions under nonparametric assumptions. Our estimators are derived from the von Mises expansion and are based on the theory of influence functions, which appear in the semiparametric statistics literature. We show that estimators based either on data-splitting or a leave-one-out technique enjoy fast rates of convergence and other favorable theoretical properties. We apply this framework to derive estimators for several popular information theoretic quantities, and via empirical evaluation, show the advantage of this approach over existing estimators.
  @cite_27: 
  @cite_63: We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n-1 2 when the densities' smoothness, s, are both at least d 4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d 4 is necessary to achieve the n-1 2 rate of convergence. We validate our theoretical guarantees with a number of simulations.
summary: In this paper we have considered the smoothness of the class of pdfs studied via bounded Hessians. In nonparametric estimation, a standard feature is to consider whole families of smooth pdfs as defined by how the differences of derivatives relate to the differences of the samples @cite_27 . Of specific interest is the H " o lder family: @math , i.e., for any tuple @math , define @math . Then for any @math such that @math , where @math is the largest integer smaller than @math , we have: for any @math . The rate of convergence of various nonparametric estimators depends on the parameter @math of the H " o lder family under consideration, cf. @cite_63 @cite_0 for recent work on convergence rate characterization of information theoretic quantities via KDE and resubstitution estimators as a function of the smoothness parameter @math . It is natural to ask if such smoothness considerations could lead to a refined understanding of the rates of convergence of the fixed @math -NN KL and KSG estimators studied here.