text: Estimating mutual information from i.i.d. samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications. The most popular estimator is one proposed by Kraskov and St "ogbauer and Grassberger (KSG) in 2004, and is nonparametric and based on the distances of each sample to its @math nearest neighboring sample, where @math is a fixed small integer. Despite its widespread use (part of scientific software packages), theoretical properties of this estimator have been largely unexplored. In this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the bias as a function of number of samples. We argue that the superior performance benefits of the KSG estimator stems from a curious "correlation boosting" effect and build on this intuition to modify the KSG estimator in novel ways to construct a superior estimator. As a byproduct of our investigations, we obtain nearly tight rates of convergence of the @math error of the well known fixed @math nearest neighbor estimator of differential entropy by Kozachenko and Leonenko.
  @cite_64: In the random sampling setting we estimate the entropy of a probability density distribution by the entropy of a kernel density estimator using the double exponential kernel. Under mild smoothness and moment conditions we show that the entropy of the kernel density estimator equals a sum of independent and identically distributed (i.i.d.) random variables plus a perturbation which is asymptotically negligible compared to the parametric rate n sup -1 2 . An essential part in the proof is obtained by exhibiting almost sure bounds for the Kullback-Leibler divergence between the kernel density estimator and its expected value. The basic technical tools are Doob's submartingale inequality and convexity (Jensen's inequality).
  @cite_60: For a multivariate density f with respect to Lebesgue measure μ, the estimation of MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba Waa8qaaeaacaWGkbGaaiikaiaadAgacaGGPaGaamOzaiaadsgacqaH 8oqBaSqabeqaniabgUIiYdaaaa!4404! [ J(f)fd ], and in particular MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba Waa8qaaeaacaWGMbWaaWbaaSqabeaacaaIYaaaaOGaamizaiabeY7a TbWcbeqab0Gaey4kIipaaaa!41E4! [ f^2 d ] and MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba Waa8qaaeaacaWGMbGaciiBaiaac+gacaGGNbGaamOzaiaadsgacqaH 8oqBaSqabeqaniabgUIiYdaaaa!44AC! [ f fd ], is studied. These two particular functionals are important in a number of contexts. Asymptotic bias and variance terms are obtained for the estimators MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba WaaybyaeqaleqabaGaey4jIKnaneaacaWGjbaaaOGaeyypa0Zaa8qa aeaacaWGkbGaaiikamaawagabeWcbeqaaiabgEIizdqdbaGaamOzaa aakiaacMcacaWGKbGaamOramaaBaaaleaacaWGobaabeaaaeqabeqd cqGHRiI8aaaa!4994! [ I ^ = J( f ^ )dF_N ] and MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba WaaybyaeqaleqabaGaeSipIOdaneaacaWGjbaaaOGaeyypa0Zaa8qa aeaacaWGkbGaaiikamaawagabeWcbeqaaiabgEIizdqdbaGaamOzaa aakiaacMcadaGfGbqabSqabeaacqGHNis2a0qaaiaadAgaaaGccaWG KbGaeqiVd0galeqabeqdcqGHRiI8aaaa!4C40! [ I ^ = J( f ^ ) f ^ d ], where MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba WaaybyaeqaleqabaGaey4jIKnaneaacaWGMbaaaaaa!3E9C! [ f ^ ] is a kernel density estimate of f and F n is the empirical distribution function based on the random sample X 1 ,..., X n from f. For the two functionalsmentioned above, a first order bias term for I can be made zero by appropriate choices of non-unimodal kernels. Suggestions for the choice of bandwidth are given; for MathType!MTEF!2!1!+- feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXafv3ySLgzGmvETj2BSbqefm0B1jxALjhiov2D aebbfv3ySLgzGueE0jxyaibaiiYdd9qrFfea0dXdf9vqai-hEir8Ve ea0de9qq-hbrpepeea0db9q8as0-LqLs-Jirpepeea0-as0Fb9pgea 0lrP0xe9Fve9Fve9qapdbaqaaeGacaGaaiaabeqaamaabaabcaGcba WaaybyaeqaleqabaGaey4jIKnaneaacaWGjbaaaOGaeyypa0Zaa8qa aeaadaGfGbqabSqabeaacqGHNis2a0qaaiaadAgaaaGccaWGKbGaam OramaaBaaaleaacaWGobaabeaaaeqabeqdcqGHRiI8aaaa!476C! [ I ^ = f ^ dF_N ], a study of optimal bandwidth is possible.
  @cite_41: Motivated by recent work of Joe (1989,Ann. Inst. Statist. Math.,41, 683–697), we introduce estimators of entropy and describe their properties. We study the effects of tail behaviour, distribution smoothness and dimensionality on convergence properties. In particular, we argue that root-n consistency of entropy estimation requires appropriate assumptions about each of these three features. Our estimators are different from Joe's, and may be computed without numerical integration, but it can be shown that the same interaction of tail behaviour, smoothness and dimensionality also determines the convergence rate of Joe's estimator. We study both histogram and kernel estimators of entropy, and in each case suggest empirical methods for choosing the smoothing parameter.
  @cite_19: Let F(x) be an absolutely continuous distribution having a density function f(x) with respect to the Lebesgue measure. The Shannon entropy is defined as H(f) = - f(x) f(x) dx . In this correspondence we propose, based on a random sample X_ 1 , , X_ n generated from F , a nonparametric estimate of H(f) given by H (f) = -(l n) i = 1 ^ n f (x) , where f (x) is the kernel estimate of f due to Rosenblatt and Parzen. Regularity conditions are obtained under which the first and second mean consistencies of H (f) are established. These conditions are mild and easily satisfied. Examples, such as Gamma, Weibull, and normal distributions, are considered.
  @cite_59: Introduction. Survey of Existing Methods. The Kernel Method for Univariate Data. The Kernel Method for Multivariate Data. Three Important Methods. Density Estimation in Action.
  @cite_34: We develop a ldquoplug-inrdquo kernel estimator for the differential entropy that is consistent even if the kernel width tends to zero as quickly as 1 N, where N is the number of independent and identically distributed (i.i.d.) samples. Thus, accurate density estimates are not required for accurate kernel entropy estimates; in fact, it is a good idea when estimating entropy to sacrifice some accuracy in the quality of the corresponding density estimate.
summary: A conceptually straightforward way to estimate the differential entropy and mutual information is to use a kernel density estimator (KDE) @cite_59 @cite_60 @cite_19 @cite_64 @cite_34 @cite_41 : the densities @math are separately estimated from samples and the estimated densities are then used to calculate the entropy and mutual information via the resubstitution estimator. A typical approach to avoid overfitting is to conduct data splitting (DS): split the samples and use one part for KDE and the other for the resubstitution.