Jensen's inequality
Information theory

If p (x) is the true probability density for X, and q (x) is another density, then applying Jensen's inequality for the random variable Y (X) = q (X)/ p (X) and the convex function φ (y) = −log(y) gives

Therefore:

a result called Gibbs' inequality.

It shows that the average message length is minimised when codes are assigned on the basis of the true probabilities p rather than any other distribution q. The quantity that is non-negative is called the Kullback–Leibler divergence of q from p, where D (p (x) ‖ q (x)) = ∫ p (x) log (p (x) q (x)) d x {\displaystyle D(p(x)\|q(x))=\int p(x)\log \left({\frac {p(x)}{q(x)}}\right)dx}.

Since −log(x) is a strictly convex function for x > 0, it follows that equality holds when p (x) equals q (x) almost everywhere.
detectable_content:postscript, change_case:capital_word_frequency[3], keywords:forbidden_words