Entropy (information theory)
Measure theory

Entropy can be formally defined in the language of measure theory as follows: Let (X, Σ, μ) {\displaystyle (X,\Sigma,\mu)} be a probability space. Let A ∈ Σ {\displaystyle A\in \Sigma } be an event. The surprisal of A {\displaystyle A} is σ μ (A) = − ln μ (A). {\displaystyle \sigma _{\mu }(A)=-\ln \mu (A).}

The expected surprisal of A {\displaystyle A} is h μ (A) = μ (A) σ μ (A). {\displaystyle h_{\mu }(A)=\mu (A)\sigma _{\mu }(A).}

A μ {\displaystyle \mu } -almost partition is a set family P ⊆ P (X) {\displaystyle P\subseteq {\mathcal {P}}(X)} such that μ (∪ P) = 1 {\displaystyle \mu (\mathop {\cup } P)=1} and μ (A ∩ B) = 0 {\displaystyle \mu (A\cap B)=0} for all distinct A, B ∈ P {\displaystyle A,B\in P}. (This is a relaxation of the usual conditions for a partition.) The entropy of P {\displaystyle P} is H μ (P) = ∑ A ∈ P h μ (A). {\displaystyle \mathrm {H} _{\mu }(P)=\sum _{A\in P}h_{\mu }(A).}

Let M {\displaystyle M} be a sigma-algebra on X {\displaystyle X}. The entropy of M {\displaystyle M} is H μ (M) = sup P ⊆ M H μ (P). {\displaystyle \mathrm {H} _{\mu }(M)=\sup _{P\subseteq M}\mathrm {H} _{\mu }(P).} Finally, the entropy of the probability space is H μ (Σ) {\displaystyle \mathrm {H} _{\mu }(\Sigma)}, that is, the entropy with respect to μ {\displaystyle \mu } of the sigma-algebra of all measurable subsets of X {\displaystyle X}.
punctuation:no_comma, detectable_format:multiple_sections[5]