stylistic:style_transfer
Data as a Markov process

A common way to define entropy for text is based on the Markov model of text. For an order-0 source (each character is selected independent of the last characters), the binary entropy is:

where p is the probability of i. For a first-order Markov source (one in which the probability of selecting a character is dependent only on the immediately preceding character), the entropy rate is:

where i is a state (certain preceding characters) and p i (j) {\displaystyle p_{i}(j)} is the probability of j given i as the previous character.

For a second order Markov source, the entropy rate is