# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# ## Exercise 3.21 Mutual information for naive Bayes classifiers with binary features 
# Derive Equation 3.76.

# The mutual information between feature $X_j$ and the class label $Y$ is given by 
#
# $$
# I(X, Y) = \sum_{x_j}\sum_{y}p(x_j, y)\log\frac{p(x_j, y)}{p(x_j)p(y)}
# $$
#
# The mutual information can be thought of as the reduction in entropy on the label distribution once we observe the value of feature $j$.

# If the features are binary, then
#
# \begin{aligned}
# I_j & = \sum_y p(x_j=0, y)\log\frac{p(x_j=0, y)}{p(x_j=0)p(y)} + p(x_j=1, y)\log\frac{p(x_j=1, y)}{p(x_j=1)p(y)} \\
# & = \sum_y p(x_j=0|y)p(y)\log\frac{p(x_j=0| y)}{p(x_j=0)} + p(x_j=1|y)p(y)\log\frac{p(x_j=1|y)}{p(x_j=1)} \\
# & = \sum_{c}(1-p(x_j=1|y=c))p(y=c)\log\frac{(1-p(x_j=1|y=c))}{(1-p(x_j=1))} + p(x_j=1|y=c)p(y=c)\log\frac{p(x_j=1|y=c)}{p(x_j=1)}
# \end{aligned}

# If we set $\pi_c = p(y=c)$, $\theta_{jc} = p(x_j=1|y=c)$, and $\theta_j = p(x_j=1) = \sum_c\pi_c\theta_{jc}$, then we have
#
# $$
# I_j = \sum_c\left[(1-\theta_{jc})\pi_c\log\frac{1-\theta_{jc}}{1-\theta_j}+ \theta_{jc}\pi_c\log\frac{\theta_{jc}}{\theta_j}\right]
# $$

StarcoderdataJupyterScriptsDedupFiltered