# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # DATA 5600: Introduction to Regression and Machine Learning for Analytics
#
# ## __Koop Chapter 03: Correlation__ <br>
#
# Author:  <NAME> <br>
# Updated: October 31, 2021 <br>
#
# ---
#
# <br>

# +
import numpy as np
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = [10, 8]
# -

np.random.seed(7)

# ---

# <br>
#
# ## __Introduction__
#
# <br>
#
#
# These notes are taken from the book _Analysis of Economic Data 2nd Edition_ by <NAME>.
#
# <br>
#
# A solid understanding of the concept of correlation is essential for doing regression.
#
#
# <br>

# ## __Understanding Correlaiton__
#
# <br>
#
# * Let $X$ and $Y$ be two variables
#
# * Let's suppose we have data on $i = 1, 2, \ldots, N$ different units/observations
#
# * The ___correlation___ between $X$ and $Y$ is denoted by Greek letter $\rho$ ("rho")
#
#

# ### __Sample Correlation__
#
# ---
#
# The __sample correlation__ between $X$ and $Y$ is referred to by the letter $r$ and is calculated as: 
#
# <br>
#
# $$
# r = \frac{\sum\limits_{i=1}^{N} (Y_{i} - \bar{Y}) (X_{i} - \bar{X})}{\sqrt{\sum\limits_{i=1}^{N} (Y_{i} - \bar{Y})^{2} \sum\limits_{i=1}^{N} (X_{i} - \bar{X})^{2}}}
# $$
#
# <br>
#
# __NB:__ the population correlation is denoted by Greek letter $\rho$
#
# ---
#
# <br>
# <br>

# ### __Properties of Correlation__
#
# <br>
#
# 1. $r$ always lies between -1 and 1, which may be written as $-1 \le \rho \le 1$
#
# 2. Positive values of $r$ indicate a positive correlation between $X$ and $Y$. Negative values indicate a negative correlation. $r = 0$ indicates that $X$ and $Y$ are uncorrelated. 
#
# 3. Larger positive values of $r$ indicate stronger positive correlation. $r = 1$ indicates perfect positive correlation. Larger negative values of $r$ indicate stronger negative correlation. $r = -1$ indicates perfect negative correlation.
#
# 4. The correlation between $X$ and $Y$ is the same as the correlation between $Y$ and $X$.
#
# 5. The correlation between any variable and itself (e.g. the correlation between $X$ and $X$) is 1.
#
# <br>

# ## __Understanding Correlation Through Verbal Reasoning__
#
# <br>
#
# * Data scientists (statisticians, econometricians, etc) use the word correlation in much the same as the lay person
#
# * We will look at an example of deforestation/population density example to illustrate verbally
#
# <br>
#
# <u><b>Example:The Correlation Between Deforestation and Population Density</b></u> 
#
# Let's look at the file `FOREST.XLS`
#
# <br>

df = pd.read_excel('FOREST.XLS')

df.head(25)

df.tail()

df.describe()

# +
x = df['Forest loss']
y = df['Pop dens']

(x.corr(y), y.corr(x))
# -

np.corrcoef(df['Forest loss'], df['Pop dens'])

stats.pearsonr(x, y)

# <br>
#
# We find that the correlation between deforestation and population density is $0.66$
#
# Being a postive number allows us to say the following:
#
# 1. There is a positive relationship (or positive association) between deforestation and population density
#
# 2. Countries with high population densities tend to have high deforestation. Countries with low population densities tend to have low levels of deforestation
#     - NB: note the word _"tend"_ here
#     - This is not a causal relationship, but rather a "general tendency"
#     - It outlines a broad pattern that may not hold in particular cases
#
# 3. Deforestation rates vary across countries as do population densities (thus the name "variables")
#     - Some countries have high deforestation, others have low rates
#     - This high/low cross-country variance in deforestation rates tends to "match up" with the high/low variance in population density
#     
# These states are based on the positive value of $r$. If it were negative the opposite statements would be true
#
#
# 4. The degree to which deforestation rates vary across countries can be measured numerically using the formula for the standard deviation
#     - The fact that deforestation and population density are positively correlated means that their patterns of cross-country variability tend to match up
#     - The correlation squared $r^{2}$ measures the proportion of the cross-country variability in deforestation that matches up with, or is explained by, the variance in population density
#     - Correlation is a numerical measure of the degree to which patterns in $X$ and $Y$ correspond
#     - In our example $0.66^{2} = 0.44$, we can say that $44\%$ of the cross-country variance in deforestation can be explained by the cross-country variance in population density
#     
# <br>

x.corr(y) ** 2.0

# <br>
#
#
# <br>

# <br>
#
# <u><b>Example 2: House Prices in Windsor, Cananda</b></u>
#
# <br>

house_prices = pd.read_excel("HPRICE.XLS")

house_prices.head()

house_prices.describe()

# <br>
#
# * Let $Y =$ the sales price of the house
#
# * Let $X =$ the size of the lot in square feet
#
# * $r_{X,Y} = 0.54$
#
# <br>

y = house_prices['sale price']
x = house_prices['lot size']
y.corr(x)

# <br>
#
# The following statements can be made:
#
# 1. Houses with large lots tend to be worth more than those with small lots
#
# 2. There is a positive relationship between lot size and sales price
#
# 3. The variance in lot size accounts for $29\%$ (i.e. $0.54^{2} = 0.29$) or the variability in house prices
#
# <br>

y.corr(x) ** 2.0

# <br>
#
# * Now let's add a 3rd variable $Z =$ the number of bedrooms
#
# * Calculating the correlation between sales price and the number of bedroooms we obtain $r_{Y,Z} = 0.37$
#
# <br>

z = house_prices['#bedroom']
y.corr(z)

x.corr(z)

# <br>
#
# ### __Causality__
#
# * We are often interested finding out whether or not one variable "causes" another
#
# * We will not give a complete definition of causality here
#
# * We can use the positive correlation to get at this concept
#
# * Lot size is a variable that directly influences (roughly causes) sales price
#
# * House prices do not influence (cause) lot size
#
# * In other words, the direction of causality flows from lot size to house prices, but not the other way around
#
# * __Q:__ what would happen if a homeowner were to purchase some adjacent land and thereby increase the lot size?
#     - This action would tend to increase the value of the house 
#     - "will increasing the price of the house cause lot size to increase?" Obviously not.
#     
# * We can make similar statements about number of bedrooms
#
# <br>
#
# * It is important to know how to interpret results
#
# * This house price example illustrates this principle
#
# * It is not enough to simply report that $r_{Y,X} = 0.54$
#
# * Interpretation requires a good intuitive knowledge of what a correlation is in conjunction with common sense about the phenomenon under study
#
# <br>

# ### __Exercise__
#
# * __(A)__ Using the data in `HPRICE.XLS`, calculate and interpret the mean, standard deviation, minimum and maximum of $Y = $ sales price, $X =$ lot size and $Z =$ number of bedrooms
#
# * __(B)__ Verify that the correlation between $X$ and $Y$ is the same as given in the example. Repeat for $X$ and $Z$ then for $Y$ and $Z$
#
# * __(C)__ Now add a new variable, $W =$ number of bathrooms. Calculate the mean of $W$
#
# * __(D)__ Calculate and interpret the correlation between $W$ and $Y$. Discuss to what extent it can be said that $W$ causes $Y$.
#
# * __(E)__ Repeat part (d) for $W$ and $X$ and then for $W$ and $Z$.
#
# <br>

# <br>

# ## __Understanding Why Variables Are Correlated__
#
# <br>
#
# * In the deforestation/population density example there was a positive correlation
#
#
# * But what exact form does this relationship take?
#
# * We like to think in terms of causality
#
# * It may be the case that the positive correlation means that population density causes deforestation
#
# * In a wage regression it might mean that a positive correlation between education levels and wages can be interepreted as additional education causes higher wages
#
# * But we must be cautious because, while correlation can be high between variables, it need not mean that one causes the other
#
# <br>

# ### __Correlation Does Not Necessarily Imply Causality__
#
# <br>
#
# * See here: https://www.tylervigen.com/spurious-correlations
#
# <br>



StarcoderdataJupyterScriptsDedupFiltered