<jupyter_start><jupyter_text>GITHUB 121 LSDS9https://github.com/maiormarso/DS-Unit-1-Sprint-2-StatisticsLecture Youtube.https://www.youtube.com/watch?v=zhPGoHZo5i4&feature=youtu.be *Data Science Unit 1 Sprint 2 Assignment 1* Apply the t-test to real dataYour assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!Your goals:1. Load and clean the data (or determine the best method to drop observations when running tests)2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.013. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.014. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.Stretch goals:1. Refactor your code into functions so it's easy to rerun with arbitrary variables2. Apply hypothesis testing to your personal project data (for the purposes of this notebook you can type a summary of the hypothesis you formed and tested)<jupyter_code># TODO - during class, but please help!
from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel
import scipy.stats
dir(scipy.stats)
# Getting started with the assignment
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
# Load Data
import pandas as pd
import numpy as np
df = pd.read_csv('house-votes-84.data', 
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
print(df.shape)
df.head()
df = df.replace({'?':np.NaN, 'n':0, 'y':1})

df.head()
df.isnull().sum()
rep = df[df.party == "republican"]
print(rep.shape)
rep.head()
dem = df[df.party == "democrat"]
print(rep.shape)
dem.head()
df.describe()
rep.describe()
dem.describe()<jupyter_output><empty_output><jupyter_text>2. Using hypothesis testing,find an issue that democrats support more than republicans with p < 0.01<jupyter_code># Example 1 sample t-test
from scipy.stats import ttest_1samp

print(rep['aid-to-contras'].mean())
print(dem['aid-to-contras'].mean())
dem['aid-to-contras'].mean()
dem['aid-to-contras'].value_counts()
rep['aid-to-contras'].value_counts()<jupyter_output><empty_output><jupyter_text>The null hypothesis for the aid-to-contras bill is that the bill will pass. Therefor the correct parameter will be 1.The alternative hypothesis would be that the aid-to-contra bill is that the bill would not pass.Confidence level is 95%.The T.statistic is -7.354085178140069pvalue=0.004363402589282088<jupyter_code># For now, 1-sample hyopthesis - republican support is 0
ttest_1samp(dem['aid-to-contras'], 1, nan_policy='omit')<jupyter_output><empty_output><jupyter_text>3.Using hypothesis testing, find an issue thatrepublicans support more than democrats with p < 0.01<jupyter_code># Example 1 sample t-test
from scipy.stats import ttest_1samp

print(rep['el-salvador-aid'].mean())
print(dem['el-salvador-aid'].mean())
rep['el-salvador-aid'].mean()
rep['el-salvador-aid'].value_counts()
dem['el-salvador-aid'].value_counts()<jupyter_output><empty_output><jupyter_text>The null hypothesis for the el-salvador-aid bill is that the bill will pass. There for the correct parameter will be 1.The alternative hypothesis would be that the el-slavador-aid bill is that the bill would not pass.Confidence level is 95%.The T.statistic is -2.890pvalue=0.004363402589282088<jupyter_code># For now, 1-sample hyopthesis - republican support is 0
ttest_1samp(rep['el-salvador-aid'], 1, nan_policy='omit')
# For the last test, we're much closer to the null - fail to reject!
# And now, like the homework - 2 sample
# 2 sample compares the means of two samples, null is that they're the same
# 1 sample compared a mean to a given null (above)

ttest_ind(rep['el-salvador-aid'], dem['el-salvador-aid'],
          nan_policy='omit')<jupyter_output><empty_output><jupyter_text>4. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)<jupyter_code>dem.sum(axis = 0, skipna = True) 
rep.sum(axis = 0, skipna = True) 
# Example 1 sample t-test
from scipy.stats import ttest_1samp

print(dem['religious-groups'].mean())
print(rep['religious-groups'].mean())
rep['religious-groups'].mean()
rep['religious-groups'].value_counts()
dem['religious-groups'].value_counts()<jupyter_output><empty_output><jupyter_text>The null hypothesis for the religious-groups bill is that the bill will pass. There for the correct parameter will be 1.The alternative hypothesis would be that the el-slavador-aid bill is that the bill would not pass.Confidence level is 95%.The T.statistic is -2.890pvalue=0.004363402589282088<jupyter_code># For now, 1-sample hyopthesis - republican support is 0
ttest_1samp(rep['religious-groups'], 1, nan_policy='omit')
# For the last test, we're much closer to the null - fail to reject!
# And now, like the homework - 2 sample
# 2 sample compares the means of two samples, null is that they're the same
# 1 sample compared a mean to a given null (above)

ttest_ind(rep['religious-groups'], dem['religious-groups'],
          nan_policy='omit')<jupyter_output><empty_output>