<jupyter_start><jupyter_text>*Data Science Unit 1 Sprint 3 Assignment 1* Apply the t-test to real dataYour assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!Your goals:1. Load and clean the data (or determine the best method to drop observations when running tests)2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.013. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.014. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)5. Practice 1-sample t-testsNote that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.Stretch goals:1. Refactor your code into functions so it's easy to rerun with arbitrary variables2. Apply hypothesis testing to your personal project data (for the purposes of this notebook you can type a summary of the hypothesis you formed and tested)3. Add visuals<jupyter_code>import pandas as pd
import numpy as np
import scipy
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import ttest_1samp
from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
#import with column names and NaN values.
df = pd.read_csv('house-votes-84.data', 
                 header=None,
                 na_values='?',
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
df.head()
#str to int
df = df.replace({'y': 1, 'n':0})
#Separate into our 'Samples'
dems = df[df['party'] == 'democrat']
reps = df[df['party'] == 'republican']
cols=['handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa']
dems.head()
reps.head()
#looking at output and practice.
ttest_ind(reps['budget'], dems['budget'], nan_policy='omit')
#Let's do some comprehension to speed things up.
results = {column : ttest_ind(reps[column], dems[column], nan_policy='omit') for column in cols}
results
#could make this even more general as a function.
def get_test_scores(columns_as_list, DataFrame1, DataFrame2, nan_policy):
    return {col : ttest_ind(DataFrame1[col], DataFrame2[col], nan_policy=nan_policy) for col in columns_as_list}
get_test_scores(cols, reps, dems, 'omit')
#Dems support more
print(f"Dems support 'budget' more than republicans with a score of {results['budget']}")
#Reps Support more
print(f"Republicans support 'crime' more than republicans with a score of {results['crime']}")
#Pretty equal on support
print(f"Dems and republicans seem to both support 'water-project with a score of {results['water-project']}")<jupyter_output>Dems support 'budget' more than republicans with a score of Ttest_indResult(statistic=-23.21277691701378, pvalue=2.0703402795404463e-77)
Republicans support 'crime' more than republicans with a score of Ttest_indResult(statistic=16.342085656197696, pvalue=9.952342705606092e-47)
Dems and republicans seem to both support 'water-project with a score of Ttest_indResult(statistic=0.08896538137868286, pvalue=0.9291556823993485)<jupyter_text>1) Null Hypothesis:In 1-sample t-tests YOU GET TO CHOOSE YOUR NULL HYPOTHESISH0 : 0.0 - There is ZERO republican support for the duty-free bill2) Alternative HypothesisHa :  x¯≠0  - There is non-zero support for the budget bill among repulbicans.3) Confidence Level: 95% or .95<jupyter_code>ttest_1samp(reps['duty-free'], 0, nan_policy='omit')<jupyter_output><empty_output><jupyter_text>4) t-statistic: 3.95) p-value of .00013809--- Conclusion: Due to a p-value of near-zero, I reject the null hypothesis that republican support is zero and conclude that republican support is non-zero.  1) Null Hypothesis:In 1-sample t-tests YOU GET TO CHOOSE YOUR NULL HYPOTHESISH0 : 0.0 - There is ZERO republican support for the crime bill2) Alternative HypothesisHa : x¯≠0 - There is non-zero support for the budget bill among Democrats.3) Confidence Level: 95% or .95<jupyter_code>ttest_1samp(dems['crime'],0, nan_policy='omit')<jupyter_output><empty_output><jupyter_text>4) t-statistic: 11.745) p-value of 9.087409645908879e-26---Conclusion: Due to a p-value of near-zero, I reject the null hypothesis that democrat support is zero and conclude that republican support is non-zero.<jupyter_code><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup