<jupyter_start><jupyter_text>Titanic and Lusitania Data Mining<jupyter_code>import pandas as pd
import numpy as np
import re
import math
from scipy.stats import chi2_contingency
from collections import OrderedDict
from scipy.stats import norm

import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load the data into a Pandas DataFrame:<jupyter_code>titanic = pd.read_csv('../data/titanic.csv')
lusit = pd.read_csv('../data/lusitania.csv')<jupyter_output><empty_output><jupyter_text>Observations:<jupyter_code>print "Titanic dataset contains {} instances and {} features".format(titanic.shape[0], titanic.shape[1]) 
print "Titanic dataset contains {} instances and {} features".format(lusit.shape[0], lusit.shape[1])<jupyter_output>Titanic dataset contains 1961 instances and 23 features<jupyter_text>Missing Values:<jupyter_code>titanic.isnull().sum()
lusit.isnull().sum()
titanic.columns = [col.replace(".", "_") for col in titanic.columns]
lusit.columns = [col.lower().replace(" ", "_").replace("/", "_") for col in lusit.columns]<jupyter_output><empty_output><jupyter_text>Let's check the age column for any discrepancies then plot their distribution:<jupyter_code>titanic['age'].unique()
fig, ax = plt.subplots(figsize=(8,5))

ax = sns.distplot(titanic['age'].dropna(), fit=norm, kde=False)
ax.set_xlabel('Age (years)')
ax.set_ylabel('Probability density')
ax.set_title('Titanic age distribution and maximum likelihood gaussian distribution fit')

plt.xlim(0,80)
fig.tight_layout()

plt.show()
lusit['age'].unique()<jupyter_output><empty_output><jupyter_text>The lusitania age column is a mess and needs cleaning and processing:<jupyter_code>def clean_lusitAge(row):
    regex = r"(\d+)(\?)"
    
    if type(row) == str and "-" in row:
        row = round(float(row.split("-")[0])/12, 2)
    
    if type(row) == str:
        row = row.split(" ")[0]
        row = row.replace("Infant", "0.08").replace("_", "")
        row = re.sub(regex, r"\1", row)
        row = np.where(row == '?', 0, row)
        
    return row
lusit['age'] = lusit['age'].apply(clean_lusitAge)
lusit['age'] = lusit['age'].astype(float)
fig, ax = plt.subplots(figsize=(8,5))

ax = sns.distplot(lusit['age'].dropna(), fit=norm, kde=False)
ax.set_xlabel('Age (years)')
ax.set_ylabel('Probability density')
ax.set_title('Lusitania age distribution and maximum likelihood gaussian distribution fit')

plt.xlim(0,80)
fig.tight_layout()

plt.show()<jupyter_output><empty_output><jupyter_text>Data Imputation & Feature Engineering: Creating new variables<jupyter_code>print "{} missing age values in Titanic dataset".format(titanic['age'].isnull().sum())<jupyter_output>263 missing age values in Titanic dataset<jupyter_text>In the name column from Titanic, we have titles. I will extract the titles and map missing age values. Titles also may reflect social status and may help predict survival probability.<jupyter_code>def extract_titanicTitle(row):
    pattern = re.compile("(, .+?\.)")
    row = pattern.findall(row)
    row = row[0].replace(", ", "")        
    
    return row
titanic["title"] = titanic["name"].apply(extract_titanicTitle)
titanicAge_dict = titanic.groupby(['title'])['age'].mean().to_dict()
def fill_titanicAge(row):
    global titanicAge_dict
    age = row['age']
    title = row['title']
    if math.isnan(age):
        if title in titanicAge_dict.keys():
            return titanicAge_dict[title]
    else:
        return age
titanic['age'] = titanic.apply(fill_titanicAge, axis=1)
print "{} missing age values in Lusitania dataset".format(lusit['age'].isnull().sum())<jupyter_output>653 missing age values in Lusitania dataset<jupyter_text>I will use the title column here as well to impute the missing age values.<jupyter_code>def clean_lusitTitle(row):
    row = row.replace("Mr. (later Sir)", "Sir")
    row = row.replace("Master (?)", "Master")
    row = "".join(row.split(" "))   
    
    return row
lusit["title"] = lusit["title"].apply(clean_lusitTitle)
lusitAge_dict = lusit.groupby(['title'])['age'].mean().to_dict()
def fill_lusitAge(row):
    global lusitAge_dict
    age = row['age']
    title = row['title']
    if math.isnan(age) or age == 0.0:
        if title in lusitAge_dict.keys():
            return lusitAge_dict[title]
    else:
        return age
lusit["age"] = lusit.apply(fill_lusitAge, axis=1)
titanic['life'] = titanic['survived'].apply(lambda x: 'survived' if x == 1 else 'deceased')
def to_percent(y, position):
    # Ignore the passed in position. This has the effect of scaling the default
    # tick locations
    s = str(round(y/1309.0*100))

    # The percent symbol needs escaping in latex
    if plt.rcParams['text.usetex'] is True:
        return s + r'$\%$'
    else:
        return s + '%'

fig, ax = plt.subplots(figsize=(8,5))

# the histogram of the data
titanic.groupby(['life'])['age'].plot.hist(alpha=0.5, ax=ax, legend = True, bins=23)

formatter = FuncFormatter(to_percent)
plt.gca().yaxis.set_major_formatter(formatter)

ax.set_xlabel('Age Group')
ax.set_ylabel('Percent')
ax.set_title('Percent Survived vs Deceased by Age Group in Titanic Disasters')

# Tweak spacing to prevent clipping of ylabel
fig.tight_layout()
plt.grid(True)

plt.show()
def to_percent(y, position):
    # Ignore the passed in position. This has the effect of scaling the default
    # tick locations
    s = str(round(y/1961.0*100))

    # The percent symbol needs escaping in latex
    if plt.rcParams['text.usetex'] is True:
        return s + r'$\%$'
    else:
        return s + '%'

fig, ax = plt.subplots(figsize=(8,5))

# the histogram of the data
lusit.groupby(['fate'])['age'].plot.hist(alpha=0.5, ax=ax, legend = True, bins=23)

formatter = FuncFormatter(to_percent)
plt.gca().yaxis.set_major_formatter(formatter)

ax.set_xlabel('Age Group')
ax.set_ylabel('Percent')
ax.set_title('Percent Survived vs Deceased by Age Group in Lusitania Disasters')

# Tweak spacing to prevent clipping of ylabel
fig.tight_layout()
plt.grid(True)

plt.show()<jupyter_output><empty_output><jupyter_text>Distribution of ages in groups of Titanic passengers<jupyter_code>g = sns.FacetGrid(titanic, row='survived', col='pclass')
g.map(sns.distplot, "age")

plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Make age groups 0-12 child, 12-17 teenager, 17-60 adult, 60+ elder<jupyter_code>bins = [0, 12, 17, 60, np.inf]
labels = ['child', 'teenager', 'adult', 'elder']
titanic['age_group'] = pd.cut(titanic.age, bins, labels=labels)<jupyter_output><empty_output><jupyter_text>Annotated heatmap of survivors by groups of passengers:<jupyter_code>df = titanic.pivot_table(index='embarked', columns='age_group', values='survived', aggfunc=np.mean)
ax = sns.heatmap(df, annot=True, fmt=".1f")<jupyter_output><empty_output><jupyter_text>Annotated heatmap of Pearson correlation coefficients between features in Titanic<jupyter_code>ax = sns.heatmap(titanic.corr(), annot=True, fmt=".2f")<jupyter_output><empty_output><jupyter_text>Distribution of survivors by age and sex<jupyter_code>ax = sns.swarmplot(x="sex", y="age", hue="life", data=titanic)<jupyter_output><empty_output><jupyter_text>Group Statistics & Chi Square Test<jupyter_code>titanic['deceased'] = titanic['survived'].apply(lambda x: 1 if x == 0 else 0)
titanic['all'] = titanic['survived'] + titanic['deceased']
titanic['survival_rate'] = titanic['survived']/titanic['all']<jupyter_output><empty_output><jupyter_text>Survival Rate by SexFemales had a much higher survival rate than males. The p-value is 4.5899249369529454e-81, indicating a statistically significant difference at any reasonable significance level.<jupyter_code>table = pd.crosstab(titanic['sex'], titanic['life'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 4.58992493695e-81<jupyter_text>Survival Rate by Passenger ClassThere appears to be a strong relationship between class and survival rate: first-class had the highest survival rate, then second, then third.<jupyter_code>titanic.groupby(['pclass']).agg(OrderedDict([('deceased', 'sum'),
                                            ('survived', 'sum'),
                                            ('all', 'sum'),
                                            ('survival_rate', 'mean')]))
table = pd.crosstab(titanic['pclass'], titanic['life'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 1.72082595883e-28<jupyter_text>Survival Rate by Age GroupThere appears to be a strong relationship between age and survival rate: The p-value is 4.24066535673e-06, indicating a statistically significant difference at any reasonable significance level.<jupyter_code>bins = np.linspace(titanic.age.min(), titanic.age.max(), 10)
table = pd.crosstab(np.digitize(titanic.age, bins), titanic['life'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 4.24066535673e-06<jupyter_text>Survival Rate by Class and Sex<jupyter_code>titanic.groupby(['pclass','sex']).agg(OrderedDict([('deceased', 'sum'),
                                                    ('survived', 'sum'),
                                                    ('all', 'sum'),
                                                    ('survival_rate', 'mean')]))<jupyter_output><empty_output><jupyter_text>Comparative Statistics:  Lusitania<jupyter_code>lusit['lost'] = lusit['fate'].apply(lambda x: 1 if x == 'Lost' else 0)
lusit['not_on_board'] = lusit['fate'].apply(lambda x: 1 if x == 'Not on board' else np.nan)
lusit['saved'] = lusit['fate'].apply(lambda x: 1 if x == 'Saved' else 0)
lusit['died_from_trauma'] = lusit['fate'].apply(lambda x: 1 if x == 'Saved (died from trauma)' else np.nan)
lusit['all'] = lusit['value']
lusit['survival_rate'] = (lusit['all'] - lusit['lost']) / lusit['all']<jupyter_output><empty_output><jupyter_text>Survival Rate by SexThe p-value is greater than 0.05, indicating a statistically insignificant difference between survival rate of a female and male.<jupyter_code>table = pd.crosstab(lusit['sex'], lusit['saved'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 0.291182076147<jupyter_text>Survival Rate by Passenger vs CrewThe p-value is greater than 0.05, indicating a statistically insignificant difference between survival rate of a passenger vs crew. Let's drop the stowaways since we have no information on them!<jupyter_code>lusit = lusit[lusit['passenger_crew'] != 'Stowaway']
table = pd.crosstab(lusit['passenger_crew'], lusit['saved'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 0.059455252032<jupyter_text>Survival Rate by ClassPassenger accommodation:* saloon (first class)* second cabin (second class)* third classCrew accommodation:* deck* victualling* engineering    The p-value is less than 0.05, indicating a statistically significant difference between survival rate of Class.<jupyter_code>lusit['department_class'] = lusit['department_class'].str.replace(r'Third \(Distressed British Seaman\)', 'Third')
lusit.groupby(['department_class']).agg(OrderedDict([('lost', 'sum'),
                                                    ('saved', 'sum'),
                                                    ('all', 'count'),
                                                    ('survival_rate', 'mean')]))
table = pd.crosstab(lusit['department_class'], lusit['saved'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 0.0189448858527<jupyter_text>Survival Rate by Age GroupThere appears to be a strong relationship between age and survival rate: The p-value is 3.83316420923e-06, indicating a statistically significant difference at any reasonable significance level.<jupyter_code>bins = np.linspace(lusit.age.min(), lusit.age.max(), 10)
table = pd.crosstab(np.digitize(lusit.age, bins), lusit['saved'])
table
chi, p, df, expected = chi2_contingency(table)
print "p-value:", p<jupyter_output>p-value: 3.83316420923e-06
StarcoderdataJupyterStructuredCleanDedup