<jupyter_start><jupyter_text>Capstone Project: Create a Customer Segmentation Report for Arvato Financial ServicesIn this project, you will analyze demographics data for customers of a mail-order sales company in Germany, comparing it against demographics information for the general population. You'll use unsupervised learning techniques to perform customer segmentation, identifying the parts of the population that best describe the core customer base of the company. Then, you'll apply what you've learned on a third dataset with demographics information for targets of a marketing campaign for the company, and use a model to predict which individuals are most likely to convert into becoming customers for the company. The data that you will use has been provided by our partners at Bertelsmann Arvato Analytics, and represents a real-life data science task.If you completed the first term of this program, you will be familiar with the first part of this project, from the unsupervised learning project. The versions of those two datasets used in this project will include many more features and has not been pre-cleaned. You are also free to choose whatever approach you'd like to analyzing the data rather than follow pre-determined steps. In your work on this project, make sure that you carefully document your steps and decisions, since your main deliverable for this project will be a blog post reporting your findings.<jupyter_code># import libraries here; add more as necessary
import numpy as np
import pandas as pd
from collections import Counter
from operator import itemgetter
import time
from matplotlib_venn import venn2
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.pipeline import Pipeline

# magic word for producing visualizations in notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Part 0: Get to Know the DataThere are four data files associated with this project:- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers ("CUSTOMERS") are similar to or differ from the general population at large ("AZDIAS"), then use your analysis to make predictions on the other two files ("MAILOUT"), predicting which recipients are most likely to become a customer for the mail-order company.The "CUSTOMERS" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original "MAILOUT" file included one additional column, "RESPONSE", which indicated whether or not each recipient became a customer of the company. For the "TRAIN" subset, this column has been retained, but in the "TEST" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order.In the below cell, we've provided some initial code to load in the first two datasets. Note for all of the `.csv` data files in this project that they're semicolon (`;`) delimited, so an additional argument in the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) call has been included to read in the data properly. Also, considering the size of the datasets, it may take some time for them to load completely.You'll notice when the data is loaded in that a warning message will immediately pop up. Before you really start digging into the modeling and analysis, you're going to need to perform some cleaning. Take some time to browse the structure of the data and look over the informational spreadsheets to understand the data values. Make some decisions on which features to keep, which features to drop, and if any revisions need to be made on data formats. It'll be a good idea to create a function with pre-processing steps, since you'll need to clean all of the datasets before you work with them.  Load DataSince the dataset is more than 1GB, I've used `c` engine instead of `python` to load data faster. In addition, there are mixed data in column 18 & 19; more specifically some `NaN` values are represented by `X` & `XX`. These values are set as `na_values`.<jupyter_code># Load data
def load_data(azdias_filepath, customers_filepath, attributes_filepath):
    """
    Method for loading dataset from CSV & Excel
    
    Args:
        azdias_filepath (str): Azdias Filepath
        customers_filepath (str): Customers Filepath
        attributes_filepath (str): Attributes Filepath
        
    Output:
        azdias: Pandas Dataframe
        customers: Pandas Dataframe
        attributes: Pandas Dataframe
    """
    
    # Load "azdias" dataset
    azdias = pd.read_csv(azdias_filepath, na_values=["X", "XX"], engine="c")
    
    # Load "customers" dataset
    customers = pd.read_csv(customers_filepath, na_values=["X", "XX"], engine="c")
    
    # Load "attributes" dataset
    attributes = pd.read_excel("DIAS Attributes - Values 2017.xlsx", header=1).loc[:, ["Attribute", "Value", "Meaning"]] \
                                .fillna(method='ffill')
    
    return azdias, customers, attributes
azdias_filepath = "Udacity_AZDIAS_052018.csv"
customers_filepath = "Udacity_CUSTOMERS_052018.csv"
attributes_filepath = "DIAS Attributes - Values 2017.xlsx"

azdias, customers, attributes = load_data(azdias_filepath, customers_filepath, attributes_filepath)<jupyter_output><empty_output><jupyter_text>Data Exploration (azdias)Explore `azdias` dataset to gain insights about the type of data, extent of data cleaning and feature engineering<jupyter_code>azdias.head()
attributes.head()
# Show column names and datatype
azdias.info(verbose=True)
# List of unique features in different dataset
attribute_list = list(attributes.Attribute.unique())
azdias_features = azdias.columns.tolist()

# Find common and unique features between 'azdias' and 'attributes'
common_features = set(azdias_features) & set(attribute_list)
unique_azdias_features = set(azdias_features) - set(attribute_list)
num_common_features = len(common_features)
num_unique_azdias_features = len(unique_azdias_features)
print("common features: {}, unique azdias features: {}".format(num_common_features, num_unique_azdias_features))<jupyter_output>common features: 272, unique azdias features: 94<jupyter_text>**Observations:** Since 94 features in `azdias` do not have any descriptions in `attributes`, these features will be dropped. In addition, some features in `attributes` have values assigned to _unknown_ or _missing information._ These values in `azdias` will be replaced with `NaN`.<jupyter_code># Drop features that are not present in "attributes"
azdias.drop(columns=list(unique_azdias_features), inplace=True)
azdias.shape
# Create a subset of "attributes" where only unknown values are present
attributes_unknown_val = attributes[(attributes['Meaning'].str.contains("unknown") | attributes['Meaning'].str.contains("no "))]

unknown_val = []
for attribute in attributes_unknown_val['Attribute'].unique():
    # Create a list of unknown value for a feature
    val = attributes_unknown_val.loc[attributes['Attribute'] == attribute, 'Value'].astype("str").str.cat(sep=',').split(',')
    # Convert the list to "int"
    val = list(map(int, val))
    unknown_val.append(val)
    
# Create dataframe of features with unknown value
attributes_unknown_val = pd.concat([pd.Series(attributes_unknown_val['Attribute'].unique()),
                                    pd.Series(unknown_val)], axis=1)

# Rename the columns
attributes_unknown_val.columns = ['attribute', 'unknown']

attributes_unknown_val.head()
# Replace the unknown values in "azdias" with NaN based on "attributes_unknown_val" 
for row in attributes_unknown_val.itertuples(index=False):
    if row.attribute in azdias.columns.values.tolist():
        print(row.attribute)
        nan_val = attributes_unknown_val.loc[attributes_unknown_val['attribute'] == row.attribute, 'unknown'].iloc[0]
        nan_idx = azdias.loc[:, row.attribute].isin(nan_val)
        azdias.loc[nan_idx, row.attribute] = np.NaN
    else:
        continue
        
azdias.head()
# Check the total number of NaN count on each column
nan_count_column_azdias = azdias.isnull().sum()

# Plot the distribution of missing or unknown data for each column
plt.figure(figsize=(15,5))
plt.title('Distribution of Missing Data in Each Column')
plt.hist(nan_count_column_azdias, bins=100)
plt.ylabel('Number of Columns')
plt.xlabel('Number of Missing Values')
plt.show()
# Drop the columns where the NaN count is higher than 20000 i.e. more than 20%
# Select the columns that are needed to be dropped
drop_columns = nan_count_column_azdias[nan_count_column_azdias > 200000]

# Make list of the drop_columns
drop_column_list = drop_columns.index.tolist()

# Drop the columns from azdias
azdias.drop(columns=drop_column_list, inplace=True)

azdias.shape
azdias.head()
# Similar to columns, check the total NaN count for each row
count_nan_row = azdias.isnull().sum(axis=1)
plt.figure(figsize=(15,5))
plt.title('Distribution of Missing Data in Each Row')
plt.hist(count_nan_row, bins=50)
plt.ylabel('Number of Rows')
plt.xlabel('Number of Missing Values')
plt.show()
# # Drop the rows where the NaN count is higher than 50 i.e. more than 20%
count_nan_row = azdias.shape[1] - azdias.count(axis=1)
drop_row = azdias.index[count_nan_row > 50]
azdias.drop(drop_row, axis=0, inplace=True)
print(azdias.shape)
azdias.head()<jupyter_output>(737288, 237)<jupyter_text>**Observations:** There are features in `azdias` where significant number of data are missing (more than 200,000). These features are dropped. Similarly, more than 50 datapoints are missing across rows which are dropped as well. Based on these observations, a data cleaning function was created to clean `customers` dataset. In the function, if more than 20% data are missing in a column or across a row, it will be dropped.  Create function for Cleaning the Dataset<jupyter_code>def clean_data(df, attributes, col_nan_threshold, row_nan_threshold):
    """
    Method for cleaning the dataset according to attributes
    
    Args:
        df (Pandas Dataframe): Dataset to clean
        attributes (Pandas Dataframe): Reference dataset for cleaning df
        col_nan_threshold (float): Threshold value (0-1) for missing NaN count in a column
        row_nan_threshold (float): Threshold value (0-1) for missing NaN count in a row
        
    Output:
        df (Pandas Dataframe): Cleaned dataset
    """
    
    # List of attributes
    attribute_list = list(attributes.Attribute.unique())
    
    # List of df attributes
    df_attributes = list(df.columns)
    
    # Find attributes that are unique in df
    unique_df_attributes = list(set(df_attributes) - set(attribute_list))    
    
    # Drop attributes from df that are not present in attribute
    df.drop(columns=unique_df_attributes, inplace=True)
    
    # Create a subset of attributes with unknown or no information value
    unknown_val = []
    attributes_unknown_val = attributes[(attributes['Meaning'].str.contains("unknown") | 
                                         attributes['Meaning'].str.contains("no "))]
    for attribute in attributes_unknown_val['Attribute'].unique():
        val = attributes_unknown_val.loc[attributes['Attribute'] == attribute, 'Value'].astype("str") \
                                                                            .str.cat(sep=',').split(',')
        val = list(map(int, val))
        unknown_val.append(val)
    attributes_unknown_val = pd.concat([pd.Series(attributes_unknown_val['Attribute'].unique()),
                                        pd.Series(unknown_val)], axis=1)
    attributes_unknown_val.columns = ['attribute', 'unknown']
    
    # Replace unknown or missing values in df with NaN   
    for row in attributes_unknown_val.itertuples(index=False):
        if row.attribute in df.columns.values.tolist():
            nan_val = attributes_unknown_val.loc[attributes_unknown_val['attribute'] == row.attribute, 'unknown'].iloc[0]
            nan_idx = df.loc[:, row.attribute].isin(nan_val)
            df.loc[nan_idx, row.attribute] = np.NaN
        else:
            continue
    
    # Drop columns where NaN count is above the threshold
    nan_count_column = (df.isnull().sum()/df.shape[0]).sort_values(ascending=False)
    # Select the columns that are needed to be dropped
    nan_count_column = nan_count_column[nan_count_column > col_nan_threshold]
    # Make list of the drop_columns
    drop_column_list = nan_count_column.index.tolist()
    # Drop the columns from azdias
    df.drop(columns=drop_column_list, inplace=True)
    
    # Dropping rows where NaN count is above the threshold
    number_col = df.shape[1]
    count_nan_row = (df.shape[1] - df.count(axis=1)) / number_col
    drop_row = df.index[count_nan_row > row_nan_threshold]
    df.drop(drop_row, axis=0, inplace=True)
    
    return df
customers = clean_data(customers, attributes, 0.2, 0.2)
customers.shape
customers.head(20)
azdias.head(20)<jupyter_output><empty_output><jupyter_text>**Observations:** `customers` has only 37 features after cleaning while `azdias` has 237 columns. Therefore, columns that are not in `customers` dataset will be dropped from `azdias`.<jupyter_code># Find unique "azdias" features to drop
customers_feature = customers.columns.tolist()
azdias_feature = azdias.columns.tolist()
unique_azdias_feature = set(azdias_feature) - set(customers_feature)
azdias.drop(columns=list(unique_azdias_feature), inplace=True)
azdias.info(verbose=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 737288 entries, 1 to 891220
Data columns (total 37 columns):
CJT_GESAMTTYP            732906 non-null float64
FINANZ_ANLEGER           737288 non-null float64
FINANZ_HAUSBAUER         737288 non-null float64
FINANZ_MINIMALIST        737288 non-null float64
FINANZ_SPARER            737288 non-null float64
FINANZ_UNAUFFAELLIGER    737288 non-null float64
FINANZ_VORSORGER         737288 non-null float64
FINANZTYP                737288 non-null float64
GEBURTSJAHR              737288 non-null int64
GFK_URLAUBERTYP          732906 non-null float64
GREEN_AVANTGARDE         737288 non-null int64
HH_EINKOMMEN_SCORE       737288 non-null float64
LP_FAMILIE_FEIN          732906 non-null float64
LP_FAMILIE_GROB          732906 non-null float64
LP_LEBENSPHASE_FEIN      732906 non-null float64
LP_LEBENSPHASE_GROB      732906 non-null float64
LP_STATUS_FEIN           732906 non-null float64
LP_STATUS_GROB           732906 non-null float64
ONLINE_AFFI[...]<jupyter_text>Feature Encoding and Engineering<jupyter_code># Drop "LP_LEBENSPHASE_GROB", "LP_STATUS_GROB" and "LP_FAMILIE_GROB" 
# There are other columns that already captures the information that are presented in these columns
azdias.drop(columns=['LP_LEBENSPHASE_GROB', "LP_STATUS_GROB", "LP_FAMILIE_GROB"], inplace=True)
customers.drop(columns=['LP_LEBENSPHASE_GROB', "LP_STATUS_GROB", "LP_FAMILIE_GROB"], inplace=True)
azdias.shape, customers.shape
# Drop "GEBURTSJAHR" since the column represents timeseries information which is not necessarily important for classification
# In "GEBURTSJAHR" missing data are represented as 0. Fill in the missing values at "GEBURTSJAHR" with NaN
#azdias["GEBURTSJAHR"].replace(0, np.nan, inplace=True)

# Create subset of "GEBURTSJAHR" for different imputation strategy
#azdias_year = azdias.loc[:, ["GEBURTSJAHR"]]
azdias.drop(columns=["GEBURTSJAHR"], inplace=True)

# Fill missing values (0) at "GEBURTSJAHR" with NaN
#customers["GEBURTSJAHR"].replace(0, np.nan, inplace=True)

# Create subset of "GEBURTSJAHR" for different imputation strategy
#customers_year = customers.loc[:, ["GEBURTSJAHR"]]
customers.drop(columns=["GEBURTSJAHR"], inplace=True)
azdias.shape, customers.shape
# Impute numeric columns
imputer_numeric = SimpleImputer(missing_values=np.nan, strategy='median')
azdias = pd.DataFrame(imputer_numeric.fit_transform(azdias), columns = azdias.columns)
customers = pd.DataFrame(imputer_numeric.fit_transform(customers), columns = customers.columns)
# Since "GEBURTSJAHR" represents timeseries, instead of "median" "most_frequent" strategy is used to impute "GEBURTSJAHR"
# imputer_year = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
# azdias_year = pd.DataFrame(imputer_numeric.fit_transform(azdias_year), columns = azdias_year.columns)
# customers_year = pd.DataFrame(imputer_numeric.fit_transform(customers_year), columns = customers_year.columns)
# Standardize the numeric features in the dataset
scaler = StandardScaler()
azdias_scale = scaler.fit_transform(azdias)
customers_scale = scaler.transform(customers)
azdias = pd.DataFrame(azdias_scale, columns=azdias.columns)
customers = pd.DataFrame(customers_scale, columns=customers.columns)
azdias.head()
# Concat "GEBURTSJAHR" with the dataframe
#azdias = pd.concat([azdias, azdias_year], axis=1)
#customers = pd.concat([customers, customers_year], axis=1)<jupyter_output><empty_output><jupyter_text>Part 1: Customer Segmentation ReportThe main bulk of your analysis will come in this part of the project. Here, you should use unsupervised learning techniques to describe the relationship between the demographics of the company's existing customers and the general population of Germany. By the end of this part, you should be able to describe parts of the general population that are more likely to be part of the mail-order company's main customer base, and which parts of the general population are less so.<jupyter_code># Perform PCA on the dataset (taken from the classroom)
def pca_op(n_components, data):
    """
    Performs PCA to create n_components, and returns results of the transformation.
    """
    pca = PCA(n_components)
    X_pca = pca.fit_transform(data)
    return pca, X_pca
azdias_pca_all, azdias_X_pca_all = pca_op(None, azdias)
# Plot variance by each features (taken from the classroom)
def plot_variance_by_component(pca):
    plt.rcParams["figure.figsize"] = [15,5]
    plt.bar(range(len(pca.explained_variance_ratio_)), pca.explained_variance_ratio_)
    plt.title("Explained variance by component")
    plt.xlabel("Principal component")
    plt.ylabel("Explained variance ratio")
    plt.show()
# Investigate the variance accounted for by each principal component.
plot_variance_by_component(azdias_pca_all)
# Plot scree plot
def variance_cumulative(pca):
    plt.plot(range(len(pca.explained_variance_ratio_)),np.cumsum(pca.explained_variance_ratio_), '-')
    plt.title("Explained Variance Cumulative")
    plt.xlabel("Component number")
    plt.ylabel("Explained variance ratio")
    plt.show()
# Investigate the variance cumulative
variance_cumulative(azdias_pca_all)<jupyter_output><empty_output><jupyter_text>**Observations:** From the plot, it is clear that 15 components explain more than 90% variance.<jupyter_code># Re-run PCA to the data while selecting to retain 10 components.
azdias_pca_15, azdias_X_pca_15 = pca_op(15, azdias)
# Investigate the variance accounted for by each principal component when 10 components are used
plot_variance_by_component(azdias_pca_15)
# Investigate the variance cumulative when 10 components are used
variance_cumulative(azdias_pca_15)
# Get weights of each component (taken from classroom)
def get_weights(pca, component_num, data):
    df = pd.DataFrame(pca.components_, columns = list(azdias.columns))
    weights = df.iloc[component_num].sort_values(ascending=False)
    return weights
# Print relationship between azdias features and pca components
dimensions = len(azdias_pca_15.components_)
result = pd.DataFrame()
for dim in range(dimensions):
    df = get_weights(azdias_pca_15, dim, azdias)
    result = pd.concat([result, df], axis = 1)
result = result.T
result
# Plot weights for each component
def plot_weights_by_component(idx, pca_data):
    df = pd.DataFrame(pca_data.loc[idx].sort_values().reset_index())
    print(df)
    plt.rcParams["figure.figsize"] = [15,5]
    #plt.bar(result.loc[0].sort_values().index.tolist(), result.loc[0].sort_values().values.tolist())
    plt.bar(df.index.tolist(), result.loc[idx].sort_values().values.tolist())
    plt.title("Weight by feature by component")
    plt.xlabel("Feature #")
    plt.ylabel("Feature Weight")
    plt.show()
    
    plt.rcParams["figure.figsize"] = [15,5]
    plt.bar(result.loc[idx].sort_values().iloc[[0, 1, 2, -3, -2, -1]].index.tolist(), 
            result.loc[idx].sort_values().iloc[[0, 1, 2, -3, -2, -1]].values.tolist())
    plt.title("Weight by feature by component")
    plt.xlabel("Feature #")
    plt.ylabel("Feature Weight")
    plt.show()
for i in range(5):
    plot_weights_by_component(i, result)
# Over a number of different cluster counts run k-means clustering on the data and compute the average within-cluster distances
def elbow_plot(data, start_K, end_K, step):
    '''
    Generate an elbow plot to find optimal number of clusters
    graphing K values from start_K to end_K every step value
    '''
    score_list = []

    for i in range(start_K, end_K, step):
        print(i)
        start = time.time()
        kmeans = KMeans(i)
        model = kmeans.fit(data)
        score = model.score(data)
        score_list.append(abs(score))
        end = time.time()
        elapsed_time = end - start
        print(elapsed_time)
        print(score_list)

    plt.plot(range(start_K, end_K, step), 
    score_list, linestyle='--', marker='o', color='b');
    plt.xlabel('# of clusters K');
    plt.ylabel('Sum of squared errors');
    plt.title('SSE vs. K');
    plt.savefig('elbow_plot.png')
elbow_plot(azdias_X_pca_15, 1, 10, 1)
# Fit the k-means model with the selected number of clusters and obtain cluster predictions for the general population
# demographics data.
azdias_kmeans = KMeans(6)
azdias_model = azdias_kmeans.fit(azdias_X_pca_15)
azdias_labels = azdias_model.predict(azdias_X_pca_15)
# View first few labels predicted for the azdias dataset
azdias_labels[0:10]
# Perform PCA on customers dataset
customers_pca_15 = azdias_pca_15.transform(customers)
# predict and print first few labels for the customers dataset
customers_labels = azdias_model.predict(customers_pca_15)
customers_labels[0:10]<jupyter_output><empty_output><jupyter_text>Compare Customer with Demographics<jupyter_code>def calc_propotions(labels, population):
    counter = Counter(labels)
    proportions = [(i, counter[i] / population * 100.0) for i in counter]
    proportions.sort(key=itemgetter(0))
    proportions = np.array(proportions)
    
    dropped_gen_pop = 100 - round(proportions[:, 1].sum())
    if (dropped_gen_pop > 1):
        proportions = np.insert(proportions, 0, values=[dropped_gen_pop], axis=0)
    
    return proportions
customers.shape, azdias.shape
# for azdias calculated the proportions of each cluster
general_population = 737288
azdias_proportions = calc_propotions(azdias_labels, general_population)
print(azdias_proportions)
# for customers calculated the proportions of each cluster
general_population = 188439 #191652 #132907#
customers_proportions = calc_propotions(customers_labels, general_population)
print(customers_proportions)
# calculate the ration between general population (azdias) and customers
# to see which customer segments are under and over represented
representation_ratio = customers_proportions[0:, 1]/azdias_proportions[0:, 1]
representation_ratio
plt.figure(figsize=(15, 4))

ax1 = plt.subplot(131)
ax1.bar(azdias_proportions[:, 0], azdias_proportions[:, 1])
ax1.set_xticks(azdias_proportions[:, 0])
ax1.set_ylim(ymax = 35)
ax1.set_xlabel("Cluster")
ax1.set_ylabel("Proportion of Total (%)")
plt.title('General Population')

ax2 = plt.subplot(132)
ax2.bar(customers_proportions[:, 0], customers_proportions[:, 1])
ax2.set_xticks(azdias_proportions[:, 0])
ax2.set_ylim(ymax = 45)
ax2.set_xlabel("Cluster")
ax2.set_ylabel("Proportion of Total (%)")
plt.title('Customer Population')

ax3 = plt.subplot(133)
ax3.bar(azdias_proportions[:, 0], representation_ratio)
ax3.set_xticks(azdias_proportions[:, 0])
ax3.set_xlabel("Cluster")
ax3.set_ylabel("Representation Ratio")
ax3.axhline(y=1, linestyle = "--", linewidth = 0.8)
plt.title('Customer Population/General Population')
tt = azdias_pca_15.inverse_transform(azdias_model.cluster_centers_[1])
c_1 = tt
underpresented = pd.Series(data = c_1, index = customers.columns).sort_values()
underpresented
tt = azdias_pca_15.inverse_transform(azdias_model.cluster_centers_[4])
c_4 = tt
underpresented = pd.Series(data = c_4, index = customers.columns).sort_values()
underpresented
tt = azdias_pca_15.inverse_transform(azdias_model.cluster_centers_[5])
c_5 = tt
underpresented = pd.Series(data = c_5, index = customers.columns).sort_values()
underpresented<jupyter_output><empty_output><jupyter_text>**Observations:** While cluster 0, 2 and 3 are well represented in customers, cluster 1, 4 and 5 are underrepresented. These underrepresented clusters in general represents populations who are cultural minded, social and aware about the product.  Part 2: Supervised Learning ModelNow that you've found which parts of the population are more likely to be customers of the mail-order company, it's time to build a prediction model. Each of the rows in the "MAILOUT" data files represents an individual that was targeted for a mailout campaign. Ideally, we should be able to use the demographic information from each individual to decide whether or not it will be worth it to include that person in the campaign.The "MAILOUT" data has been split into two approximately equal parts, each with almost 43 000 data rows. In this part, you can verify your model with the "TRAIN" partition, which includes a column, "RESPONSE", that states whether or not a person became a customer of the company following the campaign. In the next part, you'll need to create predictions on the "TEST" partition, where the "RESPONSE" column has been withheld.<jupyter_code>mailout_train = pd.read_csv('Udacity_MAILOUT_052018_TRAIN.csv')
mailout_train.shape
# Count of response
vc = mailout_train['RESPONSE'].value_counts()
# Positive response
pos_res = vc[1]/(vc[0]+vc[1])
# Negative response
neg_res = 1 - pos_res
print("Ratio of postive response: {0: 0.3f} and negative response: {1: 0.3f}".format(pos_res, neg_res))
# Extract RESPONSE column
response = mailout_train['RESPONSE']
# Drop RESPONSE column
mailout_train.drop(labels=['RESPONSE'], axis=1, inplace=True)
mailout_train = clean_data(mailout_train, attributes, 0.2, 1)
mailout_train.shape
# Apply similar steps as part 0. Find unique features in mailout_train and drop them
unique_mailout_feature = set(mailout_train.columns.tolist()) - set(azdias.columns.tolist())
mailout_train.drop(columns=list(unique_mailout_feature), inplace=True)
mailout_train.info(verbose=True)
# Fill missing values (0) at "GEBURTSJAHR" with NaN
#mailout_train["GEBURTSJAHR"].replace(0, np.nan, inplace=True)

# Create subset of "GEBURTSJAHR" for different imputation strategy
#mailout_train_year = mailout_train.loc[:, ["GEBURTSJAHR"]]
#mailout_train.drop(columns=["GEBURTSJAHR"], inplace=True)
imputer_numeric = SimpleImputer(missing_values=np.nan, strategy='median')
mailout_train = pd.DataFrame(imputer_numeric.fit_transform(mailout_train), columns = mailout_train.columns)

scaler = StandardScaler()
mailout_train_scale = scaler.fit_transform(mailout_train)
mailout_train = pd.DataFrame(mailout_train_scale, columns=azdias.columns)
mailout_train.head()
def classify(clf, param_grid, X_train=mailout_train, y_train=response):
    """
    Fits a classifier to its training data and prints its ROC AUC score.
    
    INPUT:
    - clf (classifier): classifier to fit
    - param_grid (dict): classifier parameters used with GridSearchCV
    - X_train (DataFrame): training input
    - y_train (DataFrame): training output
            
    OUTPUT:
    - classifier: input classifier fitted to the training data
    """
    
    # cv uses StratifiedKFold
    # scoring roc_auc available as parameter
    grid = GridSearchCV(estimator=clf, param_grid=param_grid, scoring='roc_auc', cv=5)
    grid.fit(X_train, y_train)
    print(grid.best_score_)
    
    return grid.best_estimator_
# LogisticRegression
lor = LogisticRegression(random_state=0)
classify(lor, {})
# BaggingClassifier
bac = BaggingClassifier(random_state=0)
classify(bac, {})
# RandomForestClassifier
rfc = RandomForestClassifier(random_state=0)
classify(rfc, {})
# AdaBoostClassifier
abc = AdaBoostClassifier(random_state=0)
abc_best_est = classify(abc, {})
# GradientBoostingClassifier
gbc = GradientBoostingClassifier(random_state=0)
classify(gbc, {})<jupyter_output>0.5175244452709992<jupyter_text>Model Tuning<jupyter_code>"""
# tune the most promising classifier with the help of GridSearchCV
# the result is our model that will be used with the test set
lor = LogisticRegression(random_state=0)
param_grid = {"algorithm" :["SAMME", "SAMME.R"],
              'n_estimators':[80]
             }

abc_best_est = classify(abc, param_grid)
abc_best_est
"""
# LogisticRegression has the highest score. Select the logistic regression model
lor = LogisticRegression(random_state=0)
model = classify(lor, {})
# Find the feature importance
"""
fi = pd.DataFrame({'FI':abc_best_est.feature_importances_}, index=mailout_train.columns)
fi_sorted = fi.sort_values(by=['FI'], ascending=False)
fi_sorted.head(10)
"""
fi = pd.DataFrame({'FI':model.coef_[0]}, index=mailout_train.columns)
fi_sorted = fi.sort_values(by=['FI'], ascending=False)
fi_sorted.head(10)<jupyter_output><empty_output><jupyter_text>Part 3: Kaggle CompetitionNow that you've created a model to predict which individuals are most likely to respond to a mailout campaign, it's time to test that model in competition through Kaggle. If you click on the link [here](http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140), you'll be taken to the competition page where, if you have a Kaggle account, you can enter. If you're one of the top performers, you may have the chance to be contacted by a hiring manager from Arvato or Bertelsmann for an interview!Your entry to the competition should be a CSV file with two columns. The first column should be a copy of "LNR", which acts as an ID number for each individual in the "TEST" partition. The second column, "RESPONSE", should be some measure of how likely each individual became a customer – this might not be a straightforward probability. As you should have found in Part 2, there is a large output class imbalance, where most individuals did not respond to the mailout. Thus, predicting individual classes and using accuracy does not seem to be an appropriate performance evaluation method. Instead, the competition will be using AUC to evaluate performance. The exact values of the "RESPONSE" column do not matter as much: only that the higher values try to capture as many of the actual customers as possible, early in the ROC curve sweep.<jupyter_code>mailout_test = pd.read_csv("Udacity_MAILOUT_052018_TEST.csv")
mailout_test.head()
# Extract lnr for later generation of the competition result file
lnr = mailout_test.LNR
# Clean data
mailout_test = clean_data(mailout_test, attributes, 0.2, 1)
mailout_test.shape
# Keep the features that are common in azdias and mailout_test
unique_mailout_test_feature = set(mailout_test.columns.tolist()) - set(azdias.columns.tolist())
mailout_test.drop(columns=list(unique_mailout_test_feature), inplace=True)
mailout_test.shape
# Fill missing values (0) at "GEBURTSJAHR" with NaN
#mailout_test["GEBURTSJAHR"].replace(0, np.nan, inplace=True)

# Create subset of "GEBURTSJAHR" for different imputation strategy
#mailout_test_year = mailout_test.loc[:, ["GEBURTSJAHR"]]
#mailout_test.drop(columns=["GEBURTSJAHR"], inplace=True)
imputer_numeric = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
mailout_test = pd.DataFrame(imputer_numeric.fit_transform(mailout_test), columns = mailout_test.columns)
#mailout_test_year = pd.DataFrame(imputer_numeric.fit_transform(mailout_test_year), columns = mailout_test_year.columns)

scaler = StandardScaler()
mailout_test_scale = scaler.fit_transform(mailout_test)

mailout_test = pd.DataFrame(mailout_test_scale, columns=mailout_test.columns)
#mailout_test = pd.concat([mailout_test, mailout_test_year], axis=1)
mailout_test.head()
# Use the trained model from Part 2 to predict the probabilties of the testing data 
response_test = model.predict_proba(mailout_test)
response_test
# Generate result file for the competition
result = pd.DataFrame({'LNR':lnr, 'RESPONSE':response_test[:,0]})
result.to_csv("result.csv", index=False)
result.head(10)
result.describe()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup