<jupyter_start><jupyter_text>Starbucks Capstone Challenge IntroductionThis data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. Not all users receive the same offer, and that is the challenge to solve with this data set.Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer. ExampleTo give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer. CleaningThis makes data cleaning especially important and tricky.You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers. Final AdviceBecause this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).  Data SetsThe data is contained in three files:* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)* profile.json - demographic data for each customer* transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:**portfolio.json*** id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)**profile.json*** age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income**transcript.json*** event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:Then you will want to run the above command:Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.  Problem Statement In this project I will determine how likely is a customer to complete an offer. The end goal is:1. To determine, does sending more offers lead to a higher completion rate.2. Customers with lower completion rate should be sent offers or not  Exploratory Data Analysis  Read Data Files<jupyter_code>#######Run This
import pandas as pd
import numpy as np
import math
import json
import os
%matplotlib inline

portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>Save The Data Files<jupyter_code>if not os.path.isdir('explore'):
    os.makedirs('explore')
    
def data_info(data, filename):
    path = os.path.join('explore', filename)
    if not os.path.isfile(path):
        pd.DataFrame(data).to_csv(path)
    print(data.shape)

data_info(portfolio, 'portfolio.csv')
data_info(profile, 'profile.csv')
data_info(transcript, 'transcript.csv')<jupyter_output>(10, 6)
(17000, 5)
(306534, 4)<jupyter_text>Clean PortfolioBy looking at the portfolio file we can see that the channels column in grouped, so we'll use sklearn's MultiLabelBinarizer to unpack the channel column and then remove it from the DataFrame<jupyter_code>from sklearn.preprocessing import MultiLabelBinarizer

cleaned_portfolio = portfolio.copy()
cleaned_portfolio.rename(columns={'id':'offer_id'}, inplace=True)

s = cleaned_portfolio['channels']
mlb = MultiLabelBinarizer()
channels = pd.DataFrame(mlb.fit_transform(s),columns=mlb.classes_, index=cleaned_portfolio.index)

cleaned_portfolio = cleaned_portfolio.join(channels)
cleaned_portfolio.drop(['channels'], axis=1, inplace=True)
cleaned_portfolio<jupyter_output><empty_output><jupyter_text>Clean ProfileBy looking at the profile data we can see that there are missing age values, we also observe that the people with missing age values have missing income values. Therefore for now we'll remove all the rows with NaN values(~2000), deduce inference and then later combine with missing values and compare results<jupyter_code>#profile['became_member_on'] = pd.to_datetime(profile['became_member_on'], format='%Y%m%d')
profile.rename(columns={"id":"person"}, inplace=True)
undefined_group = None
cleaned_profile = None

#cleaning profile and dividing it into cleaned_profile and undefined_group
undefined_group = profile.copy()
undefined_group['gender'] = undefined_group['gender'].fillna('U')
undefined_group = undefined_group.loc[undefined_group['gender'] == 'U'].reset_index(drop=True)
cleaned_profile = profile.dropna().reset_index(drop=True)
cleaned_profile<jupyter_output><empty_output><jupyter_text>Clean TranscriptFrom the transcript we can see that the column values has 2 values i.e it has an offer id or amount spent for that transaction, we'll split the value in 2 columns offer_id, amount and then drop the value column<jupyter_code>cleaned_transcript = transcript.copy()
value = cleaned_transcript['value']
cleaned_transcript['amount'] = [int(i['amount']) if i.get('amount') else 0 for i in value]

cleaned_transcript['offer_id'] = [i['offer_id'] if i.get('offer_id') else (i['offer id'] if i.get('offer id') else '0') for i in value]

cleaned_transcript.drop(['value'], axis=1, inplace=True)

#drop the profile which have no gender or income
cleaned_transcript = cleaned_transcript[~cleaned_transcript.person.isin(undefined_group.person)]

sort_df = cleaned_transcript.sort_values(by=['person', 'time'])
sort_df<jupyter_output><empty_output><jupyter_text>Get Datamethod: get_valid_dataparams: df {df is the set of all the events for a person, lets say offer received, viewed or completed}---- The idea is to parse a set of transaction entries for a person and then divide it into offer received, offer viewed and offer completed---- Then create a new column ['g'] which stores the cumalative count for every entry lets say offer id 'a' offered twice then the corresponding g column will store the count something like this:    offer_id   g        a      0        a      1The idea behind g is that it will help us merge elements on [person, offer_id] and will prevent duplicates<jupyter_code>def get_valid_data(df):
        
    offer_received = df.loc[df['event'] == 'offer received'].reset_index(drop=True)
    offer_viewed = df.loc[df['event'] == 'offer viewed'].reset_index(drop=True)
    offer_completed = df.loc[df['event'] == 'offer completed'].reset_index(drop=True)


    offer_received['g'] = offer_received.groupby('offer_id').cumcount()
    offer_viewed['g'] = offer_viewed.groupby('offer_id').cumcount()
    offer_completed['g'] = offer_completed.groupby('offer_id').cumcount()
        

    res = pd.merge(offer_received, offer_viewed, on=['person', 'offer_id', 'g'], how='outer')
    res = pd.merge(res, offer_completed, on=['person', 'offer_id', 'g'], how='outer')

        
    return res

offers_completed = sort_df.groupby('person').apply(lambda x: get_valid_data(x))
offers_completed = offers_completed.dropna()
offers_completed = offers_completed.reset_index(drop=True)
offers_completed<jupyter_output><empty_output><jupyter_text>Combine Portfolio with the offers completed for every entry method: valid_offer_completedparameter: df {offers completed}, cleaned_portfolio {cleaned_portfolio- information about every customer like age, income} Functions1. Drop columns like amount_x, amount_y since they only have value 0 and theh drop event like offer received etc2. Merge cleaned_portfolio[offer_type, duration] to df on offer_id 3. Drop the columns where a user have completed and offer before and viewed it later i.e keep only those where time_y <= time<jupyter_code>def valid_offer_completed(df, cleaned_portfolio):
    df = df.rename(columns={"offer_id_x":"offer_id"})
    offers = cleaned_portfolio[['offer_id', 'offer_type', 'duration']]
    df = df.merge(offers,how='left', on='offer_id')
    
    
    df = df.drop(['amount_x', 'amount_y', 'amount', 'event_x', 'event_y', 'event', 'g'], axis=1).reset_index(drop=True)
    df = df[['person','offer_id','time_x','time_y', 'time', 'offer_type', 'duration']]
    
    df = df[(df.time_x <= df.time_y) & (df.time_y <= df.time)]
    return df

valid = valid_offer_completed(offers_completed, cleaned_portfolio)
valid = valid.reset_index(drop=True)
valid<jupyter_output><empty_output><jupyter_text>Find Informational OffersInformational offers do not have any offer completed record so we need to find the offer_completed time because we need to combine then with the valid dataframe later onso we'll caluate the offfer completed based on the duration of the information offer<jupyter_code>def info_offer(df):

    offer_received = df.loc[df['event'] == 'offer received'].reset_index(drop=True)
    offer_viewed = df.loc[df['event'] == 'offer viewed'].reset_index(drop=True)


    offer_received['g'] = offer_received.groupby('offer_id').cumcount()
    offer_viewed['g'] = offer_viewed.groupby('offer_id').cumcount()
        

    res = pd.merge(offer_received, offer_viewed, on=['person', 'offer_id', 'g'], how='outer')
    
    offers = cleaned_portfolio[['offer_id', 'offer_type', 'duration']]
    res = res.merge(offers,how='left', on='offer_id')
    
    res['time'] = res['time_x'] + res['duration'] * 24
    res = res.dropna()
    
    res = res[res.time_x <= res.time_y]
    res['response'] = np.where(res.time_y > res.time , 0, 1)
    res = res.loc[res.response == 1]
    
    
    res = res.drop(['response', 'amount_x', 'amount_y', 'event_x', 'event_y', 'g'], axis=1).reset_index(drop=True)
    res = res[['person','offer_id','time_x','time_y', 'time', 'offer_type', 'duration']]
    
        
    
    return res

info_df = sort_df[sort_df['offer_id'].isin(['3f207df678b143eea3cee63160fa8bed', '5a8bc65990b245e5a138643cd4eb9837'])]
info_data = info_df.groupby('person').apply(lambda x: info_offer(x))
info_data =info_data.reset_index(drop=True)
info_data<jupyter_output><empty_output><jupyter_text>Combine the valid and information dataframes<jupyter_code>complete = pd.concat([valid, info_data], ignore_index=True, sort=False)
complete<jupyter_output><empty_output><jupyter_text>Fill Profile method: fill_profileparams: gd {Grouped data is the grouped data which includes all the transction record per person}, df {df is the customer portfolio}1. Find the number of valid offers completed2. Append the total offers completed for every person in th customer portfolio<jupyter_code>df = None
def fill_profile(gd, df):
    grouped_data = gd.groupby(['person'])
    
    invalid = []
    for index, row in df.iterrows():
        if row['person'] in grouped_data.groups.keys():
            offers = grouped_data.get_group(row['person'])['offer_type'].value_counts()
            df.at[index, 'offers completed'] = offers.sum()
    
            for offer, count in offers.items():
                df.at[index, offer] = count
                
        else:
            invalid.append(row['person'])
    print(len(invalid))
    df = df.fillna(0)
    return df
df = fill_profile(complete, cleaned_profile)
df<jupyter_output>2133<jupyter_text>Find Datamethod: find_dataparameters: gd {gd is the grouped data which includes all the transction record per person}, df {df is the customer portfolio}1. Find the total number of offers received for every customer from the original transcript not the updated one2. Calculate the completion rate3. Append the new details in the customer portfiolio dataframe for each user<jupyter_code>def find_data(gd, df):
    gd = gd[(gd.event == 'offer received')].reset_index(drop=True)
    grouped_data = gd.groupby(['person'])

    
    for index, row in df.iterrows():
        if row['person'] in grouped_data.groups.keys():
            events = grouped_data.get_group(row['person'])['event'].count()
            df.at[index, 'offers received'] = events
            df.at[index, 'completion rate'] = row['offers completed'] * 100 / events
            
    return df
df = find_data(sort_df, df)
df<jupyter_output><empty_output><jupyter_text>Find amount1. Find the total amount spent by each user<jupyter_code>def find_amount(df):
    amount = pd.DataFrame()
    values = df.groupby(['person']).sum()
    amount['person'] = values.index
    amount['total amount'] = values.amount.to_numpy()
    return amount
    

total_amount = find_amount(cleaned_transcript)
df = df.merge(total_amount, on='person')
df = df.reset_index(drop=True)

########### Convert gender to M-0/F-1/O-2
df['gender'] = df['gender'].map({'M': 0, 'F': 1, 'O': 2})
df = df.fillna(0)
df
data_info(df, 'complete_profile_with_missing_values.csv')
data_info(complete, 'transcript_with_missing_values.csv')<jupyter_output>(14825, 12)
(30956, 7)<jupyter_text>Data Visualization  Visualising the Data in 1D Space<jupyter_code>import matplotlib.pyplot as plt
import matplotlib

complete.hist(bins=15, color='steelblue', edgecolor='black', linewidth=1.0,
           xlabelsize=8, ylabelsize=8, grid=False)    
plt.tight_layout(rect=(0, 0, 1.5, 1.5))   
df.hist(bins=15, color='steelblue', edgecolor='black', linewidth=2.0,
           xlabelsize=20, ylabelsize=20, grid=False)    
plt.tight_layout(rect=(0, 0, 5.8, 10))
import seaborn as sns

f, ax = plt.subplots(figsize=(12, 8))
corr = df.corr()
hm = sns.heatmap(round(corr,2), annot=True, ax=ax, cmap="coolwarm",fmt='.2f',
                 linewidths=.05)
f.subplots_adjust(top=0.93)
t= f.suptitle('Profile Attributes Correlation Heatmap', fontsize=14)
plt.figure(figsize=(15,4))
plt.plot(df['completion rate'].value_counts().sort_index())<jupyter_output><empty_output><jupyter_text>Unsupervised LearningWe will use 2 unsuoervised learning algorithms to check if our data is actually seprable in approximately 5 clustersThe reason being if we get good number of clusters(4 or 5) then we can go and label all the data points according to our logic for likeliness (will be discussed later)<jupyter_code>df.index = df['person']
df = df.drop(['person'], axis = 1)<jupyter_output><empty_output><jupyter_text>Normalizing the data Using sklearn's Min Max Scaler we will normalize the data in the range of 0 to 1 so that it becomes easier to work with supervised or unsupervised algorithms<jupyter_code>from sklearn.preprocessing import MinMaxScaler

def normalize_data(df):
    scaler = MinMaxScaler()
    df_scaled = pd.DataFrame(scaler.fit_transform(df.astype(float)))
    df_scaled.columns = df.columns
    df_scaled.index = df.index
    
    return df_scaled

df_scaled = normalize_data(df)
df_scaled<jupyter_output><empty_output><jupyter_text>Agglomerative Clustering1. Plot the dendogram2. Based on the dendogram detemine the distance threshold3. Use the distance threshold to find the number of clusters4. Check the distribution of clusters<jupyter_code>import matplotlib.pyplot as plt
from hcluster import pdist, linkage, dendrogram

X = df_scaled.T.values #Transpose values 
Y = pdist(X)
Z = linkage(Y)
plt.subplots(figsize=(18,5))
dendrogram(Z, labels = df_scaled.columns)<jupyter_output><empty_output><jupyter_text>from the dendogram gram graph we cab determine the distance threshold i.e. the line from where we can cut the graph is about 40 on the y axis<jupyter_code>from sklearn.cluster import AgglomerativeClustering

cluster = AgglomerativeClustering(n_clusters=None, affinity='euclidean', linkage='ward', distance_threshold=40)
agg_clusters = np.array(cluster.fit_predict(df_scaled))
unique, counts = np.unique(agg_clusters, return_counts=True)
dict(zip(unique, counts))<jupyter_output><empty_output><jupyter_text>K-means Clustering 1. Apply kmeans and find out the optimal number of clusters using the elbow method 2. Analyse the no of clusters formed and select the one where the clusters are equally distributed<jupyter_code>from sklearn.cluster import KMeans
from sklearn import metrics
from scipy.spatial.distance import cdist
import numpy as np
import matplotlib.pyplot as plt


# create new plot and data
plt.plot()
X = df_scaled

# k means determine k
distortions = []
for k in range(1,10):
    km = KMeans(n_clusters=k, n_init=30)
    km.fit(X)
    wcss = km.inertia_
    km_clusters = km.predict(X)
    unique, counts = np.unique(km_clusters, return_counts=True)
    print("Cluster ", k, dict(zip(unique, counts)))
    distortions.append(wcss)

# Plot the elbow
plt.plot(range(1,10), distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method showing the optimal k')
plt.show()<jupyter_output>Cluster  1 {0: 14825}
Cluster  2 {0: 7533, 1: 7292}
Cluster  3 {0: 3638, 1: 5816, 2: 5371}
Cluster  4 {0: 3948, 1: 3327, 2: 4534, 3: 3016}
Cluster  5 {0: 3125, 1: 3102, 2: 1708, 3: 2957, 4: 3933}
Cluster  6 {0: 2666, 1: 1921, 2: 2832, 3: 3749, 4: 1633, 5: 2024}
Cluster  7 {0: 1967, 1: 2151, 2: 2936, 3: 1338, 4: 1903, 5: 1855, 6: 2675}
Cluster  8 {0: 1661, 1: 2134, 2: 1250, 3: 1389, 4: 1871, 5: 2934, 6: 1731, 7: 1855}
Cluster  9 {0: 1569, 1: 1740, 2: 2703, 3: 1220, 4: 1171, 5: 1605, 6: 1388, 7: 1785, 8: 1644}<jupyter_text>from the k-means clustering we can see that for cluster 4 and cluster 5, and also from the elbow graph we can see that around 5 clusters are suitable<jupyter_code>from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=5)
kmeans.fit(df_scaled)

labels = kmeans.predict(df_scaled)
centroids = kmeans.cluster_centers_

unique, counts = np.unique(labels, return_counts=True)
print("Cluster 5",dict(zip(unique, counts)))<jupyter_output>Cluster 5 {0: 1711, 1: 3937, 2: 3105, 3: 3117, 4: 2955}<jupyter_text>Unsupervised Learning Algorithm ResultsAgglomerative clustering gives really bad results because of the variability in our dataset, whereas k-means gives an average result. We can also see that the elbow graph is not well formed but we do get an idea of the separablity in our dataset. Thereforw we will now use Supervised Learning Algorithms to properly label our data  Supervised Learning for Multi-Label Classification  Determining the likelinesslogic of determining the likeliness is column*weight,I am determining the likeliness using offer completion rate, and the the offer types i.e. bogo, informational and discount. We assign the completion rate a weightage of 3 and offer type a weightage of 1. We noramlize the dataframe so that all the values in the 3 offer type columns are in the range and therefore the logic begind weightage can be applied. total weights = 3(completion rate) + 1(offer types) = 4score  = { (bogo + informational + discount)/3 + completion_rate*3} / total_weightLabel 4 - Very Likely (score>= 80)Label 3 - Likely (score>= 60)Label 2 - Neutral(50% chance) (score>= 40)Label 1 - Unlikely(score>= 20)Label 0 - Very Unlikely (score < 20)<jupyter_code>def calculate_likeliness(rate):
    if rate >= 80 and rate <= 100:
        return 4
    elif rate >= 60 and rate < 80:
        return 3
    elif rate >= 40 and rate < 60:
        return 2
    elif rate >= 20 and rate < 40:
        return 1
    else:
        return 0


def likelihood(row):
    completion_rate= row[9]
    discount = row[7]
    informational = row[6]
    bogo = row[5]
        
    
    
    rate = ((discount + informational + bogo)/3 + completion_rate*3)/4
    
    return calculate_likeliness(rate * 100)
    

df_scaled.apply(lambda x: likelihood(x), axis=1)
df_scaled['likeliness'] = df_scaled.apply(lambda x: likelihood(x), axis=1)
df_scaled

## we can see that the data is well distributed 
df_scaled.likeliness.value_counts()
# drop the columns used for determing the likeliness, so that our supervised learning model is not able to cheat
df_scaled = df_scaled.drop(['bogo', 'informational', 'discount', 'completion rate'], axis=1)
df_scaled
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

mat = df_scaled.values
mat = df_scaled.values

X = mat[:,0:7]
Y = mat[:,7]

seed = 7
test_size = 0.33
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=test_size, random_state=seed)
def find_accuracy(y_test, y_pred):
    accuracy = accuracy_score(y_test, y_pred)
    print("Accuracy: %.2f%%" % (accuracy * 100.0))
from sklearn.svm import LinearSVC
clf = LinearSVC(random_state=0, tol=1e-5)
clf.fit(X_train, y_train)
# print(clf.coef_)
# print(clf.intercept_)
y_pred = clf.predict(X_test)
find_accuracy(y_test, y_pred)
from sklearn.svm import SVC
svclassifier = SVC(kernel='rbf')
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
find_accuracy(y_test, y_pred)
from sklearn.svm import SVC
svclassifier = SVC(kernel='poly')
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
find_accuracy(y_test, y_pred)
from sklearn.svm import SVC
svclassifier = SVC(kernel='linear')
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
find_accuracy(y_test, y_pred)
from sklearn.ensemble import RandomForestClassifier
rf_classifier = RandomForestClassifier()
rf_classifier.fit(X_train, y_train)
rf_pred = rf_classifier.predict(X_test)
find_accuracy(y_test, rf_pred)<jupyter_output>Accuracy: 98.75%<jupyter_text>Benchmark ModelXgboost Algorithm is our benchmark model because it performs tasks like multilabel classifcation with ease and with very High accuracy<jupyter_code>from xgboost import XGBClassifier
model = XGBClassifier()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
predictions = [round(value) for value in y_pred]
find_accuracy(y_test, y_pred)<jupyter_output>Accuracy: 100.00%<jupyter_text>Evaluating the Model  Now we will evaluate our random forest model<jupyter_code>from sklearn import model_selection
from sklearn.metrics import mean_absolute_error, mean_squared_error
from math import sqrt

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, rf_pred))
print(classification_report(y_test, rf_pred))

kfold = model_selection.KFold(n_splits=10, random_state=seed)
scoring = 'accuracy'
results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print('Accuracy -val set: %.2f%% (%.2f)' % (results.mean()*100, results.std()))

print("MAE test score:", mean_absolute_error(y_test, rf_pred))
print("RMSE test score:", sqrt(mean_squared_error(y_test, rf_pred)))<jupyter_output>[[1164    0    0    0    0]
 [   0 1213    1    0    0]
 [   0    0 1304    1    0]
 [   0    4   25  713    2]
 [   0    0    3   25  438]]
              precision    recall  f1-score   support

         0.0       1.00      1.00      1.00      1164
         1.0       1.00      1.00      1.00      1214
         2.0       0.98      1.00      0.99      1305
         3.0       0.96      0.96      0.96       744
         4.0       1.00      0.94      0.97       466

    accuracy                           0.99      4893
   macro avg       0.99      0.98      0.98      4893
weighted avg       0.99      0.99      0.99      4893

Accuracy -val set: 100.00% (0.00)
MAE test score: 0.01389740445534437
RMSE test score: 0.12945514583973036<jupyter_text>We can observe that our model performs really well and it has accuracy nearly as our benchmark model with really low MAE and RMSE Scores  Now solving the same problem by filling the missing values<jupyter_code>undefined = None
income_max = cleaned_profile.income.describe().max()
undefined_group = undefined_group.fillna(income_max)
# undefined = pd.concat([undefined_group, cleaned_profile], ignore_index=True)

# undefined
complete_transcript = transcript.copy()
value = complete_transcript['value']
complete_transcript['amount'] = [int(i['amount']) if i.get('amount') else 0 for i in value]

complete_transcript['offer_id'] = [i['offer_id'] if i.get('offer_id') else (i['offer id'] if i.get('offer id') else '0') for i in value]

complete_transcript.drop(['value'], axis=1, inplace=True)

sort_df = complete_transcript.sort_values(by=['person', 'time'])
sort_df
users  = sort_df.groupby('person').apply(lambda x: get_valid_data(x))
users = users.dropna() 
users = users.reset_index(drop=True)
users
valid_df = valid_offer_completed(users, cleaned_portfolio)
valid_df
complete_info = sort_df[sort_df['offer_id'].isin(['3f207df678b143eea3cee63160fa8bed', '5a8bc65990b245e5a138643cd4eb9837'])]
complete_info_data = complete_info.groupby('person').apply(lambda x: info_offer(x))
complete_info_data =complete_info_data.reset_index(drop=True)
complete_info_data
complete_df = pd.concat([valid_df, complete_info_data], ignore_index=True)
complete_df
full_profile = profile.copy()
full_profile['gender'] = full_profile['gender'].fillna('U')
full_profile['income'] = full_profile['income'].fillna(income_max)
full_profile
df2 = fill_profile(complete_df, full_profile)
df2 = find_data(sort_df, df2)
df2 = df2.fillna(0)
df2
total_amount = find_amount(complete_transcript)
df2 = df2.merge(total_amount, on='person')
df2 = df2.reset_index(drop=True)

df2.index = df2['person']
df2 = df2.drop(['person'], axis = 1)
########### Convert gender to 0/1/2
df2['gender'] = df2['gender'].map({'M': 0, 'F': 1, 'O': 2, 'U':3})
df2_scaled = normalize_data(df2)
df2_scaled
data_info(df2, 'complete_profile.csv')
data_info(complete_df, 'complete_transcript.csv')
df2_scaled.apply(lambda x: likelihood(x), axis=1)
df2_scaled['likeliness'] = df2_scaled.apply(lambda x: likelihood(x), axis=1)
df2_scaled
df2_scaled = df2_scaled.drop(['bogo', 'informational', 'discount', 'completion rate'], axis=1)
df2_scaled
mat2 = df2_scaled.values

X2 = mat2[:,0:7]
Y2 = mat2[:,7]

seed = 7
test_size = 0.20
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, Y2, test_size=test_size, random_state=seed)
##Random Forest
rf2_classifier = RandomForestClassifier()
rf2_classifier.fit(X2_train, y2_train)
rf2_pred = rf2_classifier.predict(X2_test)
find_accuracy(y2_test, rf2_pred)
## XGBoost Algorithm
model2 = XGBClassifier()
model2.fit(X2_train, y2_train)

y2_pred = model2.predict(X2_test)
predictions2 = [round(value) for value in y2_pred]

find_accuracy(y2_test, predictions2)<jupyter_output>Accuracy: 100.00%<jupyter_text>Conculsion In the project I have tried to determine how likely is a user complete an offer. I have used some data visualition to explain some realtion in the data. Then I have used unsupervised learning techniques to determine how seprable the data is and if we are actually able to divide the data in like 5 clusters. Then I determined the likeliness of every data point and removed the columns used to calculate the likeliness. This means that our supervised leaning model will not be able to deduce any inference in determining the likeliness. Then I split the data into training and test set and passed it to several SVM models with different kernel. One thing, I observed that tree models like Random Forest or XGBoost Algorithm perform really well for multi labeling tasks like this. Hence we will choose Gradient Boost as the algorithm of our choice and XGBoost as a benchmark model. Hence we will choose Gradient Boost as the algorithm of our choice and XGBoost as a benchmark model. For evaluating our model we look at the confusion matrix from where we can see High precision and High Recall which means that our results have been labeled correctly.Although we are having very good accuracy for our models but that does not mean that our model is perfect it simply means that our model has very less data for now and therefore classifying our data is an easy task, also because we do not have very high dimensional data. Multi-label Classification is easy for low dimensional data. Missing Values vs Non-Missing ValuesWhen we removed missing values from our data we found a good balance between various classes but when we added those missing values and performed some inference on that data we notice that our class imbalance increased. Otherwise, the performance is the same<jupyter_code>df2['offers received'].describe()
df2['total amount'].describe()
df['likeliness'] = df_scaled['likeliness']
def find_info(df2, class_label):
    label = df2[df2['likeliness'] == class_label]
    print("Likeliness ==", class_label)
    print("bogo", label.bogo.sum())
    print("discount", label.discount.sum())
    print("informational", label.informational.sum())
    print("offers received", label['offers received'].sum())

    print()

    
find_info(df, 4)
find_info(df, 3)
find_info(df, 2)
find_info(df, 1)
find_info(df, 0)<jupyter_output>Likeliness == 4
bogo 2244.0
discount 2331.0
informational 1433.0
offers received 6008.0

Likeliness == 3
bogo 2855.0
discount 2871.0
informational 1931.0
offers received 9628.0

Likeliness == 2
bogo 3674.0
discount 3998.0
informational 2625.0
offers received 17959.0

Likeliness == 1
bogo 1675.0
discount 2101.0
informational 1749.0
offers received 16269.0

Likeliness == 0
bogo 344.0
discount 511.0
informational 614.0
offers received 16637.0