# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Starbucks Capstone Challenge
#
# ### Introduction
#
# This data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. 
#
# Not all users receive the same offer, and that is the challenge to solve with this data set.
#
# Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.
#
# Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.
#
# You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. 
#
# Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer.
#
# ### Example
#
# To give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.
#
# However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer.
#
# ### Cleaning
#
# This makes data cleaning especially important and tricky.
#
# You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers.
#
# ### Final Advice
#
# Because this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).

# # Data Sets
#
# The data is contained in three files:
#
# * portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)
# * profile.json - demographic data for each customer
# * transcript.json - records for transactions, offers received, offers viewed, and offers completed
#
# Here is the schema and explanation of each variable in the files:
#
# **portfolio.json**
# * id (string) - offer id
# * offer_type (string) - type of offer ie BOGO, discount, informational
# * difficulty (int) - minimum required spend to complete an offer
# * reward (int) - reward given for completing an offer
# * duration (int) - time for offer to be open, in days
# * channels (list of strings)
#
# **profile.json**
# * age (int) - age of the customer 
# * became_member_on (int) - date when customer created an app account
# * gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)
# * id (str) - customer id
# * income (float) - customer's income
#
# **transcript.json**
# * event (str) - record description (ie transaction, offer received, offer viewed, etc.)
# * person (str) - customer id
# * time (int) - time in hours since start of test. The data begins at time t=0
# * value - (dict of strings) - either an offer id or transaction amount depending on the record
#
# **Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  
#
# You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:
#
# <img src="pic1.png"/>
#
# Then you will want to run the above command:
#
# <img src="pic2.png"/>
#
# Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.

# +
import pandas as pd
import numpy as np
import math
import json
% matplotlib inline

# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)
# -

# ### Reading The Datasets

portfolio.head(10)

portfolio.shape[0]

portfolio.shape[1]

print('portfolio: rows = {} ,columns = {}'.format((portfolio.shape[0]),(portfolio.shape[1])))

portfolio.describe()

portfolio.info()

portfolio.offer_type.value_counts()

portfolio.reward.value_counts()

# +
import matplotlib.pyplot as plt
plt.figure(figsize=[6,6])
fig, ax = plt.subplots()    
y_counts = portfolio['offer_type'].value_counts()
y_counts.plot(kind='barh').invert_yaxis()
       
for i, v in enumerate(y_counts):
    ax.text(v, i, str(v), fontsize=14)
    plt.title('Different offer types')
# -

# Discount and bogo are equally given and on maximum times

# +
plt.figure(figsize=[6,6])
fig, ax = plt.subplots()    
y_counts = portfolio['duration'].value_counts()
y_counts.plot(kind='barh').invert_yaxis()
       
for i, v in enumerate(y_counts):
    ax.text(v, i, str(v), color='black', fontsize=14)
    plt.title('Different offer types\' duartion')
# -

# Here we can see that most of the offers are for the duration of 7 days

# ### Profile

profile.head(8)

print('profile: rows = {} ,columns = {}'.format((profile.shape[0]),(profile.shape[1])))

profile.describe()

profile.isnull().sum()
profile.shape

import seaborn as sns

    plt.figure(figsize=[6,6])
    fig, ax = plt.subplots()    
    y_counts = profile['gender'].value_counts()
    y_counts.plot(kind='barh').invert_yaxis()
       
    for i, v in enumerate(y_counts):
        ax.text(v, i, str(v), color='black', fontsize=14)
        plt.title('Count of Genders')

plt.pie(profile['gender'].value_counts() , labels = ['Male' , 'Female' , 'Other'])

# Mostly male are interested in the offers and they are the major ones

# ### Transcript

transcript.head(9)

transcript.describe()

transcript.info()

print('transcript: rows = {} ,columns = {}'.format((profile.shape[0]),(profile.shape[1])))

# ### Cleaning The Datasets

# #### Portfolio
#
# Renaming 'id' to 'offer_id'                                                                                            

portfolio.columns = ['channels', 'difficulty', 'duration', 'offer_id', 'offer_type', 'reward']

portfolio.columns

portfolio.head()

# # Profile
#
# Renaming 'id' to 'customer_id' , filling the missing values of age and income with mean value , filling the missing values of gender with mode

profile.columns

profile.columns = ['age', 'became_member_on', 'gender', 'customer_id', 'income']

profile.columns

profile['age'].fillna(profile['age'].mean()) #filling missing age with average age
profile['income'].fillna(profile['income'].mean()) #filling missing income with average income
profile['gender'].fillna(profile['gender'].mode()[0]) #filling missing gender with the most occuring gender
profile.head()

profile.isnull().sum()

# So there is not any missing value remaining in the profile dataframe

# # Transcript
#
# Renaming 'person' to 'customer_id' , splitting the 'value' column based on its keys and
# dropping the unnecessary columns

transcript.columns

transcript.columns = ['event', 'customer_id', 'time', 'value'] #changing the column name

transcript.head()

transcript.value.astype('str').value_counts().to_dict() #converting the values in the column 'value' to dictionary

transcript['offer_id'] = transcript.value.apply(lambda x: x.get('offer_id')) #splitting the 'value' into separate columns.here is 'offer_id'
transcript['offer id'] = transcript.value.apply(lambda x: x.get('offer id')) #splitting the 'value' into separate columns.here is 'offer id'
transcript['offer_id'] = transcript.apply(lambda x : x['offer id'] if x['offer_id'] == None else x['offer_id'], axis=1) #merging both 'offer id' and 'offer_id' into the same column 'offer_id'
transcript.drop('offer id',axis = 1,inplace = True)


transcript.head(10)

#splitting the reward and amount values in the 'value'
transcript['offer_reward'] = transcript['value'].apply(lambda x: x.get('reward'))
transcript['amount'] = transcript['value'].apply(lambda x: x.get('amount'))

transcript.drop('value' ,inplace = True , axis = 1)

transcript.isnull().sum()

transcript.fillna(0 , inplace = True) #filling the missing values with 0

transcript.head(10)

# ### Exploratory Data Analysis
# ### Now we will merge the dataframes 

merge_df = pd.merge(portfolio, transcript, on='offer_id')#merging portfolio and transcript dataframes on the basis of 'offer_id'
final_df = pd.merge(merge_df, profile, on='customer_id')#merging the merged dataframe of portfolio and transcript with profile dataframe on the basis of 'customer-id'

#Exploring the final merged dataframe
final_df

# ### Now we will see the different offer types and their counts

final_df['offer_type'].value_counts().plot.barh(title = 'Offer types with their counts')

# So,we can see that discount and bogo are thr most given offer types

# ### Now we will see the different events and their counts

final_df['event'].value_counts().plot.barh(title = 'Different events and their counts')

# So,in most of the cases offer is received by the user and it is not completed by him/her,means most of the people just ignore the offers they receive

# ### Now we will analyse this data on the basis of the age of the customers 

sns.distplot(final_df['age'] , bins = 50 , hist_kws = {'alpha' : 0.4});

# As we can see that the people after the age of 100 are just acting as outliers,so we will remove them

final_df = final_df[final_df['age']<=100] 

# Now seeing the distortion plot of age
sns.distplot(final_df['age'] , bins = 50 , hist_kws = {'alpha' : 0.4});

# We can observe that most of the customers are within the age group of 45-60 are the most frequent customers and more than any other group,this is quite interesting.

# ### Now,we will analyse this data on the basis of income of the customers 

sns.distplot(final_df['income'] , bins = 50 , hist_kws = {'alpha' : 0.4});

final_df['income'].mean()

# Now we can see that most people who are the customers of Starbucks have their income within the range of 55k - 75k with a mean income of 66413.35

# ### Now,we will see how our final dataframe is depedent on the 'gender' feature

final_df['gender'].value_counts().plot.barh(title = 'Analysing the gender of customers')

# So,we can see that most of the customers are male

# ### We will analyse the dataframe on the basis of  'offer_type' on the basis of gender

sns.countplot(x = 'offer_type' , hue = 'gender' , data = final_df)

# We can see that the count of gender weather it is male or female is approximately equal in the bogo and discount offers

# ### Now,we will see the relation between gender and events 

sns.countplot(x = 'event' , hue = 'gender' , data = final_df)

# So,from the exploratory data analysis we can see that most of the customers just receive the offers and they do not view them and the people who complete the offers they receive is quite less and most of the offers made by Starbuks are BOGO and Discount and most of the people that are the customers are within the age group of 45-60 and the most common gender is male and the people who are the customers of Starbucks have their income within the range of 55k - 75k

# # Making a Machine Learning Model

# First analysing our final dataset

final_df

# #### We will  now encode the categorical features like 'offer_type' , 'gender' , 'age'
# #### We  will encode the offer_id and customer_id

# +
final_df = pd.get_dummies(final_df , columns = ['offer_type' , 'gender' , 'age'])
#processing offer_id 
offer_id = final_df['offer_id'].unique().tolist()
offer_map = dict( zip(offer_id,range(len(offer_id))) )
final_df.replace({'offer_id': offer_map},inplace=True)

#processing customer_id 
customer_id = final_df['customer_id'].unique().tolist()
customer_map = dict( zip(customer_id,range(len(customer_id))) )
final_df.replace({'customer_id': customer_map},inplace=True)
# -

final_df.head()

# #### Now we will scale the numerical data including 'income' , 'difficulty' , 'duration' and many more... 

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
numerical_columns = ['income' , 'difficulty' , 'duration' , 'offer_reward' , 'time' , 'reward' , 'amount']
final_df[numerical_columns] = scaler.fit_transform(final_df[numerical_columns])

final_df.head()

# #### We will encode the values in the 'event' column 

final_df['event'] = final_df['event'].map({'offer received':1, 'offer viewed':2, 'offer completed':3})
final_df2 = final_df.drop('event' , axis = 1)

# #### Now encoding the  channels column

final_df2['web'] = final_df2['channels'].apply(lambda x : 1 if 'web' in x else 0)
final_df2['mobile'] = final_df2['channels'].apply(lambda x : 1 if 'mobile' in x else 0)
final_df2['social'] = final_df2['channels'].apply(lambda x : 1 if 'social' in x else 0)
final_df2['email'] = final_df2['channels'].apply(lambda x : 1 if 'email' in x else 0)

#Now dropping the Channels column
final_df2.drop('channels' , axis = 1 , inplace = True)

final_df2['became_member_on'] = final_df2['became_member_on'].apply(lambda x: pd.to_datetime(str(x), format='%Y%m%d'))
#adding new columns for month & year
final_df2['month_member'] = final_df2['became_member_on'].apply(lambda x: x.day)
final_df2['year_member'] = final_df2['became_member_on'].apply(lambda x: x.year)
#dropping the  became_member_on column
final_df2.drop('became_member_on',axis=1, inplace=True)

final_df2.shape

# # Training Our Dataset 

# ### Now splitting our 'final_df' into training and test set 

independent_variables = final_df2 #our dataset containing all the independent variables excluding the 'event'
dependent_variable = final_df['event'] #our final dataset containing the 'event'

from sklearn.model_selection import train_test_split
# splitting our dataset into training and test set and the test set being the 30% of the total dataset
x_train , x_test, y_train , y_test = train_test_split(independent_variables , dependent_variable , test_size = 0.3 , random_state = 1)

x_train.shape

x_test.shape

# # Testing Our Dataset

# We will implement a number of classification machine learning methods and will determine which method is best for our model
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

#We will test the quality of the predicted output on a number of metrics,i.e. accuracy score,f1 score
#We will use f1 score because it considers the class imbalance pretty well as compared to the accuracy score and is the best metric to evaluate our this model
from sklearn.metrics import confusion_matrix , accuracy_score , fbeta_score
def train_test_f1(model):
    """
    Returns the F1 score of training and test set of any particular model
    model : model name
    Returns
    f1_score_train : F1 score of training set
    f1_score_test : F1 score of test set
    """
    predict_train = (model.fit(x_train , y_train)).predict(x_train)
    predict_test = (model.fit(x_train , y_train)).predict(x_test)
    f1_score_train = fbeta_score(y_train , predict_train , beta = 0.5 , average = 'micro')*100
    f1_score_test = fbeta_score(y_test , predict_test , beta = 0.5 , average = 'micro')*100
    return f1_score_train , f1_score_test


# ### Implementing the KNN Model 

knn = KNeighborsClassifier()
f1_score_train_knn , f1_score_test_knn = train_test_f1(knn)#calculating the F1 scores

# ### Implementing the Logistic Regression

logistic = LogisticRegression()
f1_score_train_logistic , f1_score_test_logistic = train_test_f1(logistic)#calculating the F1 scores

# ### Implementing the Random Forest Classifier
#

random_forest = RandomForestClassifier()
f1_score_train_random , f1_score_test_random = train_test_f1(random_forest)#calculating the F1 scores

# ### Implementing the Decision Tree Classifier 

decision_tree = DecisionTreeClassifier()
f1_score_train_decision , f1_score_test_decision = train_test_f1(decision_tree)#calculating the F1 scores

# # Concluding from the above models and scores

f1_scores_models = {'model_name' : [knn.__class__.__name__ , logistic.__class__.__name__ , random_forest.__class__.__name__ , decision_tree.__class__.__name__] 
                   , 'Training set F1 Score' : [f1_score_train_knn , f1_score_train_logistic , f1_score_train_random , f1_score_train_decision],
                   'Test set F1 Score' : [f1_score_test_knn , f1_score_test_logistic , f1_score_test_random , f1_score_test_decision]}
f1_scores_df = pd.DataFrame(f1_scores_models)

f1_scores_df

# So,from the above dataframe we can conclude that when we trained our training dataset according to the KNeighborsClassifier our model performed worst,on training the model on RandomForestClassifier ,the training set F1 score is quite good i.e. 93.58 but it performed badly on the test set with a F1 score of 64.266 and when we trained our model on DecisionTreeClassifier our model's performance was best with the training set F1 score of 94.89 and the test set F1 score of 86.02,which means our model was able to classify between the events of offers upto a great extent.As this is a practical case study with real world dataset,we can say that our model has performed successfully.



StarcoderdataJupyterScriptsDedupFiltered