# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/nikshanpatel/CS634-DataMiningProject-1/blob/master/MidTerm_Project1_FinalVersion.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="JRYgpzp-3ZPB" colab_type="text"
# # **PROJECT**: **[Google Analytics Customer Revenue Prediction](https://www.kaggle.com/c/ga-customer-revenue-prediction/overview)**

# + [markdown] id="u-X9WkG3QKR4" colab_type="text"
# #### **The fundamental steps we will be taking in this project are:**
#
#
# >1.   **Understand the Objective of this project**
# 2.   **Data Collection & Data Preperation**
# 3.   **Modeling & Evaluation**
# 4.   **Communicate/Present Results**
#

# + [markdown] id="_OcdqZyqKEhs" colab_type="text"
# # **1 - Understanding the Objective**
# ---

# + [markdown] id="9wU-yXsnHATp" colab_type="text"
# ### **Predict how much GStore customers will spend**
#
# In this [Project](https://www.kaggle.com/c/ga-customer-revenue-prediction/overview), we’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to **predict revenue per customer**.
#  
# **What are we predicting?**
#
# We are predicting the natural log of the sum of all transactions per user.
#
# **Evaluation Metric**
#
# ***`RMSE`*** - *Root Mean Squared Error*
#
# **RMSE** is the standard deviation of residuals (predictions errors). When we scatter plot the target_test values and predictions, the RMSE tells us how concentrated are those points around the line of best fit.
#
# ![RSME](https://github.com/SerenaD005/CS634-DataMiningProject-1/blob/master/rsme.jpg?raw=1)
#

# + [markdown] colab_type="text" id="ghmLuPovKfkx"
# # **2 - Data Collection & Data Preperation**
# ---

# + [markdown] id="JMBjve1qu2-G" colab_type="text"
# ## **Mount** the **Google** **Drive** on to the Notebook.
#
#
# ---

# + id="U2q-kEm-utZA" colab_type="code" colab={}
from google.colab import drive
drive.mount('/content/drive')

# + id="X0zyLM2SCF0B" colab_type="code" outputId="2714b432-f49b-4e18-9d77-8cfc7ad7f9fb" colab={"resources": {"http://localhost:8080/nbextensions/google.colab/files.js": {"data": "<KEY> "ok": true, "headers": [["content-type", "application/javascript"]], "status": 200, "status_text": ""}}, "base_uri": "https://localhost:8080/", "height": 67}
# to load kaggle API
# !pip install -q kaggle 
from google.colab import files 
uploaded = files.upload()

# + id="qi6fFIT_CQf0" colab_type="code" colab={}
# !mkdir -p ~/.kaggle
# !cp kaggle.json ~/.kaggle/

# + id="gU7ORYOEClCw" colab_type="code" outputId="172a708a-7381-44df-948b-973208326f61" colab={"base_uri": "https://localhost:8080/", "height": 108}
#now we install 7zip
# !apt-get install p7zip-full

# + id="KLjiKwc6Cq1L" colab_type="code" outputId="aa21657b-5458-471a-b2db-6fa3d5214971" colab={"base_uri": "https://localhost:8080/", "height": 135}
# !kaggle datasets download -d niteshmistry/minigacustomerrevenueprediction

# + id="yeygSTiTCw3Z" colab_type="code" outputId="46320cb0-b821-4799-f7b1-64b74de1205e" colab={"base_uri": "https://localhost:8080/", "height": 556}
!7za e minigacustomerrevenueprediction.zip

# + [markdown] colab_type="text" id="Ni7QQp6xwjfR"
# ## **Import** the necessary **libraries** that are required to run this notebook:
# ---
# >1. **JSON**
# 2. **Pandas**
# 3. **NumPy**
# 4. **MatPlotLib**
# 5. **Plotly**  - Plot graphs
# 6. **SKLearn** - For model selection, preprocessing, metrics
#
#

# + colab_type="code" id="WAxWFR3014rL" colab={}
import os
import time
import json
import numpy as np
import pandas as pd
from pandas.io.json import json_normalize
import matplotlib.pyplot as plt
from matplotlib.ticker import StrMethodFormatter
from sklearn import linear_model
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
import datetime
from sklearn.linear_model import LinearRegression
import sklearn.metrics as metrics



# %matplotlib inline


# + [markdown] colab_type="text" id="itYfivY-13aH"
# ## **Datasets** that are required for this project that are necessary to run the notebook :
# ---
# >1. **TRAIN Dataset** - *`train.csv`* (315 MB)
# 2. **TEST Dataset** - *`test.csv`* (32 MB)
#
# ### Each row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user.
#
# ## The **data fields** in the given files are
#
# * **fullVisitorId**- A unique identifier for each user of the Google Merchandise Store.
# * **channelGrouping** - The channel via which the user came to the Store.
# * **date** - The date on which the user visited the Store.
# * **device** - The specifications for the device used to access the Store.
# * **geoNetwork** - This section contains information about the geography of the user.
# * **sessionId** - A unique identifier for this visit to the store.
# * **socialEngagementType** - Engagement type, either "Socially Engaged" or "Not Socially Engaged".
# * **totals** - This section contains aggregate values across the session.
# * **trafficSource** - This section contains information about the Traffic Source from which the session originated.
# * **visitId** - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.
# * **visitNumber** - The session number for this user. If this is the first session, then this is set to 1.
# * **visitStartTime** - The timestamp (expressed as POSIX time).
#
# ***NOTE***: Some of the fields are in json format. We will have to transform them into Dataframe
#

# + [markdown] id="ux5ZG4TwWCxl" colab_type="text"
# ## **LOAD** the data files that are required:
# ---

# + id="NKFpK2CFWh9Y" colab_type="code" outputId="6b87cc68-f7b7-4230-898d-e2e03a4e95fd" colab={"base_uri": "https://localhost:8080/", "height": 527}
TRAIN_FILE = '/content/mini-train.csv'
TEST_FILE = '/content/mini-test.csv'

#------------------------------------------------------------------------------------------------
# COMMON CSV DATA FILE LOADER - RETURN DATAFRAME
#------------------------------------------------------------------------------------------------

def convertCSVtoDF(csv_path, nrows):

    df = pd.read_csv(csv_path, dtype={"fullVisitorId": "str"}, nrows=nrows)
    print(f"LOADED: {os.path.basename(csv_path)}   SHAPE: {df.shape}")
    
    return df

#------------------------------------------------------------------------------------------------
# LOAD TRAINING DATASET file into DATAFRAME
#------------------------------------------------------------------------------------------------
train_df = convertCSVtoDF(TRAIN_FILE,None)
test_df = convertCSVtoDF(TEST_FILE,None)

#------------------------------------------------------------------------------------------------
# LISTING HEADERS AND SAMPLE RECORDS (5)
#------------------------------------------------------------------------------------------------
print("\n")
print(train_df.columns)


# + id="-pS5ABNd5jcn" colab_type="code" colab={}
#------------------------------------------------------------------------------------------------
# FUNCTION REMOVING EMPTY COLUMNS with value "not available in demo dataset"
#------------------------------------------------------------------------------------------------

def removeUselessColumns(df):
  EMPTY_COLUMNS = []

  dataframe = df

  for column in train_df.columns:
    if train_df[column][0] == 'not available in demo dataset':
      EMPTY_COLUMNS.append(column)
  
  print("DATAFRAME IS " + str(len(dataframe.columns)) + " COLUMNS...")
  print("FOUND " + str(len(EMPTY_COLUMNS)) + " USELESS COLUMNS...")
  print("COLUMNS:  " + str(EMPTY_COLUMNS))
  print("REMOVING " + str(len(EMPTY_COLUMNS)) + " COLUMNS...")
  dataframe.drop(EMPTY_COLUMNS, axis=1, inplace=True)
  print("DATAFRAME IS NOW " + str(len(dataframe.columns)) + " COLUMNS...")

  return dataframe



# + [markdown] id="ijT7k9-W5wqP" colab_type="text"
#
# **REMOVE USELESS COLUMNS** that has values "**NOT AVAILABLE IN DEMO DATASET**

# + id="wHQDZ6xgF52G" colab_type="code" outputId="cfcff494-3929-473a-d334-1cafbeb825ef" colab={"base_uri": "https://localhost:8080/", "height": 488}
#------------------------------------------------------------------------------------------------
# TRAINING DATAFRAME - REMOVING EMPTY COLUMNS with value "not available in demo dataset"
#------------------------------------------------------------------------------------------------

train_df = removeUselessColumns(train_df)
train_df.head(5)

# + id="RrUJjyzrF-Xi" colab_type="code" outputId="5e78fca0-bc32-4b40-a747-03f708ecf69f" colab={"base_uri": "https://localhost:8080/", "height": 330}
train_df.describe()

# + id="n7dc9a8HeKUD" colab_type="code" outputId="5c8bfb96-b584-42a2-8e2e-b8b133168324" colab={"base_uri": "https://localhost:8080/", "height": 115}
train_df['date'].head()

# + id="ZjAuXXzNCJrq" colab_type="code" outputId="3fd88888-4e77-493f-95b1-839d97da943b" colab={"base_uri": "https://localhost:8080/", "height": 98}
#------------------------------------------------------------------------------------------------
# TESTING DATAFRAME - REMOVING EMPTY COLUMNS with value "not available in demo dataset"
#------------------------------------------------------------------------------------------------
test_df = removeUselessColumns(test_df)


# + id="iFjbxvn2EIkf" colab_type="code" outputId="6234d4c1-1c67-442c-8aff-bdb0db462e48" colab={"base_uri": "https://localhost:8080/", "height": 404}
test_df.head(5)

# + id="S9eu2vlJENyw" colab_type="code" outputId="342e8ef9-8658-4984-ffdd-f8cfd1081810" colab={"base_uri": "https://localhost:8080/", "height": 330}
test_df.describe()

# + [markdown] id="PcT43DABT7_6" colab_type="text"
# **Impute 0 for ALL missing target values (totals.transactionRevenue)**

# + [markdown] id="zeleDYJTHH16" colab_type="text"
# ## Check Missing Data

# + id="G87kpIOaHbAd" colab_type="code" outputId="c16f8259-9a81-43c6-eaf8-aa1f931e558a" colab={"base_uri": "https://localhost:8080/", "height": 296}
# checking missing data
miss_per = {}
for k, v in dict(train_df.isna().sum(axis=0)).items():
    if v == 0:
        continue
    miss_per[k] = 100 * float(v) / len(train_df)
miss_per

# + [markdown] id="pWM7iHTyX9pV" colab_type="text"
# **Remove columns that are in TRAINING which are not there in TEST**

# + id="dd44J7YCYKU0" colab_type="code" outputId="be70dc0d-a94e-4041-bfdc-7e28118b4ff8" colab={"base_uri": "https://localhost:8080/", "height": 32}
print("Columns not in TEST but in TRAIN : ", set(train_df.columns).difference(set(test_df.columns)))

# + [markdown] colab_type="text" id="go1wEnjaTjQo"
# # **3 - MODELING & EVALUTATION**
# ---

# + [markdown] id="egPQKGV4Ah8B" colab_type="text"
#  
#  
# Since we are predicting the natural log of sum of all transactions of the user, let us sum up the transaction revenue at user level and take a log and then do a scatter plot.
#

# + [markdown] id="u_BWXJJ4QFNq" colab_type="text"
# First we have to change the datatype to 'FLOAT' to make sure the value is a dollar value

# + id="80ejCkQSJhIh" colab_type="code" colab={}
train_df["totals.transactionRevenue"] = train_df["totals.transactionRevenue"].astype('float')

# + [markdown] id="mmAh-mjOQo_x" colab_type="text"
# Next we have to group the USER level transaction to by using the GROUPBY function on the dataframe. This will give us the Total Transaction Value for given user. 

# + id="2DQId6KsRBQS" colab_type="code" outputId="a45b32c3-0108-499b-8ad9-0d284f4edc26" colab={"base_uri": "https://localhost:8080/", "height": 411}
grouped_df_sum = train_df.groupby("fullVisitorId")["totals.transactionRevenue"].sum().reset_index()
grouped_df_sum.sort_values(by=['totals.transactionRevenue'], inplace=True, ascending=False)
grouped_df_sum

# + id="T_TKLoAC6FRc" colab_type="code" outputId="a0ed6dee-0fc2-41b7-b501-b576f9103532" colab={"base_uri": "https://localhost:8080/", "height": 515}
ax = grouped_df_sum.hist('totals.transactionRevenue', bins=25, grid=False, figsize=(12,8), color='#86bf91', zorder=2, rwidth=0.9)
ax = ax[0]
for x in ax:

    # Despine
    x.spines['right'].set_visible(False)
    x.spines['top'].set_visible(False)
    x.spines['left'].set_visible(False)

    # Switch off ticks
    x.tick_params(axis="both", which="both", bottom="off", top="off", labelbottom="on", left="off", right="off", labelleft="on")

    # Draw horizontal axis lines
    vals = x.get_yticks()
    for tick in vals:
        x.axhline(y=tick, linestyle='dashed', alpha=0.4, color='#eeeeee', zorder=1)

    # Remove title
    x.set_title("")

    # Set x-axis label
    x.set_xlabel("Total Transaction Revenue (per User)", labelpad=20, weight='bold', size=12)

    # Set y-axis label
    x.set_ylabel("Visitors", labelpad=20, weight='bold', size=14)



# + colab_type="code" id="rasz-WAS8baj" outputId="5b50b906-d9e0-4a7b-997b-859cf73a912d" colab={"base_uri": "https://localhost:8080/", "height": 411}
grouped_df_count = train_df.groupby("fullVisitorId")["totals.transactionRevenue"].count().reset_index()
grouped_df_count.sort_values(by=['totals.transactionRevenue'], inplace=True, ascending=False)
grouped_df_count

# + colab_type="code" id="187w0umb8EsZ" outputId="78f7a834-0621-4b56-b977-025fcd9af1b1" colab={"base_uri": "https://localhost:8080/", "height": 515}
ax = grouped_df_count.hist('totals.transactionRevenue', bins=25, grid=False, figsize=(12,8), color='#86bf91', zorder=2, rwidth=0.9)
ax = ax[0]
for x in ax:

    # Despine
    x.spines['right'].set_visible(False)
    x.spines['top'].set_visible(False)
    x.spines['left'].set_visible(False)

    # Switch off ticks
    x.tick_params(axis="both", which="both", bottom="off", top="off", labelbottom="on", left="off", right="off", labelleft="on")

    # Draw horizontal axis lines
    vals = x.get_yticks()
    for tick in vals:
        x.axhline(y=tick, linestyle='dashed', alpha=0.4, color='#eeeeee', zorder=1)

    # Remove title
    x.set_title("")

    # Set x-axis label
    x.set_xlabel("Total Transaction Count (per User)", labelpad=20, weight='bold', size=12)

    # Set y-axis label
    x.set_ylabel("Visitors", labelpad=20, weight='bold', size=14)



# + [markdown] id="kqWDhqmgPbMh" colab_type="text"
# ## **numpy.log1p() in Python**
# **numpy.log1p(arr, out = None, *, where = True, casting = ‘same_kind’, order = ‘K’, dtype = None, ufunc ‘log1p’) :**
#
# This mathematical function helps user to calculate **natural logarithmic value of x+1** where x belongs to all the input array elements.
#
# * **log1p is reverse of exp(x) – 1.**
#

# + [markdown] id="hNJ7jHMaPyis" colab_type="text"
# ## Plotting log of the revenue by customers

# + id="xsf9oYbUJAaq" colab_type="code" outputId="d0a35c5a-4c5b-45d0-8751-fe4ad74fba5a" colab={"base_uri": "https://localhost:8080/", "height": 623}
plt.figure(figsize=(15,10))
plt.scatter(range(grouped_df_sum.shape[0]), np.sort(np.log1p(grouped_df_sum["totals.transactionRevenue"].values)))
plt.title('Total Transaction Revenue for each Visitor')
plt.xlabel('index', fontsize=14)
plt.ylabel('TransactionRevenue', fontsize=14)
plt.show()

# + colab_type="code" id="yTLpjO2JOHCF" outputId="b3a14136-1c80-4171-bedc-a7309de88c09" colab={"base_uri": "https://localhost:8080/", "height": 49}
non_zero_instance = pd.notnull(train_df["totals.transactionRevenue"]).sum()
print("Number of instances in train set with non-zero revenue : ", non_zero_instance, " and ratio is : ", non_zero_instance / train_df.shape[0])
non_zero_revenue = (grouped_df_sum["totals.transactionRevenue"]>0).sum()
print("Number of unique customers with non-zero revenue : ", non_zero_revenue, "and the ratio is : ", non_zero_revenue / train_df.shape[0])

# + id="CBPJDcBxZOg4" colab_type="code" outputId="2bf6ae2e-a57f-4974-ccc4-f0f0c813e7ea" colab={"base_uri": "https://localhost:8080/", "height": 65}
print("Number of unique visitors in train set : ",train_df.fullVisitorId.nunique(), " out of rows : ",train_df.shape[0])
print("Number of unique visitors in test set : ",test_df.fullVisitorId.nunique(), " out of rows : ",test_df.shape[0])
print("Number of common visitors in train and test set : ",len(set(train_df.fullVisitorId.unique()).intersection(set(test_df.fullVisitorId.unique())) ))

# + [markdown] id="lWcuMirAQOjg" colab_type="text"
# ## Removing instances with zero revenue

# + id="VaGPWegVQaNL" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 761} outputId="c29c382c-469f-4e1c-efd2-3bce1ee10623"
nzi_train_df = train_df[train_df['totals.transactionRevenue'] > 0]
nzi_train_df

# + id="vYw7MxocZJ00" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1000} outputId="ae97cc74-763c-47d7-ebf2-892ef0c36926"
nzi_test_df = test_df[test_df['totals.transactionRevenue'] > 0]
nzi_test_df

# + id="4D3t9TYXa7bN" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1000} outputId="2006bb8d-9185-4d17-b804-7052769277dd"
nzi_test_df.count()

# + id="x31RCpA2UtHy" colab_type="code" outputId="aaf10b66-0580-430a-9ae4-6d2cde8fbcbe" colab={"base_uri": "https://localhost:8080/", "height": 623}
# replotting revenue
nzi_grouped_df_sum = nzi_train_df.groupby("fullVisitorId")["totals.transactionRevenue"].sum().reset_index()
plt.figure(figsize=(15,10))
plt.scatter(range(nzi_grouped_df_sum.shape[0]), np.sort(nzi_grouped_df_sum["totals.transactionRevenue"].values))
plt.title('Total Log of Transaction Revenue for each Visitor')
plt.xlabel('index', fontsize=14)
plt.ylabel('TransactionRevenue', fontsize=14)
plt.show()

# + id="XJQarD1DT-gI" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 623} outputId="74971197-d959-47ef-90b5-de501e2d1550"
# replotting log(revenue)

plt.figure(figsize=(15,10))
plt.scatter(range(nzi_grouped_df_sum.shape[0]), np.sort(np.log1p(nzi_grouped_df_sum["totals.transactionRevenue"].values)))
plt.title('Total Log of Transaction Revenue for each Visitor')
plt.xlabel('index', fontsize=14)
plt.ylabel('TransactionRevenue', fontsize=14)
plt.show()

# + id="jU4AkdvQgLSh" colab_type="code" colab={}
nzi_grouped_df_test_sum = nzi_test_df.groupby("fullVisitorId")["totals.transactionRevenue"].sum().reset_index()

# + [markdown] id="rgs4Ju5wV2QA" colab_type="text"
# ## Running Regression Model
#
# next, we split 80% of the data to the training set while 20% of the data to test set. The test_size variable is where we actually specifiy the proportion of the test set.

# + id="i3sRGKezj3Cq" colab_type="code" colab={}


# + id="pUoyen1WXQwW" colab_type="code" colab={}
X = nzi_grouped_df_sum['fullVisitorId'].values.reshape(-1,1)
y = nzi_grouped_df_sum['totals.transactionRevenue'].values.reshape(-1,1)

# + id="4QOI-Z1WfMYU" colab_type="code" colab={}
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# + id="7nRRRg5mfeNc" colab_type="code" colab={}
nzi_grouped_df_sum.fillna(-99999, inplace=True) # training
nzi_grouped_df_test_sum.fillna(-99999, inplace=True) # testing

# + id="f0g7vJ7Ugcs7" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 32} outputId="e67ff6db-2fa5-49d1-8fef-7b35ec195c00"
#train the linear regression algorithm
regressor = LinearRegression()  
regressor.fit(X_train, y_train)

# + id="bzcnw5x9gf2J" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 49} outputId="748815c1-9190-4c97-a3d8-c275adf7b182"
#to retrieve the intercept
print(regressor.intercept_)

#to retrieve the slope
print(regressor.coef_)

# + id="6C3ZfHq6gi6b" colab_type="code" colab={}
#make predicitions on the test data 
y_pred = regressor.predict(X_test)

# + id="CMIganfMgmCW" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 291} outputId="1467b25b-66ab-43cf-cc1d-cf072b066d57"
nzi_group_df_test_sum = pd.DataFrame({'Actual': y_test.flatten(), 'Predicted': y_pred.flatten()})
nzi_group_df_test_sum.describe()


# + id="E5VvcP8PgwXS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1000} outputId="5080f07a-c3fa-4bec-f625-2e4cdeb9705a"
nzi_group_df_test_sum

# + id="cEqYLZgDgy8j" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 603} outputId="31f1446a-3175-47ec-850b-1c1e5243078d"
#visualize compairson results with a bar graph
df1 = nzi_group_df_test_sum.head(25)
df1.plot(kind='bar',figsize=(16,10))
plt.grid(which='major', linestyle='-', linewidth='0.5', color='green')
plt.grid(which='minor', linestyle=':', linewidth="0.5", color='black')
plt.show()

# + id="Bysyz_xpmhnk" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 104} outputId="2cb336c6-fe71-4912-d965-d5706690bde7"
x_test_array = []
y_test_array = []
y_pred_array = []

for item1 in y_test:
  y_test_array.append(item1[0])

for item2 in X_test:
  x_test_array.append(item2[0])

for item3 in y_pred:
  y_pred_array.append(item3[0])

print(x_test_array)
print(y_test_array)
print(y_pred_array)

# + id="GmUyd5tsg52V" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 275} outputId="8fd0557e-2414-459c-ded0-d1c7a51286aa"
import numpy as np
import matplotlib.pyplot as plt
#test scatter plot with a straight line
plt.scatter(x_test_array, y_test_array, color='grey')
plt.plot(x_test_array, y_pred_array, color='red', linewidth=2)
plt.show()

# + id="cYoP0GzBuhdl" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 32} outputId="20779024-8228-4c47-c609-2c306cc4e980"
print('Root Mean Square Error:', np.sqrt(metrics.mean_squared_error(y_test_array, y_pred_array)))

# + [markdown] id="3R5Uoo4OVnn7" colab_type="text"
# ## Running GBM Model 

# + [markdown] id="N7-Kc3KBWo30" colab_type="text"
# We also tried GBM (Gradient Boosting Machine) model. 
# We got the kernal from *sudalairajkumar/simple-exploration-baseline-ga-customer-revenue*
#
#
#

# + id="3JmgT9p5YwPB" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 49} outputId="9e436a65-6f9e-43e9-d609-41276bdbd38d"
TRAIN_FILE = '/content/mini-train.csv'
TEST_FILE = '/content/mini-test.csv'
train_df = convertCSVtoDF(TRAIN_FILE,None)
test_df = convertCSVtoDF(TEST_FILE,None)


# + id="H3cfSQAL6ACd" colab_type="code" colab={}
# Remove 0 instances
train_df = train_df[train_df["totals.transactionRevenue"] > 0]
test_df = test_df[test_df["totals.transactionRevenue"] > 0 ]

# + id="Foz1YX-Z1fxg" colab_type="code" colab={}
train_y = train_df["totals.transactionRevenue"].values
train_id = train_df["fullVisitorId"].values
test_id = test_df["fullVisitorId"].values
test_y = test_df["totals.transactionRevenue"].values

# label encode the categorical variables and convert the numerical variables to float
cat_cols = ["channelGrouping", "device.browser", 
            "device.deviceCategory", "device.operatingSystem", 
            "geoNetwork.city", "geoNetwork.continent", 
            "geoNetwork.country", "geoNetwork.metro",
            "geoNetwork.networkDomain", "geoNetwork.region", 
            "geoNetwork.subContinent", "trafficSource.adContent", 
            "trafficSource.adwordsClickInfo.adNetworkType", 
            "trafficSource.adwordsClickInfo.gclId", 
            "trafficSource.adwordsClickInfo.page", 
            "trafficSource.adwordsClickInfo.slot", "trafficSource.campaign",
            "trafficSource.keyword", "trafficSource.medium", 
            "trafficSource.referralPath", "trafficSource.source",
            'trafficSource.adwordsClickInfo.isVideoAd', 'trafficSource.isTrueDirect']

# + id="XFCQ1IKx0kmz" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 395} outputId="1b91c430-17cf-4924-fb5e-3b3a77866222"
for col in cat_cols:
    print(col)
    lbl = preprocessing.LabelEncoder()
    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))
    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))
    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))



# + id="ny-yT97pc2eT" colab_type="code" colab={}
num_cols = ["totals.hits", "totals.pageviews", "visitNumber", "visitStartTime", 'totals.bounces',  'totals.newVisits']    
for col in num_cols:
    train_df[col] = train_df[col].astype(float)
    test_df[col] = test_df[col].astype(float)


# + id="C3Wxa41AdAVB" colab_type="code" colab={}
# Split the train dataset into development and valid 
dev_df = train_df[train_df['date']<=20170531]
val_df = train_df[train_df['date']>20170531]
dev_y = np.log1p(dev_df["totals.transactionRevenue"].values)
val_y = np.log1p(val_df["totals.transactionRevenue"].values)

dev_X = dev_df[cat_cols + num_cols] 
val_X = val_df[cat_cols + num_cols] 
test_X = test_df[cat_cols + num_cols] 

# + id="sGn5jfDB7udz" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 450} outputId="03859a43-70f1-4e20-bc96-778e29a3665f"
val_X


# + id="fSeIByY8iAVu" colab_type="code" colab={}
# custom function to run light gbm model
def run_lgb(train_X, train_y, val_X, val_y, test_X):
    params = {
        "objective" : "regression",
        "metric" : "rmse", 
        "num_leaves" : 30,
        "min_child_samples" : 100,
        "learning_rate" : 0.1,
        "bagging_fraction" : 0.7,
        "feature_fraction" : 0.5,
        "bagging_frequency" : 5,
        "bagging_seed" : 2018,
        "verbosity" : -1
    }
    
    lgtrain = lgb.Dataset(train_X, label=train_y)
    lgval = lgb.Dataset(val_X, label=val_y)
    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100)
    
    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)
    pred_val_y = model.predict(val_X, num_iteration=model.best_iteration)
    return pred_test_y, model, pred_val_y



# + id="PRlGDVAqiqbi" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 82} outputId="57c151c9-bbe0-4447-ddb4-fcdeee82b894"
# Training the model #
pred_test, model, pred_val = run_lgb(dev_X, dev_y, val_X, val_y, test_X)

# + id="-YvuZhuG1ost" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 291} outputId="9f42b7da-1a69-45f1-c58f-60224cdfe51c"
test_df_gbm = pd.DataFrame({'Actual': test_y.flatten(), "Predicted_gbm": pred_test.flatten()})
test_df_gbm.describe()

# + id="KWMBPm2L9nTV" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 32} outputId="e15a76a9-d882-4b43-feb2-910d78a3253f"
print('Root Mean Square Error:', np.sqrt(metrics.mean_squared_error(test_y, pred_test)))

StarcoderdataJupyterScriptsDedupFiltered