# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="B9PKJggkxfMo"
# ## Preparation

# + [markdown] id="EBGxhcG7xZzr"
# Welcome to the Vectice tutorial notebook!
#
#
# Through this notebook, we will be illustrating how to log the following information into Vectice using the Vectice SDK:
# - Dataset versions
# - Model versions
# - Runs 
#
# For more information on the tutorial, please refer to the "Vectice Tutorial Page" inside the app.
#
#

# + [markdown] id="3McxMcLPxkdJ"
# ### Install Vectice and GCS packages

# + colab={"base_uri": "https://localhost:8080/"} id="aXrYRKNiIlOn" outputId="1d81b635-be99-4f15-d056-3608c2d1142a"
## Requirements
# !pip3 install --q fsspec
# !pip3 install --q gcsfs

#Install Vectice Python library 
# In this tutorial we will do code versioning using github, we also support gitlab
# and bitbucket: !pip install -q "vectice[github, gitlab, bitbucket]"
# !pip3 install --q vectice[github]

# + colab={"base_uri": "https://localhost:8080/"} id="hHkVwYekVYxU" outputId="32d114ad-a22a-411e-cd40-fa99b5d37d20"
#Verify if Vectice SDK was installed
# !pip3 show vectice

# + [markdown] id="8a-NGpCnxo0A"
# ### Retrieve the data from GCS
#
# We are going to load data stored in Google Cloud Storage, that is provided by Vectice for this tutorial.
#

# + colab={"base_uri": "https://localhost:8080/", "height": 223} id="X44K8EaHJBAu" outputId="fb40eaa4-3e26-43af-edf3-54dd2b758f7f"
import os
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import seaborn as sns
# %matplotlib inline

# Double check the json file name below so that it matches the name of the file that you uploaded.
# Note that the key provided for this tutorial does not have permissions for you to write to GCS. 
# You can only use it to read the data.
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'readerKey.json'

# Once your file is loaded you can view your dataset in a Pandas dataframe.
df = pd.read_csv('gs://vectice_tutorial/kc_house_data_cleaned.csv')

# Run head to make sure the data was loaded properly
df.head()


# + [markdown] id="c0k4Xz-sxtgZ"
#
# We will use the scikit-learn library for modeling and tracking experiments directly through the Vectice SDK.

# + colab={"base_uri": "https://localhost:8080/"} id="KsXBhrmeJ0sZ" outputId="eee29d29-b8df-48ee-a524-60b6d67506be"
#Import the Vectice library
from vectice import Vectice
from vectice.models import JobType
from vectice.entity.model import ModelType
# Specify the API endpoint for Vectice. This shouldn't change.
os.environ['VECTICE_API_ENDPOINT']= "beta.vectice.com"


# To use the Vectice SDK, you first need to authenticate your account using an API key.
# You can generate an API key from the Vectice UI, by going to the "API Tokens" tab in your workspace

# Copy and paste your API key here
os.environ['VECTICE_API_TOKEN'] = "API TOKEN"

# Next, you need to specify the tutorial project, where you will log the assets that will be generated in this notebook. 
# For this, you need a "Project Token", that you will find under the "Settings" tab of your project

# Copy and paste your Project Token here
vectice = Vectice(project_token="Project Token")

print(vectice)

# + [markdown] id="26ChSoJtxyiu"
# ### Split dataset into training and testing
#
# Let's split the dataset into train and test sets and save them in GCS.
# (The GCS code has been commented out as the data has already been generated.)
#
#
# -

# Testing the model on the same data as it was trained on will lead to an overfit and poor performance in real-life scenarios.
# In order to avoid that, we split our data into 2 pieces: train set and test set. The most common practice is to do a 80-20 split.

# + id="G4Wu5npWJ5U1"
import string
from math import sqrt

# Load scikit-learn packages
from sklearn.model_selection import train_test_split  # Model Selection
from sklearn.metrics import mean_absolute_error, mean_squared_error  # Model Evaluation
from sklearn.linear_model import LinearRegression  # Linear Regression
from sklearn.tree import DecisionTreeRegressor, plot_tree  # Decision Tree Regression
from sklearn.ensemble import RandomForestRegressor  # Random Forest Regression


# + colab={"base_uri": "https://localhost:8080/"} id="JU50t9fHKALv" outputId="effffc0a-21f0-4dbd-8c9f-62e2039bf1d2"
# The Vectice SDK automatically detects if there have been changes to the dataset you are using.
# If it detects changes, it will generate a new version of your dataset automatically. 
# For this tutorial, we changed the data since Albert left the company.
# So, the SDK will create a new dataset version the first time you execute this code.

input_ds_version = vectice.create_dataset_version().with_parent_name("cleaned_kc_house_data")


# With this line we declare a reference to code existing in GitHub as the code at the origin of the outputs
input_code = Vectice.create_code_version_with_github_uri("https://github.com/vectice/vectice-examples",
                        "Notebooks/Vanilla/Tutorial/Tutorial_Modelling_All%20-%20Jupyter.ipynb") 



# Each execution for a given job is called a run. When creating a run you need to specify:
# 1) a job name (mandatory)
# 2) a job type (optional)
# Job names and job types are useful to group and search runs in the Vectice UI.
# For this run, we will use the job name "80/20 Split" and the job type "PREPARATION"
vectice.create_run("80/20 Split", JobType.PREPARATION)
#Start the run
## Using the with method end the run automatically. We don't need to add vectice.end_run(outputs=outputs) to end the run
with vectice.start_run(inputs=[input_ds_version,input_code]) as run:

# We will use an 80/20 split to prepare the data
    test_size = 0.2

# We will set the random seed so we always generate the same split.
    random_state = 42

    train, test = train_test_split(df, test_size = test_size, random_state = random_state)

# We commented out the code to persist the training and testing test in GCS, 
# because we already generated the data for you.
# The key provided for this tutorial give you read access only to GCS.
# We left the code below for convenience, in case you want to use your own credentials and GCS bucket.
# train.to_csv (r'gs://vectice_tutorial/training_data.csv', index = False, header = True)
# test.to_csv (r'gs://vectice_tutorial/testing_data.csv', index = False, header = True)

# Generate X_train, X_test, y_train, y_test, which we will need for modeling
    X = df.drop("price", axis=1).values
    y = df["price"].values
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=random_state)


# Let's create new versions of the training and testing dataset if the data has changed.
# We will use the existing dataset created by Albert, so that we can append new 
# dataset versions to it.
    train_ds_version = vectice.create_dataset_version().with_parent_name("train_cleaned_kc_house_data")
    test_ds_version = vectice.create_dataset_version().with_parent_name("test_cleaned_kc_house_data")

# Attach the output datasets to the run.
    run.add_outputs([train_ds_version,test_ds_version])

# We can preview one of our generated outputs to make sure that everything was executed properly.
X_train


# + [markdown] id="UWCBtvUDx2km"
# ### Name your experiments

# + [markdown] id="khVCyL5ox2Vn"
# Input the experiment names and make them unique

# + id="6QtC_bmCMhXy"
# Let's define a helper function that we will use to generate unique model version names.
# We will make the model version name optional in a future edition of the Vectice SDK
# so this function won't be necessary anymore.
import random

def get_random_string(length):
    return "".join(random.choice(string.ascii_letters) for i in range(length))



# + [markdown] id="tNlnYdQ_x8AZ"
# ## Modeling

# + [markdown] id="14f1pDCdx82I"
# ### Linear regression model
# -

# Regression is a method of modelling a target value based on independent predictors. This method is mostly used for forecasting and finding out cause and effect relationship between variables. Regression techniques mostly differ based on the number of independent variables and the type of relationship between the independent and dependent variables.

# + [markdown] id="jQU-NU7Mx_Mn"
# First, we will do a basic Linear Regression and observe the baseline accuracy metrics.

# + colab={"base_uri": "https://localhost:8080/"} id="eu5z76qtMvW6" outputId="25aa715e-d8af-4b9f-f24e-e8c2d5b84dc4"
# Each execution for a given job is called a run, for LR we will only do one run.
# Setting a job's name is mandatory when starting a run 
# and is useful to group and search runs in the Vectice UI.

# Linear regression model training
vectice.create_run("LR-Model", JobType.TRAINING)
## Start the run
## Using the with method end the run automatically. We don't need to add vectice.end_run(outputs=outputs) to end the run
with vectice.start_run(inputs=[train_ds_version,test_ds_version,input_code]) as run:

    lr_rg = LinearRegression()
    lr_rg.fit(X_train, y_train)
    lr_pred = lr_rg.predict(X_test)

# Evaluate Metrics
    MAE = mean_absolute_error(lr_pred, y_test)
    RMSE = sqrt(mean_squared_error(lr_pred, y_test))

    print("Root Mean Squared Error: ", RMSE)
    print("Mean Absolute Error: ", MAE)

# Let's log the model we trained along with its metrics, as a new version 
# of the "Regressor" model in Vectice.
# Note that we used a random string to have a unique model version name in Vectice. 
## Here we create our model version.
## We can declare the model type using the .with_type method
    model_version = vectice.create_model_version().with_parent_name("Regressor").with_type(ModelType.REGRESSION).with_algorithm("Linear Regression").with_metric("RMSE",RMSE).with_metric("MAE",MAE).with_user_version(get_random_string(12))
## Attach the model version to the run
    run.add_outputs(outputs=[model_version])


# + [markdown] id="kHfTyn5ByD0M"
# ### Decision tree model
# -

# A decision tree is an upside-down tree that makes decisions based on the conditions present in the data.
# Decision Tree algorithm has become one of the most used machine learning algorithm both in competitions like Kaggle as well as in business environment. Decision Tree can be used both in classification and regression problem. A decision tree is arriving at an estimate by asking a series of questions to the data, each question narrowing our possible values until the model get confident enough to make a single prediction. The order of the question as well as their content are being determined by the model. In addition, the questions asked  are all in a True/False form

# + [markdown] id="-FIP53giyFc_"
# In this section let's use the decision tree algorithm and compare the accuracy to the logistic regression algorithm. We will try different values for the tree_depth. We will log the model parameters and metrics in Vectice.

# + colab={"base_uri": "https://localhost:8080/", "height": 594} id="k-iP1UhKuLeN" outputId="debf8bd9-9839-4d87-9d98-66921b906dc6"
# We can do a few runs with different max depth for the tree.
# Just change the value below and re-run this cell.
# The model versions you created will show up in the Vectice UI as new versions 
# of the "Regressor" Model. You can easily compare them from there.
tree_depth = 6

vectice.create_run("DT-Model", JobType.TRAINING)
## Start the run
vectice.start_run(inputs=[train_ds_version,test_ds_version,input_code])

dtr = DecisionTreeRegressor(max_depth=tree_depth, min_samples_split=50)
dtr.fit(X_train,y_train)
dtr_pred = dtr.predict(X_test) 

data_feature_names = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'floors',
                      'waterfront', 'view', 'condition', 'grade', 'sqft_above',
                      'sqft_basement', 'yr_built', 'yr_renovated', 'zipcode', 'lat',
                      'long', 'sqft_living15', 'sqft_lot15']

# Visualize the Decision Tree Model
plt.figure(figsize=(25, 10))
plot_tree(dtr, feature_names=data_feature_names, filled=True, fontsize=10)

MAE = mean_absolute_error(dtr_pred, y_test)
RMSE = sqrt(mean_squared_error(dtr_pred, y_test))

print("Root Mean Squared Error:", RMSE)
print("Mean Absolute Error:", MAE)
## Here we create our model version.
## We can declare the model type using the .with_type method
model_version = vectice.create_model_version().with_parent_name("Regressor").with_type(ModelType.REGRESSION).with_algorithm("Decision Tree").with_property("Tree Depth",str(tree_depth)).with_metric("RMSE",RMSE).with_metric("MAE",MAE).with_user_version(get_random_string(12))
## Here we should end the run and attach the created model version to it since we don't use the with method
vectice.end_run(outputs=[model_version])

# + [markdown] id="eSA7zAZGyQmu"
# ### Random forest model
#
# Let's use the Random Forest Regression and do some hyper-parameter tuning on it.
# -

# Random Forest is also a “Tree”-based algorithm that uses the qualities features of multiple Decision Trees for making decisions
# Therefore, it can be referred to as a ‘Forest’ of trees and hence the name “Random Forest”. The term ‘Random’ is due to the fact that this algorithm is a forest of ‘Randomly created Decision Trees’.
# The Decision Tree algorithm has a major disadvantage in that it causes over-fitting. This problem can be limited by implementing the Random Forest Regression in place of the Decision Tree Regression. Additionally, the Random Forest algorithm is also very fast and robust than other regression models.

# + colab={"base_uri": "https://localhost:8080/"} id="B3I4rcTVu5PF" outputId="d849f61b-89bb-4a24-f6c1-c1345c7fff30"
# You can modify the parameters below and execute multiple runs to train 
# different versions of RF model.
nb_trees = 60
min_samples = 30

vectice.create_run("RF-Model", JobType.TRAINING)
## Start the run
## Using the with method end the run automatically. We don't need to add vectice.end_run(outputs=outputs) to end the run
with vectice.start_run(inputs=[train_ds_version,test_ds_version,input_code]) as run:

    rf_regressor = RandomForestRegressor(n_estimators=nb_trees, min_samples_leaf=min_samples)
    rf_regressor.fit(X_train, y_train)
    rf_regressor.score(X_test, y_test)
    rf_regressor_pred = rf_regressor.predict(X_test)

    MAE = mean_absolute_error(rf_regressor_pred, y_test)
    RMSE = sqrt(mean_squared_error(rf_regressor_pred, y_test))

    print("Root Mean Squared Error:", RMSE)
    print("Mean Absolute Error:", MAE)

# Here's an alternative version to declare metrics
    metrics = [("RMSE",RMSE), ("MAE",MAE)]
## Here we create our model version.
## We can declare the model type using the .with_type method
    model_version = vectice.create_model_version().with_parent_name("Regressor").with_type(ModelType.REGRESSION).with_algorithm("Random Forest").with_property("nb_trees",str(nb_trees)).with_property("min_samples",str(min_samples)).with_metrics(metrics).with_user_version(get_random_string(12))
##Attach the created model version to the run
    run.add_outputs(outputs=[model_version])


# + [markdown] id="QhotjOQUyTcH"
# We can see that the Random Forest Regressor model gives the lowest error and should be the preferred approach despite the complexity of the algorithm. Let's get the list of features' importance to discuss which variables are influencing the model the most.

# + colab={"base_uri": "https://localhost:8080/", "height": 279} id="CHcKAqzDyUQZ" outputId="9ae03f12-d1b0-4380-86e8-81af76f8c464"
columns = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'floors',
           'waterfront', 'view', 'condition', 'grade', 'sqft_above',
           'sqft_basement', 'yr_built', 'yr_renovated', 'zipcode', 'lat',
           'long', 'sqft_living15', 'sqft_lot15']

importance = pd.DataFrame({'Importance': rf_regressor.feature_importances_ * 100}, index=columns)
importance.sort_values(by="Importance", axis=0, ascending=True).plot(kind="barh", color="b")
plt.xlabel("Variable Importance")
plt.gca().legend_ = None


# + [markdown] id="z_BYeT1hyZwj"
# Thank you and congratulations! You have succesfully completed the notebook part of the tutorial.
#
# In this notebooks we have illustrated how you can capture your experiments, hyper-parameters, dataset versions and metrics using Vectice SDK. 
# You can now leverage Vectice UI for analysis, documentation and to engage a business conversation around the findings.
#
# Vectice enables you to:
# 1. Make your experiments more reproducible.
# 2. Track the data and code that is used for each experiment and model versions.
# 3. Document your projects' progress and collaborate with your team in Vectice's UI.
# 4. Discover previous work and reuse your team knowledge for new projects.
#
# We are constantly improving the Vectice SDK and the Vectice application. Let us know what improvements you would like to see in the solution and what your favorite features are after completing this tutorial. 
#
# Feel free to explore more and come up with your own ideas on how to best start leveraging Vectice!
#

StarcoderdataJupyterScriptsDedupFiltered