# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "ac72bfac81abf24f9450b9835bc85d07", "grade": false, "grade_id": "cell-06b3a652620668ef", "locked": true, "schema_version": 3, "solution": false, "task": false} id="hahZ4VWATIeZ"
# Lambda School Data Science
#
# *Unit 2, Sprint 2, Module 1*
#
# ---

# + deletable=false editable=false id="o9eSnDYhUGD7" nbgrader={"cell_type": "code", "checksum": "d065a7f431f667bb063b90c023f0dbc1", "grade": false, "grade_id": "cell-e0ed37c9a5d0e6a9", "locked": true, "schema_version": 3, "solution": false, "task": false} outputId="989d6ad6-926b-4af9-f3ae-d8d0508203b4" colab={"base_uri": "https://localhost:8080/"}
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    # !pip install category_encoders==2.*
    # !pip install pandas-profiling==2.*

DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "ff7078bb0a4fb4b482d13864539dc6d8", "grade": false, "grade_id": "cell-85703297fb27b195", "locked": true, "schema_version": 3, "solution": false, "task": false} id="DqJEv3doTIeg"
# # Decision Trees

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "ad8f90528d5c7adfc38d154586deae52", "grade": false, "grade_id": "cell-4b05735817ab984e", "locked": true, "schema_version": 3, "solution": false, "task": false} id="2aIy5ePXTIeh"
# ## Kaggle
#
# **Task 1:** [Sign up for a Kaggle account](https://www.kaggle.com/), if you don’t already have one. Go to our Kaggle InClass competition website (the URL is in Slack). Go to the Rules page. Accept the rules of the competition. Notice that the Rules page also has instructions for the Submission process. The Data page has feature definitions.

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "fd9d50fa771b27245b5fd407ed3ea2e5", "grade": false, "grade_id": "cell-6c43283f79ddc4dc", "locked": true, "schema_version": 3, "solution": false, "task": false} id="3wUZOGkXTIei"
# ## Wrangle Data
#
# **Task 2:** Add to the code below so that `id` is set as the index for `df`.

# + deletable=false nbgrader={"cell_type": "code", "checksum": "a99d8dc738e24ba553771b58c41e4e82", "grade": false, "grade_id": "cell-e2eb2544508f89ef", "locked": false, "schema_version": 3, "solution": true, "task": false} id="h1-Xb1QXTIej" outputId="4f639a9a-fe46-4571-c2b8-67f75a020521" colab={"base_uri": "https://localhost:8080/", "height": 438}
import pandas as pd

df = pd.merge(pd.read_csv(DATA_PATH + 'waterpumps/train_features.csv'),
              pd.read_csv(DATA_PATH + 'waterpumps/train_labels.csv'))

df.set_index('id', drop = True, inplace = True)

df.head()

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "58baf16b195a2c6f4cc23ee09560dbf7", "grade": false, "grade_id": "cell-f09f47f6f4e63cc9", "locked": true, "schema_version": 3, "solution": false, "task": false} id="axzWb4FyTIen"
# **Test 2**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "81daab28521dfd13be1016b273534926", "grade": true, "grade_id": "cell-caa83cb6363d0cd7", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="b1XhX5AnTIen"
'''Task 2 Testing'''

assert isinstance(df, pd.DataFrame), 'Have you created the DataFrame `df`?'
assert df.shape == (59400, 40), '`df` is the wrong shape. Did you set the index to `id`?'
assert 69572 in df.index, 'The index for `df` is has the wrong values. Did you set the index to `id`?'

# + id="OW8qEJIVn3Bj"
df['date_recorded'] = pd.to_datetime(df['date_recorded'],
                                     infer_datetime_format = True)

df['year'] = df['date_recorded'].dt.year

df.drop('date_recorded', axis = 1, inplace = True)

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "26f7a327a6b7a3e2e00173e6f75519bd", "grade": false, "grade_id": "cell-1a84f0ae77611032", "locked": true, "schema_version": 3, "solution": false, "task": false} id="9oViyTmYTIer"
# ## Split Data
#
# **Task 3:** Create your target vector `y` and feature matrix `X`.

# + deletable=false nbgrader={"cell_type": "code", "checksum": "06665db4ef96abbf7f715c4dd554f19b", "grade": false, "grade_id": "cell-c94bceb606d02353", "locked": false, "schema_version": 3, "solution": true, "task": false} id="__4BWt0jTIes"
target = 'status_group'

X = df.drop(target, axis = 1)
y = df[target]

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "0386c0e16323e2419d61f55374d0df05", "grade": false, "grade_id": "cell-48167ea99a6356ae", "locked": true, "schema_version": 3, "solution": false, "task": false} id="ruOEA22DTIev"
# **Test 3**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "5a2b2452b4ad94e5b5a562a185779dff", "grade": true, "grade_id": "cell-d0b6777daf75d805", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="UI7Dv8xBTIew"
'''Task 3 Testing'''

assert isinstance(X, pd.DataFrame), '`X` is the wrong data type.'
assert isinstance(y, pd.Series), '`y` is the wrong data type.'
assert y.shape == (59400,), '`y` is the wrong shape.'
assert X.shape == (59400,39), '`X` is the wrong shape.'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "ad5f005274a47a2acc8d9bb8bc8bbd84", "grade": false, "grade_id": "cell-96e0701792e522b6", "locked": true, "schema_version": 3, "solution": false, "task": false} id="seNZgIvPTIez"
# We already have a test set for this model, the `test.csv` that you'll use to make the predictions you'll upload to Kaggle. However, since our competition only allows for 2 submissions per day, we need a way to estimate our training error so that we can quickly iterate and improve our model. We can achieve this by creating a validation set from the data we have.
#
# **Task 4:** Split `X` and `y` into training and validation sets. Your validation set should be 20% of the data you have. You should have four variables: `X_train`, `X_val`, `y_train`, and `y_val`. 

# + deletable=false nbgrader={"cell_type": "code", "checksum": "4b417b896298dc539f4d351d4f958172", "grade": false, "grade_id": "cell-018b7c0deecc89dc", "locked": false, "schema_version": 3, "solution": true, "task": false} id="pXyv_8dzTIe0"
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, train_size = .8)

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "614aceebebbcb231254f1ae7a842a77e", "grade": false, "grade_id": "cell-ab9ee727cfdc0131", "locked": true, "schema_version": 3, "solution": false, "task": false} id="xM3urmmLTIe5"
# **Test 4**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "7b998b6a51e9a2c17bbd024717a901ab", "grade": true, "grade_id": "cell-b1436df7d5901b26", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="dJ08cK2OTIe6"
'''Task 4 Testing'''

assert X_train.shape == (47520, 39), '`X_train` is the wrong shape.'
assert X_val.shape == (11880, 39), '`X_val` is the wrong shape.'
assert y_train.shape == (47520,), '`y_train` is the wrong shape.'
assert y_val.shape == (11880,), '`y_val` is the wrong shape.'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "e787af946d17a1f823c209349a810942", "grade": false, "grade_id": "cell-4eb573c00f7e3219", "locked": true, "schema_version": 3, "solution": false, "task": false} id="rt7CeFxRTIe9"
# ## Establish Baseline
#
# **Task 5:** This is a **classification** problem, so you need to establish the baseline accuracy for your training set. Find the majority class for `y_train` and calculate the percentage of labels in `y_train` belonging to that class. Assign your answer to the variable name `baseline_acc`.

# + deletable=false nbgrader={"cell_type": "code", "checksum": "6b5e3397d4e5c153e839b6fb53525f76", "grade": false, "grade_id": "cell-c6c63e2b6ff9e101", "locked": false, "schema_version": 3, "solution": true, "task": false} id="4Smvi-kxTIe-" outputId="2403aff9-23da-44c2-9ac9-e7c68ec308ca" colab={"base_uri": "https://localhost:8080/"}
baseline_acc = y_train.value_counts(normalize = True)[0]

print('Baseline Accuracy:', baseline_acc)

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "2d6a544256ca1f163fb33b115da2345e", "grade": false, "grade_id": "cell-cf3ce1fce46a2f72", "locked": true, "schema_version": 3, "solution": false, "task": false} id="-UKx0tsaTIfB"
# **Task 5**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "dad84b7c8b303e546add1509de1a0dd7", "grade": true, "grade_id": "cell-a0c23a1103429de2", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="ARwcYNJtTIfC"
'''Task 5 Testing'''

assert isinstance(baseline_acc, float), '`baseline_acc` should be a `float`.'
assert 0.0 <= baseline_acc <= 1.0, '`baseline_acc` is a score that should be between 0 and 1.'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "8ff10811aa8b551ae8d1f93475d566fc", "grade": false, "grade_id": "cell-10c13b070533d8b5", "locked": true, "schema_version": 3, "solution": false, "task": false} id="Qy-vXX9ETIfF"
# ## Build Model

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "92d0226782f7a5b7b1a41be67a0cf1ae", "grade": false, "grade_id": "cell-9759a1c2167a5dfc", "locked": true, "schema_version": 3, "solution": false, "task": false} id="iAD_GxuSTIfG"
# **Task 6:** Create a model named `model` and train it with your training data. Your model should be a pipeline with (a) transformers that you think are appropriate to this dataset and (b) a `DecisionTreeClassifier` as your predictor. **Tip:** How can you transform categorical features and missing values in order to train your model?

# + deletable=false nbgrader={"cell_type": "code", "checksum": "8c5b2cabd3d5d1adbfad3706bd025cd4", "grade": false, "grade_id": "cell-0c7c4d20b0f0fc70", "locked": false, "schema_version": 3, "solution": true, "task": false} id="DDZYW-SyTIfH" outputId="52891c98-eb20-44e3-f175-27c4b312e642" colab={"base_uri": "https://localhost:8080/"}
from category_encoders import OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import plotly.express as px

# + id="EQ8wcsLhcvAj"
model = make_pipeline(OrdinalEncoder(handle_missing = np.NaN,
                                    return_df = True),
                     SimpleImputer(),

                     DecisionTreeClassifier(min_samples_split = 20,
                                            min_samples_leaf = 10,
                                            max_depth = 5))


X_train_clean = pd.DataFrame(model[:-1].fit_transform(X_train, y_train), columns = X_train.columns)


model[-1:].fit(X_train_clean, y_train);

# + id="sTfKNelykZ2D" outputId="ac9bdbf3-8705-453a-9d4e-9a1ceea79523" colab={"base_uri": "https://localhost:8080/", "height": 542}
feat_imp = pd.Series(model[-1].feature_importances_)
px.bar(x = X_train.columns, y = feat_imp )

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "3be2e5b6636f8de879f37e7b93e17282", "grade": false, "grade_id": "cell-bd1f0e078ca61b7a", "locked": true, "schema_version": 3, "solution": false, "task": false} id="AoLz6Xe6TIfK"
# **Test 6**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "acd5a348298e5f22c5e2dc1c4bc5f3e2", "grade": true, "grade_id": "cell-4edc1a13268269ac", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="rZgCDsUWTIfL"
'''Task 6 Testing'''

assert len(model) > 1, 'Your model pipeline should have multiple steps.'
assert isinstance(model[-1], DecisionTreeClassifier), 'Your pipline should end in a `DecisionTreeClassifier`.'
assert hasattr(model, 'classes_'), 'Have you fit your model?'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "4bde05f99cea7be1f76987db649f3452", "grade": false, "grade_id": "cell-c64ad7ac1b03f6da", "locked": true, "schema_version": 3, "solution": false, "task": false} id="LsoIsQXnTIfO"
# ## Check Metrics
#
# **Task 7:** Calculate the training and validation accuracy of your model, and assign them to the valiables `training_acc` and `validation_acc`, respectively. Your validation accuracy should be greater than your baseline accuracy.

# + id="su8S8toDJnMf" outputId="d32e01eb-34da-4a60-960a-0f5f7584d14e" colab={"base_uri": "https://localhost:8080/", "height": 344}
kbest = SelectKBest(k=10)
kbest.fit_transform(X_train_clean, y_train)
X_train_selected = X_train_clean[X_train_clean.columns[kbest.get_support()]]

X_train_selected.head()

# + id="JvTtZF7BN19x" outputId="fb2722a0-fde4-44c2-f941-a6c87b9e7057" colab={"base_uri": "https://localhost:8080/"}
kbest.pvalues_ , kbest.scores_

# + id="fQ_y1eX7OEYt" outputId="221c63f3-63ce-4759-ab25-8acb71ed0db9" colab={"base_uri": "https://localhost:8080/", "height": 542}
px.scatter(x = kbest.pvalues_, y = kbest.scores_)

# + deletable=false nbgrader={"cell_type": "code", "checksum": "5ad0c7ee9de8fae4b1f309d13a194bbc", "grade": false, "grade_id": "cell-87a17042f6131ba5", "locked": false, "schema_version": 3, "solution": true, "task": false} id="a-4I0bDETIfP" outputId="c8951f06-1b65-4275-b93e-80dfef7fbb13" colab={"base_uri": "https://localhost:8080/"}
X_val_clean = pd.DataFrame(model[:-1].transform(X_val), columns = X_val.columns)

training_acc = model.score(X_train_clean, y_train)
validation_acc = model.score(X_val_clean, y_val)

print('Training Accuracy:', training_acc)
print('Validation Accuracy:', validation_acc)

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "7187c867400846bff1a47be9c11e4651", "grade": false, "grade_id": "cell-8d44b362513262b9", "locked": true, "schema_version": 3, "solution": false, "task": false} id="HRVbXZVMTIfT"
# **Test 7**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "c28add2cc4da7249cd4a9b4e367abb1f", "grade": true, "grade_id": "cell-5b2575d28995d5be", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="cSeRROKCTIfT"
'''Task 7 Testing'''

assert isinstance(training_acc, float)
assert isinstance(validation_acc, float)
assert 0.0 <= training_acc <= 1.0
assert 0.0 <= validation_acc <= 1.0

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "dbe8e21b37bb8bbcf790f6c3bb82409d", "grade": false, "grade_id": "cell-20e570b001622e4d", "locked": true, "schema_version": 3, "solution": false, "task": false} id="LqKyYJ-5TIfX"
# ## Create Kaggle Submission
#
# **Task 8:** Load `'waterpumps/test.csv'` into a DataFrame named `X_test`. Generate a list of predictions, and then put them into a DataFrame `submission`. Be sure that `submission` has the same index as `X_test` and that the column name for your predictions is `'status_group'`.  

# + deletable=false nbgrader={"cell_type": "code", "checksum": "751e42e23ad01c840e677d9061441489", "grade": false, "grade_id": "cell-f449c26c27917323", "locked": false, "schema_version": 3, "solution": true, "task": false} id="K0InMTMITIfY" outputId="0c89da5d-c112-458c-a739-4f1e6de65327" colab={"base_uri": "https://localhost:8080/", "height": 228}
X_test = pd.read_csv(DATA_PATH + 'waterpumps/test_features.csv', index_col='id')


X_test['date_recorded'] = pd.to_datetime(X_test['date_recorded'],
                                     infer_datetime_format = True)
X_test['year'] = X_test['date_recorded'].dt.year

X_test.drop('date_recorded', axis = 1, inplace = True)

X_test_clean = pd.DataFrame(model[:-1].transform(X_test),
                            columns = X_test.columns,
                            index = X_test.index)

submission = pd.DataFrame(model[-1:].predict(X_test_clean),
                          columns = ['status_group'],
                          index = X_test_clean.index)

submission.head()

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "c3ea8f69c1f63c2fc7c6b4dc9f36a4c7", "grade": false, "grade_id": "cell-88bdf757927010d2", "locked": true, "schema_version": 3, "solution": false, "task": false} id="K3bosu8_TIfb"
# **Test 8**

# + deletable=false editable=false nbgrader={"cell_type": "code", "checksum": "de1de506bac0c572293dcf81b592fda3", "grade": true, "grade_id": "cell-7184784b1c67a971", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false} id="rdOmQQjBTIfd"
'''Task 8 Testing'''

assert isinstance(submission, pd.DataFrame), '`submission` should be a DataFrame.'
assert len(submission) == 14358, '`submission` should have 14358 rows.'

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "9e1dd2ff80244929b807a600180c8db5", "grade": false, "grade_id": "cell-7501482e73c658e0", "locked": true, "schema_version": 3, "solution": false, "task": false} id="FibHfMGpTIfg"
# **Task 9 (`stretch goal`):** Save `submission` as a csv file using [`.to_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) and submit it to our Kaggle competition.

# + deletable=false nbgrader={"cell_type": "code", "checksum": "53ceb2bae5f41f56eba8f65b2b56c38c", "grade": false, "grade_id": "cell-6a87d8169b9c48ea", "locked": false, "schema_version": 3, "solution": true, "task": false} id="sB4-GxfGTIfh"
submission.to_csv('submission.csv')

# + [markdown] deletable=false editable=false nbgrader={"cell_type": "markdown", "checksum": "cba7b6461310b5fab62266e978074a78", "grade": false, "grade_id": "cell-591ad5292d4e1ee1", "locked": true, "schema_version": 3, "solution": false, "task": false} id="19DRI7fOTIfk"
# ## Explain
#
# **Task 10 (`stretch goal`):** Make a horizontal barchart of the 10 most important features for your model.

# + id="mYuOlymRcD8U"


StarcoderdataJupyterScriptsDedupFiltered