# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/will-fong/data-science/blob/master/projects/lending-club/03_Keras_Project_Exercise-WF.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="S4VSno6_ruVK" colab_type="text"
# <a href="https://www.pieriandata.com"><img src="https://github.com/will-fong/data-science/blob/Pierian_Data_Logo.PNG?raw=1"></a>
# <strong><center>Copyright by Pierian Data Inc.</center></strong> 
# <strong><center>Created by <NAME>.</center></strong>

# + [markdown] id="BWFaWY05ruVM" colab_type="text"
# # Keras API Project Exercise
#
# ## The Data
#
# We will be using a subset of the LendingClub DataSet obtained from Kaggle: https://www.kaggle.com/wordsforthewise/lending-club
#
# ## NOTE: Do not download the full zip from the link! We provide a special version of this file that has some extra feature engineering for you to do. You won't be able to follow along with the original file!
#
# LendingClub is a US peer-to-peer lending company, headquartered in San Francisco, California.[3] It was the first peer-to-peer lender to register its offerings as securities with the Securities and Exchange Commission (SEC), and to offer loan trading on a secondary market. LendingClub is the world's largest peer-to-peer lending platform.
#
# ### Our Goal
#
# Given historical data on loans given out with information on whether or not the borrower defaulted (charge-off), can we build a model thatcan predict wether or nor a borrower will pay back their loan? This way in the future when we get a new potential customer we can assess whether or not they are likely to pay back the loan. Keep in mind classification metrics when evaluating the performance of your model!
#
# The "loan_status" column contains our label.
#
# ### Data Overview

# + [markdown] id="kEjL3bCqruVO" colab_type="text"
# ----
# -----
# There are many LendingClub data sets on Kaggle. Here is the information on this particular data set:
#
# <table border="1" class="dataframe">
#   <thead>
#     <tr style="text-align: right;">
#       <th></th>
#       <th>LoanStatNew</th>
#       <th>Description</th>
#     </tr>
#   </thead>
#   <tbody>
#     <tr>
#       <th>0</th>
#       <td>loan_amnt</td>
#       <td>The listed amount of the loan applied for by the borrower. If at some point in time, the credit department reduces the loan amount, then it will be reflected in this value.</td>
#     </tr>
#     <tr>
#       <th>1</th>
#       <td>term</td>
#       <td>The number of payments on the loan. Values are in months and can be either 36 or 60.</td>
#     </tr>
#     <tr>
#       <th>2</th>
#       <td>int_rate</td>
#       <td>Interest Rate on the loan</td>
#     </tr>
#     <tr>
#       <th>3</th>
#       <td>installment</td>
#       <td>The monthly payment owed by the borrower if the loan originates.</td>
#     </tr>
#     <tr>
#       <th>4</th>
#       <td>grade</td>
#       <td>LC assigned loan grade</td>
#     </tr>
#     <tr>
#       <th>5</th>
#       <td>sub_grade</td>
#       <td>LC assigned loan subgrade</td>
#     </tr>
#     <tr>
#       <th>6</th>
#       <td>emp_title</td>
#       <td>The job title supplied by the Borrower when applying for the loan.*</td>
#     </tr>
#     <tr>
#       <th>7</th>
#       <td>emp_length</td>
#       <td>Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.</td>
#     </tr>
#     <tr>
#       <th>8</th>
#       <td>home_ownership</td>
#       <td>The home ownership status provided by the borrower during registration or obtained from the credit report. Our values are: RENT, OWN, MORTGAGE, OTHER</td>
#     </tr>
#     <tr>
#       <th>9</th>
#       <td>annual_inc</td>
#       <td>The self-reported annual income provided by the borrower during registration.</td>
#     </tr>
#     <tr>
#       <th>10</th>
#       <td>verification_status</td>
#       <td>Indicates if income was verified by LC, not verified, or if the income source was verified</td>
#     </tr>
#     <tr>
#       <th>11</th>
#       <td>issue_d</td>
#       <td>The month which the loan was funded</td>
#     </tr>
#     <tr>
#       <th>12</th>
#       <td>loan_status</td>
#       <td>Current status of the loan</td>
#     </tr>
#     <tr>
#       <th>13</th>
#       <td>purpose</td>
#       <td>A category provided by the borrower for the loan request.</td>
#     </tr>
#     <tr>
#       <th>14</th>
#       <td>title</td>
#       <td>The loan title provided by the borrower</td>
#     </tr>
#     <tr>
#       <th>15</th>
#       <td>zip_code</td>
#       <td>The first 3 numbers of the zip code provided by the borrower in the loan application.</td>
#     </tr>
#     <tr>
#       <th>16</th>
#       <td>addr_state</td>
#       <td>The state provided by the borrower in the loan application</td>
#     </tr>
#     <tr>
#       <th>17</th>
#       <td>dti</td>
#       <td>A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.</td>
#     </tr>
#     <tr>
#       <th>18</th>
#       <td>earliest_cr_line</td>
#       <td>The month the borrower's earliest reported credit line was opened</td>
#     </tr>
#     <tr>
#       <th>19</th>
#       <td>open_acc</td>
#       <td>The number of open credit lines in the borrower's credit file.</td>
#     </tr>
#     <tr>
#       <th>20</th>
#       <td>pub_rec</td>
#       <td>Number of derogatory public records</td>
#     </tr>
#     <tr>
#       <th>21</th>
#       <td>revol_bal</td>
#       <td>Total credit revolving balance</td>
#     </tr>
#     <tr>
#       <th>22</th>
#       <td>revol_util</td>
#       <td>Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.</td>
#     </tr>
#     <tr>
#       <th>23</th>
#       <td>total_acc</td>
#       <td>The total number of credit lines currently in the borrower's credit file</td>
#     </tr>
#     <tr>
#       <th>24</th>
#       <td>initial_list_status</td>
#       <td>The initial listing status of the loan. Possible values are – W, F</td>
#     </tr>
#     <tr>
#       <th>25</th>
#       <td>application_type</td>
#       <td>Indicates whether the loan is an individual application or a joint application with two co-borrowers</td>
#     </tr>
#     <tr>
#       <th>26</th>
#       <td>mort_acc</td>
#       <td>Number of mortgage accounts.</td>
#     </tr>
#     <tr>
#       <th>27</th>
#       <td>pub_rec_bankruptcies</td>
#       <td>Number of public record bankruptcies</td>
#     </tr>
#   </tbody>
# </table>
#
# ---
# ----

# + [markdown] id="eglOhCZmruVR" colab_type="text"
# ## Starter Code
#
# #### Note: We also provide feature information on the data as a .csv file for easy lookup throughout the notebook:

# + id="MQJjYWkbruVS" colab_type="code" colab={}
import pandas as pd

# + id="VNXeLt8UruVb" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 395} outputId="24dfbf5e-3f9a-4dd8-b6b5-da37a40b6e67"
data_info = pd.read_csv('../DATA/lending_club_info.csv',index_col='LoanStatNew')

# + id="S2DKjzyaruVh" colab_type="code" colab={}
print(data_info.loc['revol_util']['Description'])


# + id="TzgVifg7ruVo" colab_type="code" colab={}
def feat_info(col_name):
    print(data_info.loc[col_name]['Description'])


# + id="C0yKPSR8ruVt" colab_type="code" outputId="547e106e-2d7c-4607-827f-ec0183515ca6" colab={}
feat_info('mort_acc')

# + [markdown] id="41PeDpARruV0" colab_type="text"
# ## Loading the data and other imports

# + id="8bU6Jo8iruV0" colab_type="code" colab={}
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# might be needed depending on your version of Jupyter
# %matplotlib inline

# + id="OfZZytY6ruV4" colab_type="code" colab={}
df = pd.read_csv('../DATA/lending_club_loan_two.csv')

# + id="Zvt9TuSpruV9" colab_type="code" outputId="a620ef79-7465-4b6a-e978-ef90f6a10b00" colab={}
df.info()

# + [markdown] id="DicL1VzbruWD" colab_type="text"
# # Project Tasks
#
# **Complete the tasks below! Keep in mind is usually more than one way to complete the task! Enjoy**
#
# -----
# ------
#
# # Section 1: Exploratory Data Analysis
#
# **OVERALL GOAL: Get an understanding for which variables are important, view summary statistics, and visualize the data**
#
#
# ----

# + [markdown] id="X1CWg_LQruWE" colab_type="text"
# **TASK: Since we will be attempting to predict loan_status, create a countplot as shown below.**

# + id="yTSkfAwuruWF" colab_type="code" colab={}
# CODE HERE

# + id="g2xDK4yHruWI" colab_type="code" outputId="84ebe599-c72a-46a3-f623-ff36ad4365ab" colab={}


# + [markdown] id="QW38YecOruWN" colab_type="text"
# **TASK: Create a histogram of the loan_amnt column.**

# + id="qQmwquG_ruWO" colab_type="code" colab={}
# CODE HERE

# + id="q4O0xj5XruWU" colab_type="code" outputId="dab6b998-8e49-4763-a8bf-5393a64a41a1" colab={}


# + [markdown] id="O5K3xoeSruWZ" colab_type="text"
# **TASK: Let's explore correlation between the continuous feature variables. Calculate the correlation between all continuous numeric variables using .corr() method.**

# + id="BSUyqCu8ruWa" colab_type="code" colab={}
# CODE HERE

# + id="L2NFZCr9ruWe" colab_type="code" outputId="759f96d2-ef6e-40a8-c7fa-f98ddf440ed7" colab={}


# + [markdown] id="WXznimSXruWk" colab_type="text"
# **TASK: Visualize this using a heatmap. Depending on your version of matplotlib, you may need to manually adjust the heatmap.**
#
# * [Heatmap info](https://seaborn.pydata.org/generated/seaborn.heatmap.html#seaborn.heatmap)
# * [Help with resizing](https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot)

# + id="4ZFz2miBruWk" colab_type="code" colab={}
# CODE HERE

# + id="HaLym0UFruWn" colab_type="code" outputId="32869dab-020d-4d25-a690-eb27c4719b26" colab={}


# + [markdown] id="9y44XdJwruWr" colab_type="text"
# **TASK: You should have noticed almost perfect correlation with the "installment" feature. Explore this feature further. Print out their descriptions and perform a scatterplot between them. Does this relationship make sense to you? Do you think there is duplicate information here?**

# + id="n71kzxQHruWs" colab_type="code" colab={}
# CODE HERE

# + id="pqkM4nSRruWv" colab_type="code" outputId="26f27ea6-2800-4977-f952-b6cbc52040b8" colab={}


# + id="9wtb1SBoruW0" colab_type="code" outputId="0555d74f-33f2-4e80-b6a0-44c8431fcae0" colab={}


# + id="QF2wlzneruW6" colab_type="code" outputId="182e1368-5fe3-47c8-eb61-0fb99bef8836" colab={}


# + [markdown] id="H3dfNhm9ruXD" colab_type="text"
# **TASK: Create a boxplot showing the relationship between the loan_status and the Loan Amount.**

# + id="eETfF1V_ruXD" colab_type="code" colab={}
# CODE HERE

# + id="9Igk9LEOruXI" colab_type="code" outputId="c54055d1-8499-4bb4-c00a-9adc82637017" colab={}


# + [markdown] id="OBqM-xLCruXM" colab_type="text"
# **TASK: Calculate the summary statistics for the loan amount, grouped by the loan_status.**

# + id="opu0SQyBruXN" colab_type="code" colab={}
# CODE HERE

# + id="I0ZQR80yruXP" colab_type="code" outputId="584d1e5c-8306-4cb8-cd81-e88cb4bd356b" colab={}


# + [markdown] id="hv4lCKBhruXT" colab_type="text"
# **TASK: Let's explore the Grade and SubGrade columns that LendingClub attributes to the loans. What are the unique possible grades and subgrades?**

# + id="qpzh85pfruXU" colab_type="code" colab={}
# CODE HERE

# + id="v-51hfDlruXW" colab_type="code" outputId="4140c8d9-5c12-4c05-d1fd-5352e26bfaa5" colab={}


# + id="1XfzdQvPruXa" colab_type="code" outputId="e1e5bac0-d8ef-4d8d-c7be-953989e7ed94" colab={}


# + [markdown] id="kBFDnwcgruXc" colab_type="text"
# **TASK: Create a countplot per grade. Set the hue to the loan_status label.**

# + id="OztnlezyruXc" colab_type="code" colab={}
# CODE HERE

# + id="VCkPgzuhruXg" colab_type="code" outputId="8192d8da-b1ce-4dd5-f8a1-50a56b2d8a0d" colab={}


# + [markdown] id="TlGnH7ooruXj" colab_type="text"
# **TASK: Display a count plot per subgrade. You may need to resize for this plot and [reorder](https://seaborn.pydata.org/generated/seaborn.countplot.html#seaborn.countplot) the x axis. Feel free to edit the color palette. Explore both all loans made per subgrade as well being separated based on the loan_status. After creating this plot, go ahead and create a similar plot, but set hue="loan_status"**

# + id="zqoG0Rj_ruXj" colab_type="code" colab={}
#CODE HERE

# + id="DbtWXXtXruXl" colab_type="code" outputId="68d8a587-5302-4c26-dd90-5c22d5d90368" colab={}


# + id="db_yv3IWruXo" colab_type="code" colab={}
# CODE HERE

# + id="6QBrcecdruXs" colab_type="code" outputId="314ea453-e212-4bdf-a586-6152d7975a18" colab={}


# + [markdown] id="upSmrtczruXu" colab_type="text"
# **TASK: It looks like F and G subgrades don't get paid back that often. Isloate those and recreate the countplot just for those subgrades.**

# + id="01v0cd6qruXu" colab_type="code" colab={}
# CODE HERE

# + id="tSoEbKUDruXx" colab_type="code" outputId="00641eb2-d914-4c68-a62f-10d8b96090dd" colab={}


# + [markdown] id="4SuU_FIiruX0" colab_type="text"
# **TASK: Create a new column called 'loan_repaid' which will contain a 1 if the loan status was "Fully Paid" and a 0 if it was "Charged Off".**

# + id="hXtley40ruX1" colab_type="code" colab={}
# CODE HERE

# + id="EqI8D5saruX4" colab_type="code" outputId="cbb2c1f4-45f1-4ad3-b389-5ff4f9f54bd4" colab={}


# + id="Sv03EgPYruX5" colab_type="code" colab={}


# + id="rA4I_s30ruX7" colab_type="code" outputId="347c83cc-204c-42c9-ac85-fa4041a81da9" colab={}


# + [markdown] id="HRdqnv6CruYA" colab_type="text"
# **CHALLENGE TASK: (Note this is hard, but can be done in one line!) Create a bar plot showing the correlation of the numeric features to the new loan_repaid column. [Helpful Link](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.bar.html)**

# + id="6xu8eJC9ruYA" colab_type="code" colab={}
#CODE HERE

# + id="Z9XUlB5FruYD" colab_type="code" outputId="753aea05-80c1-46ec-8508-e45156bb5269" colab={}


# + [markdown] id="M9J0Zsw9ruYH" colab_type="text"
# ---
# ---
# # Section 2: Data PreProcessing
#
# **Section Goals: Remove or fill any missing data. Remove unnecessary or repetitive features. Convert categorical string features to dummy variables.**
#
#

# + id="8huk8-EEruYH" colab_type="code" outputId="62883104-26c6-4125-dad9-4b6c3028d12e" colab={}


# + [markdown] id="HZLzTbLgruYK" colab_type="text"
# # Missing Data
#
# **Let's explore this missing data columns. We use a variety of factors to decide whether or not they would be useful, to see if we should keep, discard, or fill in the missing data.**

# + [markdown] id="w52Tn7-fruYL" colab_type="text"
# **TASK: What is the length of the dataframe?**

# + id="x-E_GJGJruYL" colab_type="code" colab={}
# CODE HERE

# + id="cwFVLMl_ruYN" colab_type="code" outputId="c379a2fc-74b1-452b-aa68-a729b7774862" colab={}


# + [markdown] id="JAcec4OwruYQ" colab_type="text"
# **TASK: Create a Series that displays the total count of missing values per column.**

# + id="SxAQMCnUruYQ" colab_type="code" colab={}
# CODE HERE

# + id="VYSpHjgwruYS" colab_type="code" outputId="1cc83db9-e365-4b06-e3f1-ce9cadf84fba" colab={}


# + [markdown] id="zm9ILPlCruYX" colab_type="text"
# **TASK: Convert this Series to be in term of percentage of the total DataFrame**

# + id="WkDoJ7QcruYY" colab_type="code" colab={}
# CODE HERE

# + id="OGauL_DyruYa" colab_type="code" outputId="51f3559a-ac94-4b71-98fe-792d5a326166" colab={}


# + [markdown] id="aZeKh4_zruYe" colab_type="text"
# **TASK: Let's examine emp_title and emp_length to see whether it will be okay to drop them. Print out their feature information using the feat_info() function from the top of this notebook.**

# + id="hzwCc9w1ruYe" colab_type="code" colab={}
# CODE HERE

# + id="QvnGwqHFruYg" colab_type="code" outputId="d71c67a3-310f-4646-d432-f0f387533148" colab={}


# + [markdown] id="8FDzsM8IruYk" colab_type="text"
# **TASK: How many unique employment job titles are there?**

# + id="KS39nFJFruYk" colab_type="code" colab={}
# CODE HERE

# + id="qlRzo7yjruYn" colab_type="code" outputId="0ad40694-a005-4630-9828-a2b41c0765fb" colab={}


# + id="JpkwNBfLruYp" colab_type="code" outputId="bca78f29-e33b-4f4f-eb64-e55bff7d0b11" colab={}


# + [markdown] id="qTgxqwAEruYr" colab_type="text"
# **TASK: Realistically there are too many unique job titles to try to convert this to a dummy variable feature. Let's remove that emp_title column.**

# + id="4z2XTLJzruYs" colab_type="code" colab={}
# CODE HERE

# + id="gz2QnIt_ruYu" colab_type="code" colab={}


# + [markdown] id="q2sWUd7druYv" colab_type="text"
# **TASK: Create a count plot of the emp_length feature column. Challenge: Sort the order of the values.**

# + id="GOC4eIRpruYw" colab_type="code" colab={}
# CODE HERE

# + id="AMHnnlJwruYy" colab_type="code" outputId="0962f133-eb95-41c9-cf83-f2be4a973a89" colab={}


# + id="tYgZMpmPruY1" colab_type="code" colab={}


# + id="zgVgTJY5ruY5" colab_type="code" outputId="a9cf1ec5-abc9-4966-bbb0-355523cabcd6" colab={}


# + [markdown] id="BZHbV6SsruY7" colab_type="text"
# **TASK: Plot out the countplot with a hue separating Fully Paid vs Charged Off**

# + id="dRd2oAkAruY7" colab_type="code" colab={}
# CODE HERE

# + id="sxN2RpalruY9" colab_type="code" outputId="17f78132-99b0-47af-b8fa-1034ffd0b86c" colab={}


# + [markdown] id="B4TRodxGruZA" colab_type="text"
# **CHALLENGE TASK: This still doesn't really inform us if there is a strong relationship between employment length and being charged off, what we want is the percentage of charge offs per category. Essentially informing us what percent of people per employment category didn't pay back their loan. There are a multitude of ways to create this Series. Once you've created it, see if visualize it with a [bar plot](https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.plot.html). This may be tricky, refer to solutions if you get stuck on creating this Series.**

# + id="Jd989cjSruZA" colab_type="code" colab={}
# CODE HERE

# + id="Ob6Xt5-XruZD" colab_type="code" colab={}


# + id="wS28qcaGruZE" colab_type="code" colab={}


# + id="3XW76QotruZI" colab_type="code" colab={}


# + id="LH3cdxQhruZN" colab_type="code" outputId="e459ea42-6c86-4279-cb95-b28295fee3ce" colab={}


# + id="DQYJWcmUruZO" colab_type="code" outputId="2762b68c-3c4c-43de-f538-d458fec649e6" colab={}


# + [markdown] id="rTqla-drruZR" colab_type="text"
# **TASK: Charge off rates are extremely similar across all employment lengths. Go ahead and drop the emp_length column.**

# + id="yJUB8nDHruZR" colab_type="code" colab={}
# CODE HERE

# + id="H3aGfpTPruZT" colab_type="code" colab={}


# + [markdown] id="ql4Uh3EmruZV" colab_type="text"
# **TASK: Revisit the DataFrame to see what feature columns still have missing data.**

# + id="V9EAEmhtruZV" colab_type="code" colab={}


# + id="rrjKOgjgruZX" colab_type="code" outputId="80aeadda-a373-4ad0-f731-e087f10eea71" colab={}


# + [markdown] id="mAKB6VMLruZZ" colab_type="text"
# **TASK: Review the title column vs the purpose column. Is this repeated information?**

# + id="O7XHCgbEruZZ" colab_type="code" colab={}
# CODE HERE

# + id="BuQwNLf7ruZb" colab_type="code" outputId="963a301f-1196-4a51-f06d-5d00f9d87259" colab={}


# + id="B15k28WQruZe" colab_type="code" outputId="66b45581-0f75-4f6e-fdf9-4aa7e75a3d8d" colab={}
df['title'].head(10)

# + [markdown] id="ECatmWbtruZf" colab_type="text"
# **TASK: The title column is simply a string subcategory/description of the purpose column. Go ahead and drop the title column.**

# + id="vTqF61L0ruZg" colab_type="code" colab={}
# CODE HERE

# + id="5i2K0xhWruZt" colab_type="code" colab={}


# + [markdown] id="hxuYsA8PruZv" colab_type="text"
# ---
# **NOTE: This is one of the hardest parts of the project! Refer to the solutions video if you need guidance, feel free to fill or drop the missing values of the mort_acc however you see fit! Here we're going with a very specific approach.**
#
#
# ---
# **TASK: Find out what the mort_acc feature represents**

# + id="gq6b9htrruZw" colab_type="code" colab={}
# CODE HERE

# + id="WpxH2TqqruZx" colab_type="code" outputId="37745c28-c498-4ab2-d86b-cbc8ab10e54c" colab={}


# + [markdown] id="9TemkagbruZz" colab_type="text"
# **TASK: Create a value_counts of the mort_acc column.**

# + id="IjgHtjSWruZz" colab_type="code" colab={}
# CODE HERE

# + id="wwIjWzvjruZ1" colab_type="code" outputId="1d726496-c6b3-425a-817c-d75889c6ac22" colab={}


# + [markdown] id="udPhD6B-ruZ2" colab_type="text"
# **TASK: There are many ways we could deal with this missing data. We could attempt to build a simple model to fill it in, such as a linear model, we could just fill it in based on the mean of the other columns, or you could even bin the columns into categories and then set NaN as its own category. There is no 100% correct approach! Let's review the other columsn to see which most highly correlates to mort_acc**

# + id="WX8lNDEcruZ3" colab_type="code" colab={}


# + id="nyqRbkGYruZ5" colab_type="code" outputId="02dd790d-b258-4cd9-e59f-45c86e0d49c7" colab={}


# + [markdown] id="tzDORezmruZ7" colab_type="text"
# **TASK: Looks like the total_acc feature correlates with the mort_acc , this makes sense! Let's try this fillna() approach. We will group the dataframe by the total_acc and calculate the mean value for the mort_acc per total_acc entry. To get the result below:**

# + id="au6TG-fAruZ7" colab_type="code" colab={}


# + id="fOMXmBa_ruZ9" colab_type="code" outputId="a194510e-3b98-40c7-eece-436603255e54" colab={}


# + [markdown] id="6UqRLd4HruaC" colab_type="text"
# **CHALLENGE TASK: Let's fill in the missing mort_acc values based on their total_acc value. If the mort_acc is missing, then we will fill in that missing value with the mean value corresponding to its total_acc value from the Series we created above. This involves using an .apply() method with two columns. Check out the link below for more info, or review the solutions video/notebook.**
#
# [Helpful Link](https://stackoverflow.com/questions/13331698/how-to-apply-a-function-to-two-columns-of-pandas-dataframe) 

# + id="zBG6v0EAruaC" colab_type="code" colab={}
# CODE HERE

# + id="cNsJXv-eruaE" colab_type="code" colab={}


# + id="0AswFf9bruaF" colab_type="code" outputId="002e75f2-7069-40f0-9169-dfc62c8be012" colab={}


# + id="ANM8eLAbruaH" colab_type="code" colab={}


# + id="RfKHjTnxruaI" colab_type="code" colab={}


# + id="6vEVJL1eruaK" colab_type="code" outputId="a5392e86-e82f-4fa5-fd6c-5d742a1187ea" colab={}


# + [markdown] id="mN-dp-8sruaM" colab_type="text"
# **TASK: revol_util and the pub_rec_bankruptcies have missing data points, but they account for less than 0.5% of the total data. Go ahead and remove the rows that are missing those values in those columns with dropna().**

# + id="uO5n2QhdruaN" colab_type="code" colab={}
# CODE HERE

# + id="W52-TFFBruaN" colab_type="code" colab={}


# + id="78vEyEKpruaP" colab_type="code" outputId="913aeec6-422e-460e-87cf-92641e595087" colab={}


# + [markdown] id="avPBsCTqruaR" colab_type="text"
# ## Categorical Variables and Dummy Variables
#
# **We're done working with the missing data! Now we just need to deal with the string values due to the categorical columns.**
#
# **TASK: List all the columns that are currently non-numeric. [Helpful Link](https://stackoverflow.com/questions/22470690/get-list-of-pandas-dataframe-columns-based-on-data-type)**
#
# [Another very useful method call](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html)

# + id="rZDcS0clruaR" colab_type="code" colab={}
# CODE HERE

# + id="nj6zRz_rruaS" colab_type="code" outputId="b39c59f4-b0e0-4993-e560-d2eea4ef3a84" colab={}


# + [markdown] id="-WIKx6ForuaT" colab_type="text"
# ---
# **Let's now go through all the string features to see what we should do with them.**
#
# ---
#
#
# ### term feature
#
# **TASK: Convert the term feature into either a 36 or 60 integer numeric data type using .apply() or .map().**

# + id="TYgXcl4eruaT" colab_type="code" colab={}
# CODE HERE

# + id="fXIaprwqruaV" colab_type="code" outputId="61f22968-93a3-46e4-814d-b6d8e31b40d7" colab={}


# + id="HCeOrFCFruaX" colab_type="code" colab={}


# + [markdown] id="BnLwuz9AruaY" colab_type="text"
# ### grade feature
#
# **TASK: We already know grade is part of sub_grade, so just drop the grade feature.**

# + id="aMm63a4HruaY" colab_type="code" colab={}
# CODE HERE

# + id="8euXM37xruaZ" colab_type="code" colab={}


# + [markdown] id="_K1PcUKkruab" colab_type="text"
# **TASK: Convert the subgrade into dummy variables. Then concatenate these new columns to the original dataframe. Remember to drop the original subgrade column and to add drop_first=True to your get_dummies call.**

# + id="YlnUovQNruac" colab_type="code" colab={}
# CODE HERE

# + id="er4e8CCYruag" colab_type="code" colab={}


# + id="GyQ8bLkHruag" colab_type="code" colab={}


# + id="aIHd2dTvruai" colab_type="code" outputId="a02b9e50-dfac-4758-8b73-302e2558eb1d" colab={}


# + id="hWGoCa7Nruaj" colab_type="code" outputId="0f12175b-d2e4-4ba9-cb7f-175112135673" colab={}


# + [markdown] id="N9Ak-g00rual" colab_type="text"
# ### verification_status, application_type,initial_list_status,purpose 
# **TASK: Convert these columns: ['verification_status', 'application_type','initial_list_status','purpose'] into dummy variables and concatenate them with the original dataframe. Remember to set drop_first=True and to drop the original columns.**

# + id="USmttpANrual" colab_type="code" colab={}
# CODE HERE

# + id="yB5Ec6_iruam" colab_type="code" colab={}


# + id="MxzhoO17ruao" colab_type="code" colab={}


# + [markdown] id="ooa-MrMJruap" colab_type="text"
# ### home_ownership
# **TASK:Review the value_counts for the home_ownership column.**

# + id="2FK1GcgUruap" colab_type="code" colab={}
#CODE HERE

# + id="JefNiWr7ruar" colab_type="code" outputId="ea6ce756-14be-473f-b55d-58bcd40721ac" colab={}


# + [markdown] id="_B20Glivruas" colab_type="text"
# **TASK: Convert these to dummy variables, but [replace](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.replace.html) NONE and ANY with OTHER, so that we end up with just 4 categories, MORTGAGE, RENT, OWN, OTHER. Then concatenate them with the original dataframe. Remember to set drop_first=True and to drop the original columns.**

# + id="LC7mOfefruas" colab_type="code" colab={}
#CODE HERE

# + id="gv14bJ8lruau" colab_type="code" colab={}


# + [markdown] id="MgHsCugYruav" colab_type="text"
# ### address
# **TASK: Let's feature engineer a zip code column from the address in the data set. Create a column called 'zip_code' that extracts the zip code from the address column.**

# + id="SoxoNrK9ruav" colab_type="code" colab={}
#CODE HERE

# + id="wkmSSXM7ruax" colab_type="code" colab={}


# + [markdown] id="Il-apsyzruax" colab_type="text"
# **TASK: Now make this zip_code column into dummy variables using pandas. Concatenate the result and drop the original zip_code column along with dropping the address column.**

# + id="4EP5kCq6ruay" colab_type="code" colab={}


# + id="5Q0aIdPPruaz" colab_type="code" colab={}


# + [markdown] id="f1sKsZNQrua1" colab_type="text"
# ### issue_d 
#
# **TASK: This would be data leakage, we wouldn't know beforehand whether or not a loan would be issued when using our model, so in theory we wouldn't have an issue_date, drop this feature.**

# + id="zoTX2TcOrua1" colab_type="code" colab={}
#CODE HERE

# + id="2Bw49nsXrua2" colab_type="code" colab={}


# + [markdown] id="WgTPRL2irua3" colab_type="text"
# ### earliest_cr_line
# **TASK: This appears to be a historical time stamp feature. Extract the year from this feature using a .apply function, then convert it to a numeric feature. Set this new data to a feature column called 'earliest_cr_year'.Then drop the earliest_cr_line feature.**

# + id="yKabqU0srua3" colab_type="code" colab={}
#CODE HERE

# + id="TLUyRfCRrua4" colab_type="code" colab={}


# + id="5g8pmJorrua6" colab_type="code" outputId="79c88023-c0ce-4883-8b91-7fbb32e050e1" colab={}


# + [markdown] id="CLB65ibKrua7" colab_type="text"
# ## Train Test Split

# + [markdown] id="EkQ3WYsIrua7" colab_type="text"
# **TASK: Import train_test_split from sklearn.**

# + id="gb9wks3Orua8" colab_type="code" colab={}


# + [markdown] id="CJ2RAh9grua9" colab_type="text"
# **TASK: drop the load_status column we created earlier, since its a duplicate of the loan_repaid column. We'll use the loan_repaid column since its already in 0s and 1s.**

# + id="gzy9qbNZrua9" colab_type="code" colab={}
# CODE HERE

# + id="iMN5ZbIsrua_" colab_type="code" colab={}


# + [markdown] id="i7QH8dK_rubA" colab_type="text"
# **TASK: Set X and y variables to the .values of the features and label.**

# + id="8ck6442irubB" colab_type="code" colab={}
#CODE HERE

# + id="QIfP10zprubC" colab_type="code" colab={}


# + [markdown] id="zBbsYzalrubE" colab_type="text"
# ----
# ----
#
# # OPTIONAL
#
# ## Grabbing a Sample for Training Time
#
# ### OPTIONAL: Use .sample() to grab a sample of the 490k+ entries to save time on training. Highly recommended for lower RAM computers or if you are not using GPU.
#
# ----
# ----

# + id="I3nBRioArubE" colab_type="code" outputId="df8cbe49-2138-4b14-d9cb-1ba37b0582d8" colab={}
# df = df.sample(frac=0.1,random_state=101)
print(len(df))

# + [markdown] id="Sqbn47vvrubF" colab_type="text"
# **TASK: Perform a train/test split with test_size=0.2 and a random_state of 101.**

# + id="RVRjIq_KrubF" colab_type="code" colab={}
#CODE HERE

# + id="GkIQOMbvrubG" colab_type="code" colab={}


# + [markdown] id="7A_aY9JKrubH" colab_type="text"
# ## Normalizing the Data
#
# **TASK: Use a MinMaxScaler to normalize the feature data X_train and X_test. Recall we don't want data leakge from the test set so we only fit on the X_train data.**

# + id="HiCMy9sqrubI" colab_type="code" colab={}
# CODE HERE

# + id="sEzY0QlprubJ" colab_type="code" colab={}


# + id="f21RIzZtrubW" colab_type="code" colab={}


# + id="9cZek6WWrubW" colab_type="code" colab={}


# + id="QWDA1umCrubY" colab_type="code" colab={}


# + [markdown] id="BTitkX7urubb" colab_type="text"
# # Creating the Model
#
# **TASK: Run the cell below to import the necessary Keras functions.**

# + id="bhgwG_n7rubc" colab_type="code" colab={}
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Dropout

# + [markdown] id="JuG6WmD_rube" colab_type="text"
# **TASK: Build a sequential model to will be trained on the data. You have unlimited options here, but here is what the solution uses: a model that goes 78 --> 39 --> 19--> 1 output neuron. OPTIONAL: Explore adding [Dropout layers](https://keras.io/layers/core/) [1](https://en.wikipedia.org/wiki/Dropout_(neural_networks)) [2](https://towardsdatascience.com/machine-learning-part-20-dropout-keras-layers-explained-8c9f6dc4c9ab)**

# + id="RLoESQwIrube" colab_type="code" colab={}
# CODE HERE
model = Sequential()

# Choose whatever number of layers/neurons you want.

# https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw

# Remember to compile()

# + id="Q7RNPZSHrubg" colab_type="code" colab={}


# + [markdown] id="9RGwFHH9rubh" colab_type="text"
# **TASK: Fit the model to the training data for at least 25 epochs. Also add in the validation data for later plotting. Optional: add in a batch_size of 256.**

# + id="kThoChbJrubh" colab_type="code" colab={}
# CODE HERE

# + id="E5b1jV00rubj" colab_type="code" outputId="c01309df-560c-4095-8608-7e4ac9cf3d3f" colab={}


# + [markdown] id="PsZi7i-zrubk" colab_type="text"
# **TASK: OPTIONAL: Save your model.**

# + id="PSe1tskRrubk" colab_type="code" colab={}
# CODE HERE

# + id="1rwac_RNrubl" colab_type="code" colab={}


# + id="-pcZYp-prubn" colab_type="code" colab={}


# + [markdown] id="kcBeIUfXrubn" colab_type="text"
# # Section 3: Evaluating Model Performance.
#
# **TASK: Plot out the validation loss versus the training loss.**

# + id="j0b7jxsNrubn" colab_type="code" colab={}
# CODE HERE

# + id="YCBEAnLZrubp" colab_type="code" colab={}


# + id="jDE76Z_Prubr" colab_type="code" outputId="f8a44806-8ada-46b5-f302-0ed8b5297874" colab={}


# + [markdown] id="96Tyjz55rubt" colab_type="text"
# **TASK: Create predictions from the X_test set and display a classification report and confusion matrix for the X_test set.**

# + id="uGjKQG0erubt" colab_type="code" colab={}
# CODE HERE

# + id="Y2ceDyharubv" colab_type="code" colab={}


# + id="nPwMOvv0rubw" colab_type="code" colab={}


# + id="CzDQbIRRruby" colab_type="code" outputId="57863c7a-ff0b-48ab-b4f8-92412d066af9" colab={}


# + id="p1354XK_rubz" colab_type="code" outputId="e77346d1-1118-4bda-9e96-e7b9c2a5c239" colab={}


# + [markdown] id="X4pCg0Mhrub0" colab_type="text"
# **TASK: Given the customer below, would you offer this person a loan?**

# + id="Oh2-lFolrub1" colab_type="code" outputId="ebfef49f-16aa-4d6e-8f83-69114599fb28" colab={}
import random
random.seed(101)
random_ind = random.randint(0,len(df))

new_customer = df.drop('loan_repaid',axis=1).iloc[random_ind]
new_customer

# + id="aE22jSryrub1" colab_type="code" colab={}
# CODE HERE

# + id="0yECECw2rub2" colab_type="code" outputId="b654d645-d598-4337-ac9b-7e29af342c9f" colab={}


# + [markdown] id="rKR4xawvrub4" colab_type="text"
# **TASK: Now check, did this person actually end up paying back their loan?**

# + id="yhiPdisArub4" colab_type="code" colab={}
# CODE HERE

# + id="acBPSdNVrub5" colab_type="code" outputId="f40adeae-4323-451e-c4c8-46eb842b9c34" colab={}


# + [markdown] id="KDji2S0Grub5" colab_type="text"
# # GREAT JOB!

StarcoderdataJupyterScriptsDedupFiltered