# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/DanielMartinAlarcon/DS-Sprint-01-Dealing-With-Data/blob/master/module1-afirstlookatdata/LS_DS_111_A_First_Look_at_Data.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="Okfr_uhwhS1X" colab_type="text"
# # Lambda School Data Science - A First Look at Data
#
#

# + [markdown] id="9dtJETFRhnOG" colab_type="text"
# ## Lecture - let's explore Python DS libraries and examples!
#
# The Python Data Science ecosystem is huge. You've seen some of the big pieces - pandas, scikit-learn, matplotlib. What parts do you want to see more of?

# + id="WiBkgmPJhmhE" colab_type="code" colab={}
# More Scikit-learn!

# + [markdown] id="002YYgH_CQqm" colab_type="text"
#

# + [markdown] id="lOqaPds9huME" colab_type="text"
# ## Assignment - now it's your turn
#
# Pick at least one Python DS library, and using documentation/examples reproduce in this notebook something cool. It's OK if you don't fully understand it or get it 100% working, but do put in effort and look things up.

# + id="TGUS79cOhPWj" colab_type="code" colab={}
# I,ll reproduce a quick example of implementing a regression on housing data.
# Source: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/keras/basic_regression.ipynb#scrollTo=f-OHX4DiXd8x

# The original source explained what it was doing in text cells.  I copied only
# their code cells, so I'll add my own text cells to explain what's going on.


# + [markdown] id="P7r4igooHU2b" colab_type="text"
# This example uses Keras, " a high-level API to build and train deep learning models" in TensorFlow.

# + id="RHtX8R36CvlJ" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="39ee07c3-9fde-48e6-9aa2-531f150890f1"
from __future__ import absolute_import, division, print_function

import tensorflow as tf
from tensorflow import keras

import numpy as np

print(tf.__version__)

# + [markdown] id="pvA3XlbmHqIs" colab_type="text"
# The dataset also comes from TensorFlow, and consists of Boston housing data from the 70s, so we can predict housing prices based on various neighborhood features.  Note how this dataset is already divided into treaining and testing.

# + id="97Oqh_7fDJcw" colab_type="code" colab={}
boston_housing = keras.datasets.boston_housing

(train_data, train_labels), (test_data, test_labels) = boston_housing.load_data()

# Shuffle the training set
order = np.argsort(np.random.random(train_labels.shape))
train_data = train_data[order]
train_labels = train_labels[order]

# + id="u6PA8P4YEU5r" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 51} outputId="cb74f728-f8b3-41d3-8180-86462ec6b71f"
print("Training set: {}".format(train_data.shape))  # 404 examples, 13 features
print("Testing set:  {}".format(test_data.shape))   # 102 examples, 13 features

# + [markdown] id="VH_XOYRXIp3R" colab_type="text"
# We move it into Pandas to check on the contents.

# + id="XyP_xhY1EnWM" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="3eb48c93-8000-4eac-bbd0-8d604894b1ba"
import pandas as pd

column_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',
                'TAX', 'PTRATIO', 'B', 'LSTAT']

df = pd.DataFrame(train_data, columns=column_names)
df.head()

# + [markdown] id="ytptYkhYI0_5" colab_type="text"
# All the features in this dataset vary over different ranges, so the next cell normalizes them by subtracting the mean and dividing by the standard deviation

# + id="w2uyWAUMFEO9" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 68} outputId="fdd93586-830a-4335-a8be-bf90fe63b07c"
# Test data is *not* used when calculating the mean and std

mean = train_data.mean(axis=0)
std = train_data.std(axis=0)
train_data = (train_data - mean) / std
test_data = (test_data - mean) / std

print(train_data[0])  # First training sample, normalized


# + [markdown] id="PqOdMUSEJrUY" colab_type="text"
# With the data normalized, we now build a deep learning model with " two densely connected hidden layers, and an output layer that returns a single, continuous value."  All of this is internal to TensorFlow, but it's good to know how to interface with it.  I have no idea what the internals are doing here.

# + id="I2RuUy4FFJlr" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 238} outputId="c27c1649-ef82-4267-ee34-06503c209f69"
def build_model():
  model = keras.Sequential([
    keras.layers.Dense(64, activation=tf.nn.relu,
                       input_shape=(train_data.shape[1],)),
    keras.layers.Dense(64, activation=tf.nn.relu),
    keras.layers.Dense(1)
  ])

  optimizer = tf.train.RMSPropOptimizer(0.001)

  model.compile(loss='mse',
                optimizer=optimizer,
                metrics=['mae'])
  return model

model = build_model()
model.summary()


# + [markdown] id="k75dlU9RKsPx" colab_type="text"
# The model trains over 500 epochs, which I assume means "instances of tweaking the parameters in the layer". It uses the object "history" to record stats about the model at each epoch, thta can be graphed later.

# + id="HaTpb_qRFafn" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 139} outputId="9544e6cb-12bd-41d8-b3d2-a58defa5e373"
# Display training progress by printing a single dot for each completed epoch
class PrintDot(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs):
    if epoch % 100 == 0: print('')
    print('.', end='')

EPOCHS = 500

# Store training stats
history = model.fit(train_data, train_labels, epochs=EPOCHS,
                    validation_split=0.2, verbose=0,
                    callbacks=[PrintDot()])

# + [markdown] id="3aghOBCDK-5r" colab_type="text"
# First, the model is trained over 500 epochs and we graph its mean absolute error as it goes, so that we can see how the model stops improving after a certain number of epochs.  I can't figure out the exact meaning of 'val_mean_absolute_error', the green line.

# + id="-1_k-DPCFqHf" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 365} outputId="3db47689-62e2-4f2a-89e8-fb02f4571948"
import matplotlib.pyplot as plt


def plot_history(history):
  plt.figure()
  plt.xlabel('Epoch')
  plt.ylabel('Mean Abs Error [1000$]')
  plt.plot(history.epoch, np.array(history.history['mean_absolute_error']),
           label='Train Loss')
  plt.plot(history.epoch, np.array(history.history['val_mean_absolute_error']),
           label = 'Val loss')
  plt.legend()
  plt.ylim([0, 5])

plot_history(history)

# + [markdown] id="XZxdizVDLxuk" colab_type="text"
# We create a parameter that tracks how the model is getting better, and stops when it plateaus.

# + id="fvP44_HdGEnN" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 436} outputId="f3523896-da76-422f-c162-77897e61453f"
model = build_model()

# The patience parameter is the amount of epochs to check for improvement
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)

history = model.fit(train_data, train_labels, epochs=EPOCHS,
                    validation_split=0.2, verbose=0,
                    callbacks=[early_stop, PrintDot()])

plot_history(history)

# + [markdown] id="jarGzCUYMFMr" colab_type="text"
# Testing the model on the test set:

# + id="T0-DMhFTGcpR" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="0709ed4c-ef9b-45ba-add3-6dca2f031c6d"
[loss, mae] = model.evaluate(test_data, test_labels, verbose=0)

print("Testing set Mean Abs Error: ${:7.2f}".format(mae * 1000))

# + [markdown] id="y2vYNlU9Mapq" colab_type="text"
# And finally, making predictions based on the test set and plotting those predictions against the actual values.

# + id="detWiGgOGeYe" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 361} outputId="b32f35c0-f50a-4332-8233-2e8e7c24342d"
test_predictions = model.predict(test_data).flatten()

plt.scatter(test_labels, test_predictions)
plt.xlabel('True Values [1000$]')
plt.ylabel('Predictions [1000$]')
plt.axis('equal')
plt.xlim(plt.xlim())
plt.ylim(plt.ylim())
_ = plt.plot([-100, 100], [-100, 100])


# + id="ycGI2rnMGhjn" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 361} outputId="233a34cc-ab4e-49d6-fb5d-409da75e8eed"
error = test_predictions - test_labels
plt.hist(error, bins = 50)
plt.xlabel("Prediction Error [1000$]")
_ = plt.ylabel("Count")

# + [markdown] id="2AiHYOsMMh0d" colab_type="text"
# And that's it!

# + [markdown] id="BT9gdS7viJZa" colab_type="text"
# ### Assignment questions
#
# After you've worked on some code, answer the following questions in this text block:
#
# 1.  Describe in a paragraph of text what you did and why, as if you were writing an email to somebody interested but nontechnical.
#
# I took a small amount of housing data from the 1970s in Boston, MA, and used it to train a black box called a "deep learning model". I don't know how the model works (hence 'black box'), but I cleaned up some data and fed it to the model.  The model looked at the data 500 times and tried to better understand it.  I asked the model to graph its progress as it went, and then asked it to stop trying to understand the data once it didn't seem to be understanding it any better (which happened after about 200 tries). Once the model was trained on all the training data, I asked it to make predictions for the test data and plotted the results to see how effective it had been on average.
#
# 2.  What was the most challenging part of what you did?
#
# Finding an example online that was comprehensible without being trivial.  This one is a relatively boring problem, but it's my first time trying to interface with TensorFlow.  I expect that I'll want to use that tool a lot later on. 
#
# 3.  What was the most interesting thing you learned?
#
# That it can be quite this simple to play with deep learning, at least if you know what you're doing.  Also, it's good to see that buzzword actually applied to something real, and with the details worked out of how to build a simple model.
#
# 4.  What area would you like to explore with more time?
#
# What on Earth the deep learning model is doing on the inside.  Some diagrams would be useful.
#

# + id="TTB3rLAlGg0P" colab_type="code" colab={}


# + [markdown] id="_XXg2crAipwP" colab_type="text"
# ## Stretch goals and resources
#
# Following are *optional* things for you to take a look at. Focus on the above assignment first, and make sure to commit and push your changes to GitHub (and since this is the first assignment of the sprint, open a PR as well).
#
# - [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)
# - [scikit-learn documentation](http://scikit-learn.org/stable/documentation.html)
# - [matplotlib documentation](https://matplotlib.org/contents.html)
# - [Awesome Data Science](https://github.com/bulutyazilim/awesome-datascience) - a list of many types of DS resources
#
# Stretch goals:
#
# - Find and read blogs, walkthroughs, and other examples of people working through cool things with data science - and share with your classmates!
# - Write a blog post (Medium is a popular place to publish) introducing yourself as somebody learning data science, and talking about what you've learned already and what you're excited to learn more about

StarcoderdataJupyterScriptsDedupFiltered