<jupyter_start><jupyter_text>Introduction to Regression w NNsPredict y from x<jupyter_code># Import TensorFlow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)<jupyter_output>2.5.0<jupyter_text>Creating data to view and fit<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Create features
x = np.array([-7, -4, -1, 2, 5, 8, 11, 14])
x = tf.cast(x, dtype=tf.float32)

# Create labels
y = np.array([3, 6, 9, 12, 15, 18, 21, 24])
y = tf.cast(y, dtype=tf.float32)

# Visualize it
plt.scatter(x, y)<jupyter_output><empty_output><jupyter_text>Input and output shapes<jupyter_code># Create a demo tensor for our housing price predicton problem
house_info = tf.constant(["bedroom", "bathroom", "garage"])
house_price = tf.constant([939700])
# Turn out NumPy arrays into tensors
x = tf.constant(x)
y = tf.constant(y)
x, y
input_shape = x[0].shape
output_shape = y[0].shape
# If printed, they each have no shape because they are scalars(rank 0 meanning no dimensions)
# If you just did x.shape and y.shape, we don't want the input and the output to have 8 neurons
# We just want on x to have an output y(so 1 input and output)<jupyter_output><empty_output><jupyter_text>Modeling with TensorFlow1. Create a model2. Compile a model3. Fitting a model(let the NN to find the patterns between the X and the Y)<jupyter_code># Set random seed
tf.random.set_seed(42)

#1. Create a model using the Sequential API
model = tf.keras.Sequential([
  tf.keras.layers.Dense(1)
])

#2. Compile the model
model.compile(loss=tf.keras.losses.mae, 
              optimizer=tf.keras.optimizers.SGD(), # Stochastic Gradient Descent
              metrics=["mae"]
              )

#3. Fit the model
model.fit(x, y, epochs=100, shuffle=True)
# Try and make a prediction
model.predict([17.])<jupyter_output><empty_output><jupyter_text>Improving our model1. Add more layers, increase the number of neurons within each hidden layer, or change the activation functions of each layer.2. Change the optimization function or perhaps the learning rate of the optimization function.3. Increase number of epochs or give the model more data points to learn from.<jupyter_code># Let's rebuild our model

#1. Create a model using the Sequential API
model = tf.keras.Sequential([
  tf.keras.layers.Dense(1)
])

#2. Compile the model
model.compile(loss=tf.keras.losses.mae, 
              optimizer=tf.keras.optimizers.SGD(),
              metrics=["mae"]
              )

#3. Fit the model
model.fit(x, y, epochs=100, shuffle=True) # Changed number of epochs.
# Try and make a prediction
model.predict([17.])
# Let's see if we can make another to improve our model

#1. Create the model (this time with an extra hidden layer)
model = tf.keras.Sequential([
  tf.keras.layers.Dense(100, activation=None),
  tf.keras.layers.Dense(1)
])

#2. Compile the model
model.compile(loss="mae",
              optimizer=tf.keras.optimizers.Adam(lr=0.02), # Learning rate is probably the most 
                                                           # important hyperparameter in a NN(when it is 0.02, it is good)
              metrics=["mae"])

#3. Fit the model
model.fit(x, y, epochs=100)
# Try and make a prediction
model.predict([17.])
# This has a really bad result but during training(keeping everything the same but you have a 100 unit dense relu), the mae is actually really small
# This result is due to overfitting the data during training.<jupyter_output><empty_output><jupyter_text>Evaluating a modelIn practice, a typical workflow when building NNs is:``` Build a model -> fit it -> evaluate it -> tweak the model -> fit it -> evaluate it -> . . . ```<jupyter_code># Make a bigger dataset
x = tf.range(-100, 100, 4)
# Make labels for the dataset
y = x + 10
y
# Visualize the data
plt.scatter(x, y)<jupyter_output><empty_output><jupyter_text>The 3 Sets . . .* Training Set(70-80%)* Validation Set(10-15%)* Test Set(10-15%)<jupyter_code># Split the data into train and test sets
x_train = x[:40] # first 40 are training samples (80% of the data)
y_train = y[:40]

x_test = x[40:] # last 10 are testing samples(20% of the data)
y_test = y[40:]

len(x_train), len(x_test), len(y_train), len(y_test)<jupyter_output><empty_output><jupyter_text>Visualizing dataNow we've got our data in training and test sets. Visualize again.<jupyter_code>plt.figure(figsize=(10, 7))
# Plot training data in blue
plt.scatter(x_train, y_train, c="b", label="Training data")
plt.scatter(x_test, y_test, c="g", label="Testing data")
# Show legend
plt.legend()
# Let's have a look at how to build a NN

#1. Create the model
model = tf.keras.Sequential([
  tf.keras.layers.Dense(1)                            
])

#2. Compile the model
model.compile(loss=tf.keras.losses.mae,
              optimizer=tf.keras.optimizers.SGD(),
              metrics=["mae"])

#3. Fit the model
# model.fit(x_train, y_train, epochs=100)<jupyter_output><empty_output><jupyter_text>Visualizing the model<jupyter_code># Let's create a model which builds automatically by defining 
# the input shape in the first layer
tf.random.set_seed(42)

# Create a model(same as above)
model = tf.keras.Sequential([
  tf.keras.layers.Dense(10, input_shape=[1], name="input_layer"),
  tf.keras.layers.Dense(1, name="output_layer")    
  # 10 is the number of neurons in that layer.
  # Input shape=1 means that there is one nueron in the layer before
  # connecting to the current layer.                  
], name="First_Model")

#2. Compile the model
model.compile(loss=tf.keras.losses.mae,
              optimizer=tf.keras.optimizers.SGD(learning_rate=0.01),
              metrics=["mae"])

model.summary()<jupyter_output>Model: "First_Model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_layer (Dense)          (None, 10)                20        
_________________________________________________________________
output_layer (Dense)         (None, 1)                 11        
=================================================================
Total params: 31
Trainable params: 31
Non-trainable params: 0
_________________________________________________________________<jupyter_text>* Total params - total number of parameters in the model.* Trainable parameters - these are the parameters(patterns) the model can update as it trains.* Non-training params - these parameters aren't updated during training (this is typical when you bring in already learned model-importing a model)<jupyter_code># Let's fit our model to the training data
model.fit(x_train, y_train, epochs=100, verbose=0)
# Get summary of our model
model.summary()
from tensorflow.keras.utils import plot_model

plot_model(model=model, show_shapes=True)<jupyter_output><empty_output><jupyter_text>Visualizing out model's predictionsTo visualize predictions, it's a good idea to plot them against the ground truth labels. Often you'll see this in the form of y_test or y_true versus y_pred(ground truth versus your models prediction)<jupyter_code># Make some predictions
y_pred = model.predict(x_test)
y_pred
# Let's create a plotting function
def plot_predictions(train_data=x_train,
                     train_labels=y_train,
                     test_data=x_test,
                     test_labels=y_test,
                     predictions=y_pred):
  plt.figure(figsize=(10, 7))
  # Plot training data in blue
  plt.scatter(train_data, train_labels, c="b", label="Training data")
  # Plot testing data in green
  plt.scatter(test_data, test_labels, c="g", label="Testing data")
  # Plot models predictions in red
  plt.scatter(test_data, predictions, c="r", label="Predictions")
  # Show the legend
  plt.legend()
plot_predictions(train_data=x_train,
                 train_labels=y_train,
                 test_data=x_test,
                 test_labels=y_test,
                 predictions=y_pred)<jupyter_output><empty_output><jupyter_text>Evaluating our model's predictions with regression evaluation metricsSince we're working on a regression, two of the main metrics:* MAE - mean absolute error, "on average, how wrong is each of my model's predictions"* MSE - mean square error, "square the average errors"<jupyter_code># Evaluate the model on the test
model.evaluate(x_test, y_test)
# Calculate the mean absolute error
mae = tf.metrics.mean_absolute_error(y_test, y_pred)
# y_pred has one extra dimension than y_test so we have to make them
# the same dimension. Have to squeeze(removes all one dimensions of an array).
mae = tf.keras.losses.MAE(y_test, tf.squeeze(y_pred))
mae
# Calculate the mean square error
mse = tf.metrics.mean_squared_error(y_true=y_test,
                                    y_pred=tf.squeeze(y_pred))
mse
# Make some functions to reuse MAE and MSE
def MAE(y_true, y_pred):
  return tf.keras.losses.MAE(y_test, tf.squeeze(y_pred))

def MSE(y_true, y_pred):
  return tf.keras.losses.MSE(y_test, tf.squeeze(y_pred))<jupyter_output><empty_output><jupyter_text>Running experiments to improve our model***TWEAK TIME:***1. Get more data to train on2. Make your model larger(using more hidden nuerons or hidden layers)3. Train for longer.***Let's do 3 experiments***1. Original model2. 2 layers trained with 100 epochs3. 2 layers trained for 500 epochs. ***Model 1***<jupyter_code># Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_1 = tf.keras.Sequential([
  tf.keras.layers.Dense(1)                               
])

# 2. Compile the model
model_1.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_1.fit(x_train, y_train, epochs=100)
# Make and plot the predictions for model_1
y_preds_1 = model_1.predict(x_test)
plot_predictions(predictions=y_preds_1)
# Calculate model_1 evaluation metrics
mae_1 = MAE(y_test, y_preds_1)
mse_1 = MSE(y_test, y_preds_1)
mae_1, mse_1<jupyter_output><empty_output><jupyter_text>***Model 2***<jupyter_code># Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)                               
])

# 2. Compile the model
model_2.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mse"])

# 3. Fit the model
model_2.fit(x_train, y_train, epochs=100)
# Make and plot predictions of model_2
y_preds_2 = model_2.predict(x_test)
plot_predictions(predictions=y_preds_2)
# Calculate model_2 evaluation metrics
mae_2 = MAE(y_test, y_preds_2)
mse_2 = MSE(y_test, y_preds_2)
mae_2, mse_2<jupyter_output><empty_output><jupyter_text>***Model 3***<jupyter_code># Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1),                               
])

# 2. Compile the model
model_3.compile(loss=tf.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_3.fit(x_train, y_train, epochs=500)
# Make and plot some predictions
y_preds_3 = model_3.predict(x_test)
plot_predictions(predictions=y_preds_3) # Hella overfitting
# Calculate model_3 evaluation metrics
mae_3 = MAE(y_test, y_preds_3)
mse_3 = MSE(y_test, y_preds_3)
mae_3, mse_3<jupyter_output><empty_output><jupyter_text>Comparing the results of our experimentsLet's compare the results of our experiments<jupyter_code># Let's compare our model's results using a pandas DataFrame
model_results = [["model_1", mae_1.numpy(), mse_1.numpy()],
                 ["model_2", mae_2.numpy(), mse_2.numpy()],
                 ["model_3", mae_3.numpy(), mse_3.numpy()]]
all_results = pd.DataFrame(model_results, columns=["model", "mae", "mse"])                 
all_results<jupyter_output><empty_output><jupyter_text>Tracking your experimentsOne really good habit in machine learning modeling is tracking the results of your experiments. This can be slow so there are tools to help us:1. TensorBoard2. Weights & Biases  Saving our modelsSaving models lets us use them outside of Google ColabThere are two main ways to save:1. The SavedModel format2. The HDF5 format<jupyter_code># Save model using the SavedModel format
model_2.save("best_model_SavedModel_format")
# Save model using the HDF5 format
model_2.save("best_model_HDF5_format.h5")<jupyter_output><empty_output><jupyter_text>Loading in a saved model<jupyter_code># Load in the SavedModel format model
loaded_SavedModel_format = tf.keras.models.load_model("/content/best_model_SavedModel_format")
loaded_SavedModel_format.summary()
# Compare model_2 predictions with SavedModel format model predictions 
model_2_preds = model_2.predict(x_test)
loaded_SavedModel_format_preds = loaded_SavedModel_format.predict(x_test)
model_2_preds == loaded_SavedModel_format_preds
# Load in the .h5 format
loaded_h5_model = tf.keras.models.load_model("/content/best_model_SavedModel_format")
loaded_h5_model.summary()
# Check to see if loaded .h5 model predictions match model_2
model_2_preds = model_2.predict(x_test)
loaded_h5_model_preds = loaded_h5_model.predict(x_test)
model_2_preds == loaded_SavedModel_format_preds<jupyter_output><empty_output><jupyter_text>Download a model (or any other model) from Google ColabIf you want to download your files from Colab:1. Go to files and right click then download2. Use code below3. Use google drive<jupyter_code># Download a file form Google Colab
from google.colab import files
files.download("/content/best_model_SavedModel_format")<jupyter_output><empty_output><jupyter_text>Larger Example<jupyter_code># Import required libraries
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt

# Read in the insurance dataset
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance 
insurance["bmi"], insurance["sex"]
# If you run the code above, you can see that different inputs have different types
# For example, bmi is a float while sex is an object
# ALL NN INPUTS HAVE TO BE NUMERICAL so to turn the objects into numbers,
# you have to use one-hot encoding to numerically encode non-numbers
# Let's try one-hot encode our DataFrame so they are all numbers
insurance_onehot = pd.get_dummies(insurance)
insurance_onehot
# Create x and y values (features and labels)
x = insurance_onehot.drop("charges", axis=1)
y = insurance_onehot["charges"]
x.head(), y.head()
# Create training and test sets
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)
len(x), len(x_train), len(x_test)
# Build a NN (sort of like model_2 above)
tf.random.set_seed(42)

# 1. Create model
insurance_model = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)                                       
])

# 2. Compile the model
insurance_model.compile(loss=tf.losses.mae,
                        optimizer=tf.keras.optimizers.SGD(),
                        metrics=["mae"])

# 3. Fit the model
insurance_model.fit(x_train, y_train, epochs=100)
# Check the results of the insurance_model on the test data
insurance_model.evaluate(x_test, y_test)<jupyter_output>9/9 [==============================] - 0s 2ms/step - loss: 7023.3291 - mae: 7023.3291<jupyter_text>Let's try and improve our model1. Add an extra layer with more hidden units and Adam optimizer2. Same as above and train for longer<jupyter_code>tf.random.set_seed(42)

# 1. Create the model
insurance_model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)                                         
])

# 2. Compile the model
insurance_model_2.compile(loss=tf.losses.mae,
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# 3. Fit the model
insurance_model_2.fit(x_train, y_train, epochs=100)
# Evaluate the larger model
insurance_model_2.evaluate(x_test, y_test)
tf.random.set_seed(42)

# 1. Create the model
insurance_model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)                                         
])

# 2. Compile the model
insurance_model_3.compile(loss=tf.losses.mae,
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# 3. Fit the model
history = insurance_model_3.fit(x_train, y_train, epochs=200)
# Evaluate the model
insurance_model_3.evaluate(x_test, y_test)
# Plot history (also known as a loss curve)
pd.DataFrame(history.history).plot()
plt.ylabel("loss")
plt.xlabel("epochs")<jupyter_output><empty_output><jupyter_text>Preprocessing data (normalization and standardization)Neural Nets prefer normalized inputs.<jupyter_code>print(x["age"].plot(kind="hist"))
# This histogram is not normal so we got to make it normal
# Read in the insurance dataframe
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

# Create our column transformer
ct = make_column_transformer(
    (MinMaxScaler(), ["age", "bmi", "children"]), # Turn all these values to between 0 and 1
    (OneHotEncoder(handle_unknown="ignore"), ["sex", "smoker", "region"])
)

# Create x and y
x = insurance.drop("charges", axis=1)
y = insurance["charges"]

# Build our train and test sets
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

# Fit the column transformer to our traingin data
ct.fit(x_train)

# Transform training and test data wiht normalization (MinMaxScaler) and OneHotEncoder) 
x_train_normal = ct.transform(x_train)
x_test_normal = ct.transform(x_test)
# What does our data look like now?
print(x_train.loc[0]) # Takes the first row

x_train.shape, x_train_normal.shape
tf.random.set_seed(42)

# 1. Create the model
insurance_model_4 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)                                       
])

# 2. Compile the model
insurance_model_4.compile(loss=tf.losses.mae,
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# 3. Fit the model
insurance_model_4.fit(x_train_normal, y_train, epochs=100)
# Evaluate our insurance model trained on normalized data
insurance_model_4.evaluate(x_test_normal, y_test)<jupyter_output>9/9 [==============================] - 0s 2ms/step - loss: 3438.7844 - mae: 3438.7844
StarcoderdataJupyterStructuredCleanDedup