<jupyter_start><jupyter_text>Import DataRead in the csv containing the merged data from the weather API and Nest data.<jupyter_code>data = pd.read_csv('merged_data.csv')
data.head(10)<jupyter_output><empty_output><jupyter_text>Clean the DataRemove unnecessary columns with lots of missing values and check for ways to fix missing values (set to zero, median, mean etc.<jupyter_code># update NA values to zero for certain columns
data['precipAccumulation'] = data['precipAccumulation'].fillna(0)  # if precipitation accumulation is NAN assume 0
data['precipIntensity'] = data['precipIntensity'].fillna(0)  # if precipitation intensity is NAN assume 0
data['nearestStormBearing'] = data['nearestStormBearing'].fillna(0)

# drop precipType since it is redundant to icon or summary column
data.drop(['precipType'], axis = 1, inplace = True)
data.drop(['precipIntensityError'], axis = 1, inplace = True)
#data['precipType'] = data['precipType'].fillna('None')  # if precipitation intensity is NAN assume 0
data.head(10)<jupyter_output><empty_output><jupyter_text>Convert Categorical Data to NumericalIn order to properly train a model we want to convert all of our dataframe columns that hold categorical data (ex. rainy, cloudy etc) into numerical data. There are several techniques to do so. The one I will be using is one hot encoding. This technique makes sure each different categorical value is weighted the same which the technique label encoding does not do.One hot encoding adds additional feature columns to the dataset.<jupyter_code># summary column one hot encoding
summary_encode = pd.get_dummies(data.summary, prefix='summary')
# icon column one hot encoding
icon_encode = pd.get_dummies(data.icon, prefix='icon')
icon_encode.head()
# concat the one hot encoded columns onto our data and drop the original column
data = pd.concat([data, summary_encode], axis=1)
data = pd.concat([data, icon_encode], axis=1)
data.drop(['summary'], axis = 1, inplace=True)
data.drop(['icon'], axis = 1, inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Data ExplorationLet's take a quick glance at the properties of our dataset in more detail.<jupyter_code># let's replace all our empty string cells with NaN and then drop them
data.replace('', np.nan, inplace=True)
data = data.dropna(axis = 0)
data.describe().transpose()<jupyter_output><empty_output><jupyter_text>Splitting the DataWill split the data into 70% training data, 15% for validation,and 15% for testing.<jupyter_code>data.drop(['time'], axis = 1, inplace = True)
# was getting strange NAN values within these two columns (not sure why, need to look into it)
data.drop(['summary_Rain'], axis = 1, inplace = True)
data.drop(['summary_Light Snow'], axis = 1, inplace = True)

column_indices = {name: i for i, name in enumerate(data.columns)}

size = len(data)
train_df = data[0:int(size * 0.7)]
val_df = data[int(size * 0.7):int(size * 0.85)]
test_df = data[int(size * 0.85):]

num_features = data.shape[1]<jupyter_output><empty_output><jupyter_text>Normalizing the DataNormalizing the data is a useful step to bring all the features into similar ranges of value so that one doesn't have a greater impact on the data then the others.For normalization let's use the technique of subtracting the mean and dividing by the standard deviation of each feature. **This is commonly known as Z-score normalization.**The mean and standard deviation should only be computed using the training data so that the models have no access to the values in the validation and test sets.<jupyter_code>data.head()
train_mean = train_df.mean()
train_std = train_df.std()

train_df = (train_df - train_mean) / train_std
val_df = (val_df - train_mean) / train_std
test_df = (test_df - train_mean) / train_std
df_std = (data - train_mean) / train_std
df_std = df_std.melt(var_name='Column', value_name='Normalized')
plt.figure(figsize=(12, 6))
ax = sns.violinplot(x='Column', y='Normalized', data=df_std)
_ = ax.set_xticklabels(data.keys(), rotation=90)<jupyter_output><empty_output><jupyter_text>Might want to look into removing the features with the long wicks.  Window GenerationWe need to generate a window that we will make a prediction based off of and that will slide through our data.The models in this tutorial will make a set of predictions based on a window of consecutive samples from the data.<jupyter_code>class WindowGenerator():
    def __init__(self, input_width, label_width, shift, train_df=train_df, val_df=val_df, test_df=test_df, label_columns = None):
        # store data
        self.train_df = train_df
        self.val_df = val_df
        self.test_df = test_df
        # Work out the label column indices.
        self.label_columns = label_columns
        if label_columns is not None:
            self.label_columns_indices = {name: i for i, name in
                                        enumerate(label_columns)}
        self.column_indices = {name: i for i, name in
                               enumerate(train_df.columns)}
        # Work out the window parameters.
        self.input_width = input_width
        self.label_width = label_width
        self.shift = shift

        self.total_window_size = input_width + shift

        self.input_slice = slice(0, input_width)
        self.input_indices = np.arange(self.total_window_size)[self.input_slice]

        self.label_start = self.total_window_size - self.label_width
        self.labels_slice = slice(self.label_start, None)
        self.label_indices = np.arange(self.total_window_size)[self.labels_slice]

    def __repr__(self):
        return '\n'.join([
            f'Total window size: {self.total_window_size}',
            f'Input indices: {self.input_indices}',
            f'Label indices: {self.label_indices}',
            f'Label column name(s): {self.label_columns}'])
    
    # split the features and labels for each sequence properly
    def split_window(self, features):
        inputs = features[:, self.input_slice, :]
        labels = features[:, self.labels_slice, :]
        if self.label_columns is not None:
            labels = tf.stack(
                [labels[:, :, self.column_indices[name]] for name in self.label_columns],
                axis=-1)

        # Slicing doesn't preserve static shape information, so set the shapes
        # manually. This way the `tf.data.Datasets` are easier to inspect.
        inputs.set_shape([None, self.input_width, None])
        labels.set_shape([None, self.label_width, None])

        return inputs, labels
    
    def make_dataset(self, data):
        data = np.array(data, dtype=np.float32)
        ds = tf.keras.preprocessing.timeseries_dataset_from_array(
        data=data,
        targets=None,
        sequence_length=self.total_window_size,
        sequence_stride=1,
        shuffle=True,
        batch_size=32,)

        ds = ds.map(self.split_window)

        return ds
    
    @property
    def train(self):
        return self.make_dataset(self.train_df)

    @property
    def val(self):
        return self.make_dataset(self.val_df)

    @property
    def test(self):
        return self.make_dataset(self.test_df)

# example window 
window = WindowGenerator(input_width = 8, label_width = 1, shift = 1, label_columns = ['avg(temp)'])
window
# ability to loop through data and windows
for example_inputs, example_labels in window.train.take(1):
    print(f'Inputs shape (batch, time, features): {example_inputs.shape}')
    print(f'Labels shape (batch, time, features): {example_labels.shape}')<jupyter_output>Inputs shape (batch, time, features): (32, 8, 46)
Labels shape (batch, time, features): (32, 1, 1)<jupyter_text>LSTM ModelBuilding my LSTM model.<jupyter_code>MAX_EPOCHS = 20

def compile_and_fit(model, window, patience=2):
    early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss',
                                                    patience=patience,
                                                    mode='min')

    model.compile(loss=tf.losses.MeanSquaredError(),
                optimizer=tf.optimizers.Adam(),
                metrics=[tf.metrics.MeanAbsoluteError()])

    history = model.fit(window.train, epochs=MAX_EPOCHS,
                      validation_data=window.val,
                      callbacks=[early_stopping])
    return history
val_performance = {}
performance = {}
lstm_model = tf.keras.models.Sequential([
    # Shape [batch, time, features] => [batch, time, lstm_units]
    tf.keras.layers.LSTM(32, return_sequences=True),
    # Shape => [batch, time, features]
    tf.keras.layers.Dense(units=1)
])
history = compile_and_fit(lstm_model, window)

val_performance['LSTM'] = lstm_model.evaluate(window.val)
performance['LSTM'] = lstm_model.evaluate(window.test, verbose=0)
performance['LSTM']
val_performance['LSTM']<jupyter_output><empty_output><jupyter_text>Stacking Windows (maybe useful)<jupyter_code># Stack three slices, the length of the total window:
example_window = tf.stack([np.array(train_df[:window.total_window_size]),
                           np.array(train_df[200:200+window.total_window_size]),
                           np.array(train_df[400:400+window.total_window_size])])


example_inputs, example_labels = window.split_window(example_window)

print('All shapes are: (batch, time, features)')
print(f'Window shape: {example_window.shape}')
print(f'Inputs shape: {example_inputs.shape}')
print(f'labels shape: {example_labels.shape}')<jupyter_output>All shapes are: (batch, time, features)
Window shape: (3, 9, 48)
Inputs shape: (3, 8, 48)
labels shape: (3, 1, 1)