# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python [Root]
#     language: python
#     name: Python [Root]
# ---

# # PREPROCESSING

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
% matplotlib inline

# ## LOAD DATA
#
# The follow features were dropped to prevent overfitting: CountyName, State, and Date.

df = pd.read_csv('data/wheat-2013-supervised.csv')
drop_cols = ['CountyName','State','Date']
df.drop(drop_cols,axis=1,inplace=True)
df.head()

# ## DATA SIZE CHECK
#
# I check for data length to double check if I should run on my local computer or AWS.

df.shape

# ## NULL CHECK
#
# This is important to note for the preprocessing stage, where I will impute a value (such as average) for missing data.
#
# The following features have missing data: precipIntensity, precipIntensityMax, precipProbability, pressure, visibility

df.isnull().sum()

# ## FEATURE VARIANCE CHECK
#
# I check for zero variance features (i.e. features with just one value). The following feature(s) have zero variance: precipTypeIsOther

for col in df.columns[5:]:
    if df[col].var() == 0:
        print '*****LOW VARIANCE WARNING***** ==> {} ==> var:{}'.format(col,df[col].var())
    else:
        print '{} ==> var:{}'.format(col,df[col].var())

# ## REMOVE USELESS FEATURE(S)
#
# As you can see below, all values for "precipTypeIsOther" is all zero; thus making it a useless feature. So I removed the feature.

df.precipTypeIsOther.value_counts().plot(kind='bar')
df.drop('precipTypeIsOther',axis=1,inplace=True)

# ## IMPUTATION | DROP NULL ROWS
#
# I'm typically frugal with my data and try not to throw anything out (i.e. I would impute averages or other extrapolated values that are appropriate). However, the ratio of NaNs to data points is so small enough for me to drop any NaNs/null values.

df.dropna(inplace=True)

# ## SAVE DATA TO CSV

df.to_csv('data/wheat-2013-supervised-edited.csv')

StarcoderdataJupyterScriptsDedupFiltered