<jupyter_start><jupyter_text>Data Science em Produção     =-=- ROSSMANN - STORE SALES PREDICTION -=-=   0. Imports<jupyter_code>import inflection  # helper function

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>0.1. Helper Functions<jupyter_code># Notebook Setups

sns.set_style('darkgrid')
sns.set_context('talk')
sns.set_palette('Set2')

# Functions

def snakecase(list_of_names):
    """Returns a list of names in snake case, which refers to the style of  
    writing in which each space is replaced by an underscore (_) character."""
    
    new_list = list(map(inflection.underscore, list_of_names))
    return new_list<jupyter_output><empty_output><jupyter_text>0.2. Loading Data<jupyter_code># loading historical data - including Sales
df_sales_raw = pd.read_csv('../raw_data/train.csv', low_memory=False)

# loading information about the stores
df_store_raw = pd.read_csv('../raw_data/store.csv', low_memory=False)

# merging dataframes
df_raw = pd.merge(df_sales_raw, df_store_raw, how='left', on='Store')<jupyter_output><empty_output><jupyter_text>Data Fields  **Most of the fields are self-explanatory. The following are descriptions for those that aren't.**- **Id** - an Id that represents a (Store, Date) duple within the test set;- **Store** - a unique Id for each store;- **Sales** - the turnover for any given day (this is what you are predicting);- **Customers** - the number of customers on a given day;- **Open** - an indicator for whether the store was open: 0 = closed, 1 = open;- **StateHoliday** - indicates a state holiday. Normally all stores, with few exceptions, are closed on state holidays. Note that all schools are closed on public holidays and weekends. a = public holiday, b = Easter holiday, c = Christmas, 0 = None;- **SchoolHoliday** - indicates if the (Store, Date) was affected by the closure of public schools;- **StoreType** - differentiates between 4 different store models: a, b, c, d;- **Assortment** - describes an assortment level: a = basic, b = extra, c = extended;- **CompetitionDistance** - distance in meters to the nearest competitor store;- **CompetitionOpenSince[Month/Year]** - gives the approximate year and month of the time the nearest competitor was opened;- **Promo** - indicates whether a store is running a promo on that day;- **Promo2** - Promo2 is a continuing and consecutive promotion for some stores: 0 = store is not participating, 1 = store is participating;- **Promo2Since[Year/Week]** - describes the year and calendar week when the store started participating in Promo2;- **PromoInterval** - describes the consecutive intervals Promo2 is started, naming the months the promotion is started anew. E.g. "Feb,May,Aug,Nov" means each round starts in February, May, August, November of any given year for that store.   1. Descriptive Data Analysis    1.0. Dataframe in Progress Backup<jupyter_code>df1 = df_raw.copy()<jupyter_output><empty_output><jupyter_text>1.1. Column Renaming<jupyter_code># renaming df1 column names
df1.columns = snakecase(df1.columns)<jupyter_output><empty_output><jupyter_text>1.2. Data Dimension<jupyter_code>print(f'Store Dataframe - Number of Rows: {df1.shape[0]}. \nStore Dataframe - Number of Columns: {df1.shape[1]}.')<jupyter_output>Store Dataframe - Number of Rows: 1017209. 
Store Dataframe - Number of Columns: 18.<jupyter_text>1.3. Data Types<jupyter_code># dataframe data types
df1.dtypes
# setting date column as datetime type
df1['date'] = pd.to_datetime(df1['date'])<jupyter_output><empty_output><jupyter_text>1.4. NA Check<jupyter_code># checking NA - All NA values came from store.csv
df1.isna().sum()
# checking NA using info()
df1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1017209 entries, 0 to 1017208
Data columns (total 18 columns):
 #   Column                        Non-Null Count    Dtype         
---  ------                        --------------    -----         
 0   store                         1017209 non-null  int64         
 1   day_of_week                   1017209 non-null  int64         
 2   date                          1017209 non-null  datetime64[ns]
 3   sales                         1017209 non-null  int64         
 4   customers                     1017209 non-null  int64         
 5   open                          1017209 non-null  int64         
 6   promo                         1017209 non-null  int64         
 7   state_holiday                 1017209 non-null  object        
 8   school_holiday                1017209 non-null  int64         
 9   store_type                    1017209 non-null  object        
 10  assortment                    1017209 non-null  object        
 11[...]<jupyter_text>1.5. Filling in Missing/Null Values  **Number of NA Values**    competition_distance              2642    competition_open_since_month    323348    competition_open_since_year     323348    promo2_since_week               508031    promo2_since_year               508031    promo_interval                  508031<jupyter_code># competition_distance

# maximun distance x 2
max_dist_x_2 = df1['competition_distance'].max() * 2

# assuming competitors are twice as far away as the greatest distance found
df1['competition_distance'] = df1['competition_distance'].apply(lambda x: max_dist_x_2 if np.isnan(x) else x)


# competition_open_since_year

# frequency per year of existing competition_open_since_year data
frequency = df1['competition_open_since_year'].value_counts(
                                                normalize=True).reset_index().rename(
                                                columns={'index': 'year', 
                                                         'competition_open_since_year': 'percent'})

# True/False missing/Null Series
missing = df1['competition_open_since_year'].isna()

# Using Numpy's random.choice to fill out missing data based on the frequency of existing info
df1.loc[missing,'competition_open_since_year'] = np.random.choice(frequency.year, 
                                                                  size=len(df1[missing]), 
                                                                  p=frequency.percent)


# competition_open_since_month

# frequency per month of existing competition_open_since_month data
frequency = df1['competition_open_since_month'].value_counts(
                                                normalize=True).reset_index().rename(
                                                columns={'index': 'month', 
                                                         'competition_open_since_month': 'percent'})

# True/False missing/Null Series
missing = df1['competition_open_since_month'].isna()

# Using Numpy's random.choice to fill out missing data based on the frequency of existing info
df1.loc[missing,'competition_open_since_month'] = np.random.choice(frequency.month, 
                                                                   size=len(df1[missing]), 
                                                                   p=frequency.percent)


# promo2_since_week AND promo2_since_year

# the same date of sale will be used as a reference to fill in the NA values
# then a new timedelta column will be created (promo2_duration)

#promo2_since_week
df1['promo2_since_week'] = df1[['date', 'promo2_since_week']].apply(lambda x: x['date'].week if np.isnan(x['promo2_since_week']) else x['promo2_since_week'], axis=1)

# promo2_since_year
df1['promo2_since_year'] = df1[['date', 'promo2_since_year']].apply(lambda x: x['date'].year if np.isnan(x['promo2_since_year']) else x['promo2_since_year'], axis=1)


# promo_interval

# filling in NA with 'none'
df1['promo_interval'].fillna(value='none', inplace=True)
# creating a column with current month
df1['curr_month'] = df1['date'].dt.strftime('%b')
# creating a column to indicate whether promo2 is active
df1['promo2_active'] = df1[['curr_month', 'promo_interval']].apply(lambda x: 1 if x['curr_month'] in x['promo_interval'].split(',') else 0, axis=1)<jupyter_output><empty_output><jupyter_text>1.6. Changing Data Types<jupyter_code>df1.dtypes
# Changing DTypes from float to integer

df1['competition_distance'] = df1['competition_distance'].astype(int)
df1['competition_open_since_month'] = df1['competition_open_since_month'].astype(int)
df1['competition_open_since_year'] = df1['competition_open_since_year'].astype(int)
df1['promo2_since_week'] = df1['promo2_since_week'].astype(int)
df1['promo2_since_year'] = df1['promo2_since_year'].astype(int)<jupyter_output><empty_output><jupyter_text>1.7. Descriptive Statistics    1.7.0. Numeric vs Categorical - Attributes Split<jupyter_code>df_numeric = df1.select_dtypes(include=['int64', 'float64'])
df_categorical = df1.select_dtypes(exclude=['int64', 'float64', 'datetime64[ns]'])<jupyter_output><empty_output><jupyter_text>1.7.1. Numeric Attributes<jupyter_code># using DF describe() method
df1.describe().T
# central tendency metrics - mean, median
ct_mean = df_numeric.apply(np.mean)
ct_median = df_numeric.apply(np.median)

# dispersion metrics - std, min, max, range, skew, kurtosis
d_std = df_numeric.apply(np.std)
d_min = df_numeric.apply(min)
d_max = df_numeric.apply(max)
d_range = df_numeric.apply(lambda x: x.max() - x.min())
d_skew = df_numeric.apply(lambda x: x.skew())
d_kurtosis = df_numeric.apply(lambda x: x.kurtosis())

metrics = pd.DataFrame({
    'min': d_min, 'max': d_max, 'range': d_range, 'mean': ct_mean, 
    'median': ct_median, 'std': d_std, 'skew': d_skew, 'kurtosis': d_kurtosis
})
metrics<jupyter_output><empty_output><jupyter_text>**competition_distance** - Skew: highly skewed data, high positive value means that the right-hand tail is much longer than the left-hand tail. - Kurtosis: increases as the tails become heavier, the high positive value indicates a very peaked curve.**competition_open_since_year** - Skew: highly skewed data, high negative value means that the left-hand tail is longer than the right-hand tail. - Kurtosis: increases as the tails become heavier, the high positive value indicates a very peaked curve.**sales** - Skewness is close to zero, indicating that the data is not too skewed<jupyter_code># sales histogram - not considering when sales is zero

ax = sns.histplot(data=df_numeric[df_numeric['sales'] > 0], x='sales', stat='proportion', bins=100)
ax.figure.set_size_inches(14, 7)
ax.set_title('Sales Histogram', fontsize=20, pad=10)

median = np.median(df_numeric['sales'])
ax.vlines(x=5744, ymin=0, ymax=0.07, linestyles='dashed', label='median', colors='firebrick')
ax.annotate(f'median = {median}', xy=(7000, 0.061), fontsize=14, color='firebrick')

ax;<jupyter_output><empty_output><jupyter_text>1.7.2. Categorical Attributes<jupyter_code># verifying unique valuesfor each categorical attribute
df_categorical.apply(lambda x: len(x.unique()))<jupyter_output><empty_output><jupyter_text>**BOXPLOT OF CATEGORICAL ATTRIBUTES**<jupyter_code># Boxplot -- state_holiday

# not considering when: sales = 0 
aux = df1[df1['sales'] > 0]

plt.figure(figsize=(24,10))

plt.subplot(1, 3, 1)
ax1 = sns.boxplot(x='state_holiday', y='sales', data=aux)
ax1.set_title('Boxplot - state_holiday', fontsize=18, pad=10)
ax1.set_xticklabels(labels=['None', 'Public', 'Easter', 'Christmas'])

plt.subplot(1, 3, 2)
ax2 = sns.boxplot(x='store_type', y='sales', data=aux)
ax2.set_title('Boxplot - store_type', fontsize=18, pad=10)

plt.subplot(1, 3, 3)
ax3 = sns.boxplot(x='assortment', y='sales', data=aux)
ax3.set_title('Boxplot - assortment', fontsize=18, pad=10)

plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup