<jupyter_start><jupyter_text>Client Churn Prediction--- The Top Bank company operates in Europe with a bank account as the main product, this product can keep client's salary and make payments. This account doesn't have any cost in the first 12 months, however, after that time trial, the client needs to rehire the bank for upcoming 12 months and redo this process every year. Recently the Analytics Team noticed that the churn rate is increasing.As a Data Science Consultant, you need to create an action plan to decrease the number of churn customers and show the financial return on your solution.At the end of your consultancy, you need to deliver to the TopBottom CEO a model in production, which will receive a customer base via API and return that same base with an extra column with the probability of each customer going into churn.In addition, you will need to provide a report reporting your model's performance and the financial impact of your solution. Questions that the CEO and the Analytics team would like to see in their report:1.  What is Top Bank's current Churn rate? How does it vary monthly?2. What is the performance of the model in classifying customers as churns?3. What is the expected return, in terms of revenue, if the company uses its model to avoid churn from customers? > Disclaimer: This is a fictional bussiness case  PLANNING  Input- Predict wheter customer will be in churn;- Dataset with sales records and customers info.  Output- Which customer will be in churn;- Churn rate of the company;- Performance of the model;- Action plan  Tasks 1. Which customer will be in churn:    - What is the criterion?        - Downtime        - Time remaining until the contract ends2. Current churn rate of the company:    - Calculate churn rate    - Calculate monthly churn rate and variation3. Performance of the model:    - TopK score    - Precision at K score    - Recall at K score4. Action plan:    - Discount    - Voucher    - Deposit bonus  IMPORTS<jupyter_code>import pickle
import warnings
import inflection



import numpy                    as np
import pandas                   as pd
import seaborn                  as sns
import scikitplot               as skplt

from IPython.core.display                   import HTML
from IPython.core.display                   import Image
from boruta                                 import BorutaPy
from sklearn.model_selection                import train_test_split
from imblearn.combine                       import SMOTETomek 
from xgboost                                import XGBClassifier, DMatrix
from sklearn.ensemble                       import RandomForestClassifier
from sklearn.linear_model                   import LogisticRegression
from sklearn.neighbors                      import KNeighborsClassifier
from sklearn.preprocessing                  import MinMaxScaler, RobustScaler<jupyter_output><empty_output><jupyter_text>HELPER FUNCTIONS<jupyter_code>def numerical_metrics(data):
    """
    Shows the main values ​​for descriptive statistics in numerical variables.

    Args:
        data ([float64 and int64]): Insert all numerical attributes in the dataset

    Returns:
        [dataframe]: A dataframe with mean, median, std deviation, skewness, kurtosis, min, max and range
    """
    
    
    data_mean = pd.DataFrame(data.apply(np.mean)).T
    data_median = pd.DataFrame(data.apply(np.median)).T
    data_std = pd.DataFrame(data.apply(np.std)).T
    data_min = pd.DataFrame(data.apply(min)).T
    data_max = pd.DataFrame(data.apply(max)).T
    data_range = pd.DataFrame(data.apply(lambda x: x.max() - x.min())).T
    data_q1 = pd.DataFrame(data.apply(lambda x: np.quantile(x, .25) )).T
    data_q3 = pd.DataFrame(data.apply(lambda x: np.quantile(x, .75) )).T
    data_skew = pd.DataFrame(data.apply(lambda x: x.skew())).T
    data_kurtosis = pd.DataFrame(data.apply(lambda x: x.kurtosis())).T
    num_attributes = pd.concat([data_min,data_max,data_range,data_mean,data_median, data_q1, data_q3,data_std,data_skew,data_kurtosis]).T.reset_index()
    num_attributes.columns = ['attributes','min','max','range','mean','median','q1','q3', 'st deviation','skewness','kurtosis']

    return num_attributes


def categorical_metrics(data, col):
    """
    Shows the the absolute and percent values in categorical variables.

    Args:
        data ([object]): Insert all categorical attributes in the dataset

    Returns:
        [dataframe]: A dataframe with absolute and percent values
    """
    


    return pd.DataFrame({'absolute': data[col].value_counts(), 'percent %': data[col].value_counts(normalize = True) * 100 })

def plot_label(title, xlabel, ylabel, fontsize):
    plt.title(title, fontsize = 18)
    plt.xlabel(xlabel, fontsize = fontsize)
    plt.ylabel(ylabel, fontsize = fontsize)
    
def jupyter_settings():
    %matplotlib inline
    %pylab inline
    plt.style.use( 'tableau-colorblind10' )
    plt.rcParams['figure.figsize'] = [25, 12]
    plt.rcParams['font.size'] = 24
    display( HTML( '<style>.container { width:100% !important; }</style>') )
    pd.options.display.max_columns = None
    pd.options.display.max_rows = None
    pd.set_option( 'display.expand_frame_repr', False )
    sns.set()
    
jupyter_settings()


def correlation_matrix(data, id, method):
    """Generates a correlation matrix of numerical variables

    Args:
        data ([DataFrame]): [The dataframe of the EDA]
        id ([int]): [The customer reference number]
        method ([string]): [The method used, it can be ‘pearson’, ‘kendall’ or ‘spearman’]

    Returns:
        [Image]: [The correlation matrix plot made with seaborn]
    """
    
    # correlation
    num_attributes = data.select_dtypes(include = ['int64', 'float64'])
    num_attributes = num_attributes.drop(columns = id )
    correlation = num_attributes.corr(method = method)

    # mask
    mask = np.zeros_like(correlation)
    mask = np.triu(np.ones_like(correlation, dtype=np.bool))

    # plot
    ax = sns.heatmap(correlation, mask = mask, fmt = '.2f', vmin = -1, vmax = 1, annot = True, cmap = 'magma', square = True)

    return ax
def sum_of_na(data):
    return pd.DataFrame({'Sum of NA': data.isna().sum(), '% NA': data.isna().sum()/data.shape[0]})


# ignoring errors
warnings.filterwarnings('ignore')

# round
pd.options.display.float_format = '{:.3f}'.format

seed = 42<jupyter_output><empty_output><jupyter_text>DATA This dataset is avaliable in: https://www.kaggle.com/mervetorkan/churndataset- **Data fields**- **RowNumber**: the number of the columns- **CustomerID**: unique identifier of clients- **Surname**: client's last name- **CreditScore**: clien'ts credit score for the financial market- **Geography**: the country of the client- **Gender**: the gender of the client- **Age**: the client's age- **Tenure**: number of years the client is in the bank - **Balance**: the amount that the client has in their account - **NumOfProducts**: the number of products that the client bought - **HasCrCard**: if the client has a credit card - **IsActiveMember**: if the client is active (within the last 12 months) - **EstimateSalary**: estimative of anual salary of clients - **Exited**: if the client is a churn (*target variable*)<jupyter_code>homepath = '/home/gutto/repos/pa003_client_churn/'
df_raw = pd.read_csv(homepath + 'data/raw/churn.csv', low_memory = False)
pickle.dump(df_raw, open(homepath + 'data/processed/df_raw.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>1. DATA DESCRIPTION - **Tasks**    - Dataset First Look    - Organizing Columns    - Data Overview    - Check Zero Values    - Check Data Types    - Descriptive Statistics  1.1 Dataset First Look<jupyter_code>df1 = pickle.load(open(homepath + 'data/processed/df_raw.pkl', 'rb'))
df1.head().T
df1.duplicated('CustomerId').sum()<jupyter_output><empty_output><jupyter_text>1.2 Organizing Columns These columns will not be used<jupyter_code>#Remove columns
df1 = df1.drop(['CustomerId', 'RowNumber', 'Surname'], axis = 1)
# changing to snakecase type
cols_old = df1.columns
snakecase = lambda x: inflection.underscore(x)

cols_new = list(map(snakecase, cols_old))
df1.columns = cols_new
df1.columns
df1 = df1.reindex(columns=['is_active_member', 'exited', 'has_cr_card', 'credit_score', 'num_of_products', 'estimated_salary', 'balance', 'tenure', 'age', 'gender', 'geography'])
df1.head()<jupyter_output><empty_output><jupyter_text>1.3 Data Overview<jupyter_code>print(f'Number of rows: {df1.shape[0]} \nNumber of columns: {df1.shape[1]}')
df1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   is_active_member  10000 non-null  int64  
 1   exited            10000 non-null  int64  
 2   has_cr_card       10000 non-null  int64  
 3   credit_score      10000 non-null  int64  
 4   num_of_products   10000 non-null  int64  
 5   estimated_salary  10000 non-null  float64
 6   balance           10000 non-null  float64
 7   tenure            10000 non-null  int64  
 8   age               10000 non-null  int64  
 9   gender            10000 non-null  object 
 10  geography         10000 non-null  object 
dtypes: float64(2), int64(7), object(2)
memory usage: 859.5+ KB<jupyter_text>is_active_member, exited and has_cr_card should be strings There's no NA in this dataset, however, there's a possibility that there are zeros in place of NAs  1.4 Check Zero Values<jupyter_code>df1.head()
df1_verify = df1[['credit_score', 'num_of_products', 'estimated_salary', 'balance', 'age']]

for col in df1_verify.columns:
    df1_verify[col] = df1_verify[col].apply(lambda x: '0' if x ==0 else 'another value')
df1_verify.apply(lambda x: x.unique())
df1_verify['balance'].value_counts()<jupyter_output><empty_output><jupyter_text>The balance columns has more than 1/3 of zero values, it doesn't mean that they're missing values but the DS team should get more information about this  1.5 Change Data Types<jupyter_code>col_cat = ['has_cr_card', 'is_active_member', 'exited']

for col in col_cat:
    df1[col] = df1[col].apply(lambda x: 'yes' if x == 1 else 'no')<jupyter_output><empty_output><jupyter_text>1.6 Descriptive Statistics  1.6.1 Numerical Attributes<jupyter_code>num_attributes = df1.select_dtypes(include = ['int64', 'float64'])
num_cols = num_attributes.columns.tolist()
num_attributes_result = numerical_metrics(num_attributes[num_cols])
num_attributes_result<jupyter_output><empty_output><jupyter_text>1.6.2 Outliers<jupyter_code>i = 1
for col in df1[num_cols]:
    plt.subplot(3, 3, i)
    ax = sns.boxplot(data = df1, x = col)
    i += 1<jupyter_output><empty_output><jupyter_text>1.6.3 Categorical Attributes<jupyter_code>categorical_attributes = df1.select_dtypes(exclude = ['int64', 'float64'])
categorical_attributes.columns
categorical_metrics(categorical_attributes, 'is_active_member')
categorical_metrics(categorical_attributes, 'has_cr_card')
categorical_metrics(categorical_attributes, 'exited')
categorical_metrics(categorical_attributes, 'gender')
categorical_metrics(categorical_attributes, 'geography')<jupyter_output><empty_output><jupyter_text>Important informations:- There are outliers in **credit_score, num_of_products and age**- The **churn ratio is 20.37%**;- **70.6%** of the members **has credit card**;- More than **50% of the clients** are **from France**  1.7 Save State<jupyter_code>pickle.dump(df1, open(homepath + 'data/processed/df1.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>2. FEATURE ENGINEERING - **Tasks**    - Dataset Checkpoint    - Features Creatrion    - Save State  2.1 Dataset Checkpoint<jupyter_code>df2 = pickle.load(open(homepath + 'data/processed/df1.pkl', 'rb'))<jupyter_output><empty_output><jupyter_text>2.2 Features Creation<jupyter_code>df2.columns<jupyter_output><empty_output><jupyter_text>1. Creating `avg_salary`:<jupyter_code>avg_salary = np.mean(df2.loc[:, 'estimated_salary'].values)
df2['avg_salary'] = avg_salary<jupyter_output><empty_output><jupyter_text>2. Creating `avg_credit_score`:<jupyter_code>avg_credit_score = np.mean(df2.loc[:, 'credit_score'].values)
df2['avg_credit_score'] = avg_credit_score<jupyter_output><empty_output><jupyter_text>3. Creating `avg_balance`:<jupyter_code>avg_balance = np.mean(df2.loc[:, 'balance'].values)
df2['avg_balance'] = avg_balance<jupyter_output><empty_output><jupyter_text>4. Creating `avg_tenure`:<jupyter_code>avg_tenure = np.mean(df2.loc[:, 'tenure'].values)
df2['avg_tenure'] = avg_tenure<jupyter_output><empty_output><jupyter_text>5. Creating `estimated_salary_per_credit_score`:<jupyter_code>estimated_salary_cs = df2.loc[:, ['credit_score', 'estimated_salary']].groupby('credit_score').mean().reset_index()
estimated_salary_cs.columns = ['credit_score', 'estimated_salary_per_credit_score']

# merge
df2 = pd.merge(df2, estimated_salary_cs, on = 'credit_score', how = 'left')<jupyter_output><empty_output><jupyter_text>6. Creating `estimated_salary_per_gender`:<jupyter_code>estimated_salary_gender = df2.loc[:, ['gender', 'estimated_salary']].groupby('gender').mean().reset_index()
estimated_salary_gender.columns = ['gender', 'estimated_salary_per_gender']

# merge
df2 = pd.merge(df2, estimated_salary_gender, on = 'gender', how = 'left')<jupyter_output><empty_output><jupyter_text>7. Creating `estimated_salary_per_country`:<jupyter_code>estimated_salary_country = df2.loc[:, ['geography', 'estimated_salary']].groupby('geography').mean().reset_index()
estimated_salary_country.columns = ['geography', '_estimated_salary_per_country']

# merge
df2 = pd.merge(df2, estimated_salary_country, on = 'geography', how = 'left')<jupyter_output><empty_output><jupyter_text>8. Creating `balance_per_age`:<jupyter_code>balance_age = df2.loc[:, ['age', 'balance']].groupby('age').mean().reset_index()
balance_age.columns = ['age', 'balance_per_age']

# merge
df2 = pd.merge(df2, balance_age, on = 'age', how = 'left')<jupyter_output><empty_output><jupyter_text>9. Creating `balance_per_country`:<jupyter_code>balance_country = df2.loc[:, ['geography', 'balance']].groupby('geography').mean().reset_index()
balance_country.columns = ['geography', 'balance_per_country']

# merge
df2 = pd.merge(df2, balance_country, on = 'geography', how = 'left')<jupyter_output><empty_output><jupyter_text>10. Creating `balance_per_num_of_products`:<jupyter_code>balance_number_of_products = df2.loc[:, ['num_of_products', 'balance']].groupby('num_of_products').mean().reset_index()
balance_number_of_products.columns = ['num_of_products', 'balance_per_num_of_products']

# merge
df2 = pd.merge(df2, balance_number_of_products, on = 'num_of_products', how = 'left')<jupyter_output><empty_output><jupyter_text>11. Creating LTV:<jupyter_code>balance_tenure = df2.loc[:, ['tenure', 'balance']].groupby('tenure').mean().reset_index()
balance_tenure.columns = ['tenure', 'LTV']

# merge
df2 = pd.merge(df2, balance_tenure, on = 'tenure', how = 'left')<jupyter_output><empty_output><jupyter_text>12. Creating estimated_salary_credit_card:<jupyter_code>estimated_salary_credit_card = df2.loc[:, ['has_cr_card', 'estimated_salary']].groupby('has_cr_card').mean().reset_index()
estimated_salary_credit_card.columns = ['has_cr_card', 'estimated_salary_credit_card']

# merge
df2 = pd.merge(df2, estimated_salary_credit_card, on = 'has_cr_card', how = 'left')<jupyter_output><empty_output><jupyter_text>13. Creating tenure_per_country:<jupyter_code>tenure_country = df2.loc[:, ['geography', 'tenure']].groupby('geography').mean().reset_index()
tenure_country.columns = ['geography', 'tenure_per_country']

# merge
df2 = pd.merge(df2, tenure_country, on = 'geography', how = 'left')<jupyter_output><empty_output><jupyter_text>14. Creating tenure_per_num_of_products:<jupyter_code>tenure_num_of_products = df2.loc[:, ['num_of_products', 'tenure']].groupby('num_of_products').mean().reset_index()
tenure_num_of_products.columns = ['num_of_products', 'tenure_per_num_of_products']

# merge
df2 = pd.merge(df2, tenure_num_of_products, on = 'num_of_products', how = 'left')<jupyter_output><empty_output><jupyter_text>15. Creating num_of_products_per_age:<jupyter_code>num_of_products_age = df2.loc[:, ['age', 'num_of_products']].groupby('age').mean().reset_index()
num_of_products_age.columns = ['age', 'num_of_products_per_age']

# merge
df2 = pd.merge(df2, num_of_products_age, on = 'age', how = 'left')<jupyter_output><empty_output><jupyter_text>16. Creating num_of_products_per_country:<jupyter_code>num_of_products_country = df2.loc[:, ['geography', 'num_of_products']].groupby('geography').mean().reset_index()
num_of_products_country.columns = ['geography', 'num_of_products_per_country']

# merge
df2 = pd.merge(df2, num_of_products_country, on = 'geography', how = 'left')
verify_nan = pd.DataFrame({'Sum of NA': df2.isna().sum(), '% NA': df2.isna().sum()/df2.shape[0]})
verify_nan
df2.sample()<jupyter_output><empty_output><jupyter_text>2.3 Save State<jupyter_code>pickle.dump(df2, open(homepath + 'data/processed/df2.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>3. DATA FILTERING - **Tasks**    - Dataset Checkpoint    - Features    - Save State  3.1 Dataset Checkpoint<jupyter_code>df3 = pickle.load(open(homepath + 'data/processed/df2.pkl', 'rb'))<jupyter_output><empty_output><jupyter_text>3.3 Save State<jupyter_code>pickle.dump(df3, open(homepath + 'data/processed/df3.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>4.0 EXPLORATORY DATA ANALYSIS - **Tasks**    - Univariate Analysis    - Bivariate Analysis    - Multivariate Analysis  4.1 Dataset Checkpoint<jupyter_code>df4 = pickle.load(open(homepath + 'data/processed/df3.pkl', 'rb'))<jupyter_output><empty_output><jupyter_text>4.3 Univariate Analysis  4.4 Bivariate Analysis  4.5 Multivariate Analysis  4.6 Save State<jupyter_code>pickle.dump(df4, open(homepath + 'data/processed/df4.pkl', 'wb'))<jupyter_output><empty_output><jupyter_text>5.0 DATA PREPARATION  - **Tasks**    - Dataset Checkpoint    - Encoding    - Rescaling    - Split Data    - Balancing Data    - Feature Selection    - Save State  5.1 Dataset Checkpoint<jupyter_code>df5 = pickle.load(open(homepath + 'data/processed/df4.pkl', 'rb'))
df5.drop(['avg_salary', 'avg_credit_score', 'avg_balance', 'avg_tenure'], axis = 1, inplace = True)<jupyter_output><empty_output><jupyter_text>5.2 Encoding A form to transform categorical variables into numerical<jupyter_code># label encoding 
gender_encoding = {'Male': 0 , 'Female': 1}
df5['gender'] = df5['gender'].map(gender_encoding)

questions_encoding = {'yes': 1,'no': 0}
df5['is_active_member'] = df3['is_active_member'].map(questions_encoding)
df5['has_cr_card'] = df5['has_cr_card'].map(questions_encoding)
df5['exited'] = df5['exited'].map(questions_encoding)

# one hot encoding encoding 
df5 = pd.get_dummies(df5, prefix=['country'], columns=['geography'])
df5 = pd.get_dummies(df5, prefix=['gender'], columns=['gender'])
df5.columns<jupyter_output><empty_output><jupyter_text>5.3 Rescaling [Reference](https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02)<jupyter_code>i = 1
for col in df5.columns:
    plt.subplot(8, 4, i)
    sns.kdeplot(df5[col])
    i += 1
df5.columns
# Min-Max Scaler
mms = MinMaxScaler()

min_max_scaler = ['is_active_member', 'exited', 'has_cr_card', 'estimated_salary', 'balance', 'tenure',
'estimated_salary_per_gender', '_estimated_salary_per_country', 'balance_per_country', 'LTV', 'estimated_salary_credit_card',
'tenure_per_country', 'country_France', 'country_Germany', 'country_Spain', 'gender_0', 'gender_1']

for col in min_max_scaler:
    df5[col] = mms.fit_transform(df5[[col]].values)

# Robust Scaler
rs = RobustScaler()

robust_scaler = ['credit_score', 'num_of_products', 'age', 'estimated_salary_per_credit_score', 'balance_per_age',
'balance_per_num_of_products', 'tenure_per_num_of_products', 'num_of_products_per_age',
'num_of_products_per_country']

for col in robust_scaler:
    df5[col] = mms.fit_transform(df5[[col]].values)
i = 1
for col in df5.columns:
    plt.subplot(8, 4, i)
    sns.kdeplot(df5[col])
    i += 1<jupyter_output><empty_output><jupyter_text>5.4 Split Data The target is **exited**<jupyter_code>X = df5.drop(['exited'], axis = 1)
#target
y = df5['exited']

X_train_imbalanced, X_test, y_train_imbalanced, y_test = train_test_split(X , y, test_size = 0.33, shuffle = True, random_state = seed)<jupyter_output><empty_output><jupyter_text>5.5 Balancing Data In this dataset, **the proportion of churn clients is 20.4% and the the rest is not churn clients**. The target variable is imbalanced, a problem for Machine Learning Algorithms that demand **balanced data for a good performance**.[Reference](https://towardsdatascience.com/applying-smote-for-class-imbalance-with-just-a-few-lines-of-code-python-cdf603e58688)<jupyter_code>balancer = SMOTETomek(sampling_strategy = 'minority', random_state = seed, n_jobs = 1)

X_train, y_train = balancer.fit_resample(X_train_imbalanced, y_train_imbalanced)
# creating subplots
fig, (ax1, ax2) = plt.subplots(1, 2)

ax = sns.countplot(x = y_train_imbalanced, ax = ax1, palette = 'Set2').set_title('Imbalanced Data')
ax = sns.countplot(x = y_train, ax = ax2, palette = 'Set2').set_title('Balanced Data')<jupyter_output><empty_output><jupyter_text>5.6 Feature Selection<jupyter_code>rf = RandomForestClassifier(n_estimators = 500, n_jobs = -1, random_state = seed)

boruta = BorutaPy(rf, n_estimators = 'auto', verbose = 2, random_state = seed).fit(X_train.values, y_train.values)
cols_selected = boruta.support_.tolist()

cols_selected_boruta = X_train.iloc[:, cols_selected].columns.to_list()

cols_not_selected_boruta = list( np.setdiff1d(X_train.columns, cols_selected_boruta))
cols_selected_boruta
cols_not_selected_boruta
X_train.columns
cols_drop = ['_estimated_salary_per_country', 'balance_per_country', 'country_France', 'country_Germany', 'country_Spain', 'estimated_salary_credit_card', 'estimated_salary_per_gender',
'gender_0', 'gender_1', 'has_cr_card', 'is_active_member', 'num_of_products_per_country',
'tenure_per_country']
X_train.drop(cols_drop, axis=1, inplace=True)
X_test.drop(cols_drop, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>6.0 MACHINE LEARNING MODELLING  - **Tasks**    -<jupyter_code>#model definition
rf = RandomForestClassifier(n_estimators = 300, random_state = seed)

#model training
rf.fit(X_train, y_train)

#model prediction
yhat_rf = rf.predict_proba(X_test)

# Accumulative gain
skplt.metrics.plot_cumulative_gain(y_test, yhat_rf);
plt.title( 'Random Forest', fontsize = 18);<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup