<jupyter_start><jupyter_text>Bank Marketing Campaign- Predicting Term Deposit (CRISP-DM)  Problem Definition:  One of the most common marketing strategy In Banking sector is direct marketing campaigns through phone calls ,it is a form of advertising that allows organizations to communicate directly with customers to offer their services based on the client’s existing bank profile .Here we will consider term deposit as a banking service .Direct marketing has significant challenges:* The increasing number of marketing campaigns over time has reduced their effects on the general public.* According to study positive response rates of direct marketing tend to be around 1-3% * Startegy like Telemarketing may have high financial and resource costs, but less ROI * Many people find direct marketing annoying and intrusive.* If we reach a consumer who isn't interested in services, they are likely to find it irritating and this can create a negative brand association.  Business Goal : to build a list of target customers who are likey to subscribe a term deposite. The more targeted our campaigns, the more successful they are likely to be.  Project Objective : We will convert this problem into a machine learning classification problem .We will build a model to predict whether a client will subscribe a term deposit or not so that the banks can arrange a better management of available resources by focusing on the potential customers “predicted” by the classifier .This will help them to design a more efficient and precise campaign strategy to reduce the costs,improve the profits and customer satisfaction . Data Mining Technique to be used : Classification   Data Set Information:The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed.source : uci machine learning repository , link: http://archive.ics.uci.edu/ml/datasets/Bank+Marketing   Attribute Information:**Bank client data:*** Age (numeric)* Job : type of job (categorical: 'admin.', 'blue-collar', 'entrepreneur', 'housemaid', 'management', 'retired', 'self-employed', 'services', 'student', 'technician', 'unemployed', 'unknown')* Marital : marital status (categorical: 'divorced', 'married', 'single', 'unknown' ; note: 'divorced' means divorced or widowed)* Education (categorical: 'basic.4y', 'basic.6y', 'basic.9y', 'high.school', 'illiterate', 'professional.course', 'university.degree', 'unknown')* Default: has credit in default? (categorical: 'no', 'yes', 'unknown')* Housing: has housing loan? (categorical: 'no', 'yes', 'unknown')* Loan: has personal loan? (categorical: 'no', 'yes', 'unknown')**Related with the last contact of the current campaign:*** Contact: contact communication type (categorical:  'cellular','telephone')* Month: last contact month of year (categorical: 'jan', 'feb', 'mar',   …, 'nov', 'dec')* Dayofweek: last contact day of the week (categorical:  'mon','tue','wed','thu','fri')* Duration: last contact duration, in seconds (numeric). Important  note: this attribute highly affects the output target (e.g., if  duration=0 then y='no'). Yet, the duration is not known before a call  is performed. Also, after the end of the call y is obviously known.  Thus, this input should only be included for benchmark purposes and  should be discarded if the intention is to have a realistic  predictive model.**Other attributes:***  Campaign: number of contacts performed during this campaign and for    this client (numeric, includes last contact)*  Pdays: number of days that passed by after the client was last   contacted from a previous campaign (numeric; 999 means client was not   previously contacted)*  Previous: number of contacts performed before this campaign and for   this client (numeric)* Poutcome: outcome of the previous marketing campaign (categorical:  'failure','nonexistent','success')**Social and economic context attributes*** Emp.var.rate: employment variation rate - quarterly indicator, it defines as a measure of the extent to which available labour resources (people available to work) are being used.  (numeric)* Cons.price.idx: consumer price index - monthly indicator (numeric), it expresses the change in the current prices of the market basket in terms of the prices during the same month in the previous year. * Cons.conf.idx: consumer confidence index - monthly indicator , CCI is a survey administered by The Conference Board, that measures how optimistic or pessimistic consumers are regarding their expected financial situation  (numeric)* Euribor3m: euribor 3 month rate - daily indicator (numeric), it is the interest rate at which a selection of European banks lend one another funds denominated in euros whereby the loans have a maturity of 3 months  Nr.employed: number of employees - quarterly indicator (numeric)  **Output variable (desired target):*** y - has the client subscribed a term deposit? (binary: 'yes', 'no')  Preliminary Data Analysis<jupyter_code>import pandas as pd    
import seaborn as sns     
import numpy as np        
import matplotlib.pyplot as plt     
from scipy import stats       
df=pd.read_csv(r'D:\Linear Regression\bank-additional_bank-additional-full.csv')
df.head()
df.shape
df.info()
df.isnull().sum()
# no null value present in the data set 
df.nunique()
df.describe()
df.describe(include='object')
# renameing 'y' as 'target'
df=df.rename(columns={'y':'target'})
df
df.target.value_counts()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis   Visualizing Categorical Variables<jupyter_code>sns.set_style('whitegrid')
plt.figure(figsize=(6,4))
sns.countplot(x='target',data=df)
plt.figure(figsize=(6,4))
sns.countplot(x='marital',data=df)
plt.figure(figsize=(10,6))
sns.countplot(x='education',data=df)
plt.xticks(rotation=45, horizontalalignment='right')
plt.figure(figsize=(12,6))
sns.countplot(x="job",data=df)
plt.xticks(rotation=45, horizontalalignment='right')
fig,(ax1,ax2,ax3) =plt.subplots(nrows = 1, ncols = 3,figsize=(14,6))
sns.countplot(x="loan",data=df,ax=ax1)
ax1.set_xlabel('personal loan')
sns.countplot(x="housing",data=df,ax=ax2)
ax2.set_xlabel('house loan')
sns.countplot(x="default",data=df,ax=ax3)
ax3.set_xlabel('default')
fig,ax=plt.subplots()
sns.countplot(x="poutcome",data=df)
ax.set_xlabel('previous outcome')
fig,ax=plt.subplots()
sns.countplot(x="contact",data=df)
ax.set_xlabel("contcat type")
plt.figure(figsize=(10,6))
sns.countplot(x="month",data=df)
plt.figure(figsize=(8,4))
sns.countplot(x="day_of_week",data=df)<jupyter_output><empty_output><jupyter_text>Visualizing Numerical Variable<jupyter_code>plt.figure(figsize=(8,6))
sns.distplot(df['age'],bins=40)
df.age.describe()
plt.figure(figsize=(8,6))
fig = sns.boxplot(x='age', data=df,color='royalblue')
plt.figure(figsize=(8,6))
sns.distplot(df['duration'],bins=40)
df.duration.describe()
plt.figure(figsize=(8,6))
fig = sns.boxplot(x='duration', data=df,color='royalblue')
#converting target variable in 0 for no and 1 for yes
df['target']=df['target'].replace({'no':0,'yes':1})
df.target.value_counts()<jupyter_output><empty_output><jupyter_text>Bivariate analysis<jupyter_code>plt.figure(figsize=(10,6))
sns.barplot(x='job', y='target', data=df)
plt.title('job vs term deposit(likey to subscribe)',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
df1 = df.groupby('job').agg({'target':'sum'})
plt.figure(figsize=(10,6))
sns.barplot(x = df1.index, y = df1.target, data = df1)
plt.title('job vs term deposit (total count)',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
df.job.value_counts()
plt.figure(figsize=(10,6))
sns.barplot(x='marital', y='target', data=df)
plt.title('marital vs term deposit(likey to subscribe)',fontsize=16)
df1 = df.groupby('marital').agg({'target':'sum'})
plt.figure(figsize=(10,6))
sns.barplot(x = df1.index, y = df1.target, data = df1)
plt.title('marital vs term deposit (total count)',fontsize=16)
df.marital.value_counts()
plt.figure(figsize=(10,6))
sns.barplot(x='education', y='target', data=df)
plt.title('education vs term deposit(likey to subscribe)',fontsize=16)
df1 = df.groupby('education').agg({'target':'sum'})
plt.figure(figsize=(10,6))
sns.barplot(x = df1.index, y = df1.target, data = df1)
plt.title('education vs term deposit',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
df.education.value_counts()
plt.figure(figsize=(10,6))
sns.barplot(x='job', y='target', hue='marital',data=df)
plt.title('job vs marital vs term deposit(likey to subscribe)',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
plt.figure(figsize=(10,6))
sns.barplot(x='job', y='target',hue='housing', data=df)
plt.title('job vs housing vs term deposit(likey to subscribe)',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
plt.figure(figsize=(10,6))
sns.barplot(x='job', y='target',hue='loan', data=df)
plt.title('job vs loan vs term deposit(likey to subscribe)',fontsize=16)
plt.xticks(rotation=45, horizontalalignment='right')
plt.figure(figsize=(10,6))
sns.barplot(x='month', y='target', data=df)
plt.title('month vs term deposit(likey to subscribe)',fontsize=16)
df1 = df.groupby('month').agg({'target':'sum'})
plt.figure(figsize=(10,6))
sns.barplot(x = df1.index, y = df1.target, data = df1)
plt.title('month vs term deposit (total)',fontsize=16)
df.month.value_counts()
plt.figure(figsize=(10,6))
sns.barplot(x='day_of_week', y='target', data=df)
plt.title('day_of_week vs term deposit(likey to subscribe)',fontsize=16)
df1 = df.groupby('day_of_week').agg({'target':'sum'})
plt.figure(figsize=(10,6))
sns.barplot(x = df1.index, y = df1.target, data = df1)
plt.title('day_of_week vs term deposit (total)',fontsize=16)
df.info()
corrmat = df.corr()
fig, ax = plt.subplots(figsize=(12, 9))
sns.heatmap(corrmat,annot=True,)
plt.title('correlation matrix for numerical variables',fontsize=18)<jupyter_output><empty_output><jupyter_text>Observations :'emp.var.rate', 'cons.price.idx' , 'cons.conf.idx' , 'euribor3m' , and 'nr.employed' are highly correlated , we can not drop any of them without advise from domain expert ,may be we can do PCA on these varibales in later stage   Data Preparation We have to drop the 'duration' column beacuse the duration will be unknown before a call is performed so in training dataset we can not keep this, moreover after the the call anhow the outcome will be khnown<jupyter_code>df=df.drop('duration',axis=1)
df.shape
df
df.columns
df.pdays.value_counts()
# in pdays 39673 values are 999 which means client was not previously contacted, and the range for other values is
# 3-27 and count is significantly low , now we can not keep this coulmn like this ,it will add bias to our model , 
#so we will convert this coulm as categorical column 
df['pdays_cat'] =['cont' if each==999 else 'not_cont' for each in df.pdays]
df.head()
df.pdays_cat.value_counts()
df=df.drop(['pdays'], axis=1)
df.shape<jupyter_output><empty_output><jupyter_text>separating categorical variable<jupyter_code>cat_columns_df=df.select_dtypes(include=['object'])
cat_columns_df<jupyter_output><empty_output><jupyter_text>separating numerical value<jupyter_code>num_columns_df=df.select_dtypes(exclude=['object'])
num_columns_df
num_columns_df = num_columns_df.drop('target',axis=1)
num_columns_df.shape
num_columns_df.columns
#creating test data set 
df_test=df.target.astype(dtype='bool')
df_test.shape
df_test.dtype<jupyter_output><empty_output><jupyter_text>creating dummy variables form categorical varibales<jupyter_code>dummy_columns_df=pd.get_dummies(cat_columns_df,drop_first=True)
dummy_columns_df.shape<jupyter_output><empty_output><jupyter_text>scaling numerical variables<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
num_columns_scaled_df=pd.DataFrame(scaler.fit_transform(num_columns_df),columns=num_columns_df.columns)
num_columns_scaled_df<jupyter_output><empty_output><jupyter_text>Creating Train Data set<jupyter_code>df_train = pd.concat([num_columns_scaled_df, dummy_columns_df], axis=1,sort=False)
df_train
df_train.shape<jupyter_output><empty_output><jupyter_text>Train Test Spliting<jupyter_code>from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(df_train,df_test,test_size=0.2,random_state=101)
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)<jupyter_output>(32950, 52)
(8238, 52)
(32950,)
(8238,)<jupyter_text>Handling Imbalance Data<jupyter_code>from collections import Counter
print(Counter(y_train))<jupyter_output>Counter({False: 29269, True: 3681})<jupyter_text>Yes : 88.82 %No : 11.17 %  We will use Random Oversampling technique to balance the data , this method randomly selectes examples from the minority class, with replacement, and adding them to the training dataset.<jupyter_code>from imblearn.over_sampling import RandomOverSampler
RO = RandomOverSampler(sampling_strategy='minority')
X_train_os,y_train_os = RO.fit_sample(X_train,y_train)
X_train_os.shape,y_train_os.shape
from collections import Counter
print(Counter(y_train))
print(Counter(y_train_os))<jupyter_output>Counter({False: 29269, True: 3681})
Counter({False: 29269, True: 29269})<jupyter_text>Model and Evaluation   Model used * Logistic Regression * K-Nearest Neighbor* Decision Tree* Random Forest  Evaluation metrics :* Accuracy Score * F1 Score* Precision Score * Recall Score * ROC AUC Score  Evaluation method :* As this is a classification problem we will mainly consider the F1 and ROC     AUC score * Satisfactory Recall Score : Our objective will be reducing False Negetive       count,we can not miss out our potential customer * Satisfactory Precision Score : We have to bring Flase Positive count low so     that we save ourselves from wasting money and effort .<jupyter_code>from sklearn import metrics 
from sklearn.metrics import confusion_matrix,classification_report
from sklearn.metrics import roc_curve,accuracy_score,precision_score, recall_score
def classification_metrics(x,y) :
    accuracy = metrics.accuracy_score(x,y)
    F1_weighted= metrics.f1_score(x,y,average='weighted')
    F1_macro= metrics.f1_score(x,y,average='macro')
    precision = metrics.precision_score(x,y)
    recall=metrics.recall_score(x,y)
    cm= metrics.confusion_matrix(x,y)
    
    print('accuracy score is :',accuracy )
    print('F1 weighted score is :',F1_weighted )
    print('F1 macro score is :',F1_macro )
    print('precision score is :',precision )
    print('recall score is :',recall )
    sns.heatmap(cm,annot=True,cmap='Blues',fmt='g')
    plt.title('Confusion Matrix',fontsize=16)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
LR=LogisticRegression()
LR.fit(X_train_os,y_train_os)
y_predLR=LR.predict(X_test)
classification_metrics(y_test,y_predLR)
LR_probs = LR.predict_proba(X_test)[:,1]
NS_probs = [0 for _ in range(len(y_test))]
lr_fpr, lr_tpr, lr_threshold = roc_curve(y_test, LR_probs)
ns_fpr, ns_tpr, ns_threshold = roc_curve(y_test, NS_probs)
plt.plot(lr_fpr, lr_tpr, marker='.', label='Logistic')
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve',fontsize=16)
plt.legend()
plt.show()

print('AUC ROC score is :' ,metrics.roc_auc_score(y_test,y_predLR))<jupyter_output><empty_output><jupyter_text>Knn Classifier<jupyter_code>from  sklearn.neighbors import KNeighborsClassifier
scores = []
recall=[]
for k in range(1,25,2) :
    knn=KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train_os,y_train_os)
    y_pred=knn.predict(X_test)
    scores.append(metrics.recall_score(y_test,y_pred))
    recall.append(metrics.recall_score(y_test,y_pred))
plt.plot(range(1,25,2), scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Testing Accuracy')
#will consider the optimun k value as 14 
knn=KNeighborsClassifier(n_neighbors=14)
knn.fit(X_train_os,y_train_os)
y_predKNN=knn.predict(X_test)
classification_metrics(y_test,y_predKNN)
KNN_probs = knn.predict_proba(X_test)[:,1]
NS_probs = [0 for _ in range(len(y_test))]
knn_fpr, knn_tpr, _ = roc_curve(y_test, KNN_probs)
ns_fpr, ns_tpr, ns_threshold = roc_curve(y_test, NS_probs)
plt.plot(knn_fpr, knn_tpr, marker='.', label='Decision Tree')
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve',fontsize=16)
plt.legend()
plt.show()
print('AUC ROC score is :' ,metrics.roc_auc_score(y_test,y_predKNN))<jupyter_output><empty_output><jupyter_text>Decision Tree Classifier<jupyter_code>from sklearn.tree import DecisionTreeClassifier
DT=DecisionTreeClassifier()
DT.fit(X_train_os,y_train_os)
y_predDT=DT.predict(X_test)
classification_metrics(y_test,y_predDT)
DT_probs = DT.predict_proba(X_test)[:,1]
NS_probs = [0 for _ in range(len(y_test))]
dt_fpr, dt_tpr, _ = roc_curve(y_test, DT_probs)
ns_fpr, ns_tpr, ns_threshold = roc_curve(y_test, NS_probs)
plt.plot(dt_fpr, dt_tpr, marker='.', label='Decision Tree')
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve',fontsize=16)
plt.legend()
plt.show()
print('AUC ROC score is :' ,metrics.roc_auc_score(y_test,y_predDT))<jupyter_output><empty_output><jupyter_text>Random Forest Classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier
RF=RandomForestClassifier()
RF.fit(X_train_os,y_train_os)
y_predRF=RF.predict(X_test)
classification_metrics(y_test,y_predRF)
RF_probs = RF.predict_proba(X_test)[:,1]
NS_probs = [0 for _ in range(len(y_test))]
rf_fpr, rf_tpr, _ = roc_curve(y_test, RF_probs)
ns_fpr, ns_tpr, ns_threshold = roc_curve(y_test, NS_probs)
plt.plot(rf_fpr, rf_tpr, marker='.', label='Decision Tree')
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve',fontsize=16)
plt.legend()
plt.show()
print('AUC ROC score is :' ,metrics.roc_auc_score(y_test,y_predRF))<jupyter_output><empty_output><jupyter_text>The False Negative count for Logistic Regression is the least, also the Recall, F1, AUC ROC  scores are comparatively better for this model, we will try to manipulate the threshold value so that we can improve the Recall value .<jupyter_code>LR_probs = LR.predict_proba(X_test)[:,1]
LR_probs
lr_threshold
recall_sc = []
for thres in lr_threshold:
    y_predLR = np.where(LR_probs>thres,1,0)
    recall_sc.append(metrics.recall_score(y_test, y_predLR))
    
df_score = pd.concat([pd.Series(lr_threshold), pd.Series(recall_sc)],axis=1)
df_score.columns = ['thresholds', 'recall_sc']
df_score.head
#we will try to increase the recall value upto 70 percent 
df_score = df_score[df_score['recall_sc']< 0.80]
df_score.sort_values(by='recall_sc', ascending=False, inplace=True)
df_score
#for threshold 0.40 the recall score is maximum 
thres=0.33
y_predLR_final = np.where(LR_probs>thres,1,0)
classification_metrics(y_test,y_predLR_final)
print('AUC ROC score is :' ,metrics.roc_auc_score(y_test,y_predLR_final))<jupyter_output>AUC ROC score is : 0.6868271045837147<jupyter_text>Though we were able to improve the recall lvalue and decrease the FN count but the FP count also increase heavily, so there is a trade-off ,now considering the business expectation we can take decision about this and tune the model ,<jupyter_code>### later on we will try to hypertune the knn,decison tree and random forest  model and will check the results<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup