<jupyter_start><jupyter_text>![](https://autonexa.com/wp-content/uploads/2018/08/INS-1-780x405.jpg)  Problem Statement Our client is an Insurance company that has provided Health Insurance to its customers now they need your help in building a model to predict whether the policyholders (customers) from past year will also be interested in Vehicle Insurance provided by the company.An insurance policy is an arrangement by which a company undertakes to provide a guarantee of compensation for specified loss, damage, illness, or death in return for the payment of a specified premium. A premium is a sum of money that the customer needs to pay regularly to an insurance company for this guarantee.For example, you may pay a premium of Rs. 5000 each year for a health insurance cover of Rs. 200,000/- so that if, God forbid, you fall ill and need to be hospitalised in that year, the insurance provider company will bear the cost of hospitalisation etc. for upto Rs. 200,000. Now if you are wondering how can company bear such high hospitalisation cost when it charges a premium of only Rs. 5000/-, that is where the concept of probabilities comes in picture. For example, like you, there may be 100 customers who would be paying a premium of Rs. 5000 every year, but only a few of them (say 2-3) would get hospitalised that year and not everyone. This way everyone shares the risk of everyone else.Just like medical insurance, there is vehicle insurance where every year customer needs to pay a premium of certain amount to insurance provider company so that in case of unfortunate accident by the vehicle, the insurance provider company will provide a compensation (called ‘sum assured’) to the customer.Building a model to predict whether a customer would be interested in Vehicle Insurance is extremely helpful for the company because it can then accordingly plan its communication strategy to reach out to those customers and optimise its business model and revenue.Now, in order to predict, whether the customer would be interested in Vehicle insurance, you have information about demographics (gender, age, region code type), Vehicles (Vehicle Age, Damage), Policy (Premium, sourcing channel) etc.  Bussiness Goal Building a model to predict whether a customer would be interested in Vehicle Insurance is extremely helpful for the company because it can then accordingly plan its communication strategy to reach out to those customers and optimise its business model and revenue.Now, in order to predict, whether the customer would be interested in Vehicle insurance, you have information about demographics (gender, age, region code type), Vehicles (Vehicle Age, Damage), Policy (Premium, sourcing channel) etc. Objective:To predict if an insurance policy holder would be interested to buy a vehicle insurance as well. Building a model to predict whether a customer would be interested in Vehicle Insurance is extremely helpful for the company because it can then accordingly plan its communication strategy to reach out to those customers and optimize its business model and revenue.The aim of this project is to leverage the machine learning algorithms such as Logistic Regression and Random Forest to create a predictive model using statistically significant variables from the given data set.Model accuracy will be assessed using different techniques such as ROC (Receiver operating characteristic), AUC (Area under the ROC curve) and Confusion Matrix. This Notebook will cover -  1. Exploratory Data Analysis 2. Data Modelling and Evaluation  Step 1: Perform data preparation & cleaning- Load the dataset into a data frame using Pandas- Explore the number of rows & columns, ranges of values etc.- Handle missing, incorrect and invalid data- Perform any additional steps (parsing dates, creating additional columns, merging multiple dataset etc.) Step 2: Perform exploratory analysis & visualization- Compute the mean, sum, range and other interesting statistics for numeric columns- Explore distributions of numeric columns using histograms etc.- Explore relationship between columns using scatter plots, bar charts etc.- Make a note of interesting insights from the exploratory analysis Step 3: Ask & answer questions about the data- Ask at least 4 interesting questions about your dataset- Answer the questions either by computing the results using Numpy/Pandas or by plotting graphs using Matplotlib/Seaborn- Create new columns, merge multiple dataset and perform grouping/aggregation wherever necessary- Wherever you're using a library function from Pandas/Numpy/Matplotlib etc. explain briefly what it does Step 4: Summarize your inferences & write a conclusion- Write a summary of what you've learned from the analysis- Include interesting insights and graphs from previous sections- Share ideas for future work on the same topic using other relevant datasets- Share links to resources you found useful during your analysis ---- The Structure of this notebook :* Initial Library imports * Exploring File : EdStatsCountries.csv  * Reading data and exploring which columns are necessary  * Geographic and Economic Special- Macro- country codes  * Grouping countries by Geographic and Economic groups ( Size-Heatmap )* Exploring File : EdStatsSeriies.csv  * Reading data and exploring which columns are necessary  * Understanding the indicator series :     * What about what topics are these indicators     * How many indicators per topic  * Utility Function : get_indicator_details() :     * This function return the details of an indicator as dictionary ,     * useful for retrieving definition and name during plots* Exploring File : EdStatsData.csv  * Reading data and exploring which columns are necessary  * Removing Columns for Future Years ,   * Removing Rows which are Empty  * Understanding the availability of Data for each Year    * Both before and after cleaning with Bar Plots* Utility Function : Comparative analysis  * get_pivot_similiarity():  * this function takes two countries as an input and calculates how similar they are based on all available indicators , from 1970 to 2015  * the calculation is based on formula :```  ( indicator_value[country1] - indicator_value[country2] ) /  ( indicator_value[country1] + indicator_value[country2] )```* Utility Function for full Report Generation   * generate_report():  * This function takes a list of countries and one indicator as Input   * It generates multiple different plot so that we can instantly get an idea of what is going on for that indicator  * the plots generated include :    * Line graph of values of that indicator for all countries in the list passed , over time from 1970 to 2015    * Pie Chart Plot for those counties for one particular year passed as input     * Similarity Heat Map - for visualizing which countries are more similar to each other , for each country pair inside the list of countries      * this function is based on the Utility function for Comparative analysis called : get_pivot_similiarity      * the output generated is similar to that of a correlation plot .* World Map Visualization :   * Dash Plotly library used for making a world map chloropleth plot   * Indicators such as Population , Population Growth and GDP(PPP) are visualized in this section* Case Study : BRICS Countries :   * BRICS Countries : Brazil Russia India China South-Africa  * Study of Population Growth   * Study of How the expenditure on education impacts the education outcomes  * Study of correlation between Educated, GDP and  Mortality * Case Study : Comparison between Education in India and America  * Study of Primary and Primary and Tertiary Education Outcomes and Expenditures  * Comparative study of Drop-Out rate   **Attribute Information** 1. id :	Unique ID for the customer2. Gender	: Gender of the customer3. Age :	Age of the customer4. Driving_License	0 : Customer does not have DL, 1 : Customer already has DL5. Region_Code :	Unique code for the region of the customer6. Previously_Insured	: 1 : Customer already has Vehicle Insurance, 0 : Customer doesn't have Vehicle Insurance7. Vehicle_Age :	Age of the Vehicle8. Vehicle_Damage	 :1 : Customer got his/her vehicle damaged in the past. 0 : Customer didn't get his/her vehicle damaged in the past.9. Annual_Premium	: The amount customer needs to pay as premium in the year10. PolicySalesChannel :	Anonymized Code for the channel of outreaching to the customer ie. Different Agents, Over Mail, Over Phone, In Person, etc.11. Vintage :	Number of Days, Customer has been associated with the company12. Response :	1 : Customer is interested, 0 : Customer is not interested  Import Libraries --- Installing and Importing Libraries  Setting up Google Drive Folder where Data is storedFor Graphing : * Matplotib* Seaborn * heatmapz* Dash-PlotlyDatawrangling : * Numpy* Pandas* JsonOutput Formatting :* pprint - Pretty Printer<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'

import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Import Dataset<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
filepath = '/content/drive/MyDrive/AlmaBetter/Capstone2_Supervised_Classification/Code/Data/main_data.csv'
data = pd.read_csv(filepath)
dataset=data.copy()
dataset['id'].nunique()/(len(dataset))
dataset.set_index('id', inplace= True)
dataset.head()
dataset.info()
dataset.isnull().sum()<jupyter_output><empty_output><jupyter_text>*  There is no missing value in the data*  By looking at the info of the dataset we can get a rough idea on the numeric and the string columns * By looking at the summary of the data we can infer the mean,standard deviation, min and max of the * We will be able to get a idea on the outliers here by the percentiles ( In the Annual_Premium the 99th percentile is 72963 and the   max is 540165 this represents the outliers in this column)<jupyter_code>dataset.describe(percentiles = [.25,.50,.75,.95,.99]).T
numerical_features = ['Age','Vintage','Annual_Premium', 'Vehicle_Age']
binary_features =  ['Gender', 'Driving_License', 'Previously_Insured','Vehicle_Damage']
categorical_features =['Region_Code','Policy_Sales_Channel']
target = ['Response']<jupyter_output><empty_output><jupyter_text>Data Preparation and Cleaning<jupyter_code>dataset['Annual_Premium'] = dataset['Annual_Premium'].astype('int64')
dataset['Policy_Sales_Channel'] = dataset['Policy_Sales_Channel'].astype('int64')
dataset['Region_Code'] = dataset['Region_Code'].astype('int64')

for column in categorical_features:
    dataset[column] = dataset[column].astype('str')
dataset.info()

dataset.head(10)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis   Light Data Exploration 1) For numeric data * Made histograms to understand distributions * Corrplot * Pivot table comparing survival rate across numeric variables  2) For Categorical Data * Made bar charts to understand balance of classes * Made pivot tables to understand relationship with survival<jupyter_code>pd.pivot_table( dataset, index='Response', values=numerical_features , aggfunc=['mean','median'] )  #Edit
pd.pivot_table( data, index='Response', values=categorical_features , aggfunc=['median'] )
dataset.groupby('Response').apply(lambda x : x[numerical_features].describe() )
dataset.groupby('Response').apply(lambda x : x[categorical_features].describe() )
pd.pivot_table( dataset, index='Response', values=binary_features , aggfunc='mean' )<jupyter_output><empty_output><jupyter_text>Variable wise EDA Most of the vehicles of customers with response 1 are between the ages of 1-2 and their vehicles are damaged.  Target Variable (Response)<jupyter_code>count_1 = dataset[dataset["Response"] == 1].value_counts().sum()
totalResponse = dataset["Response"].value_counts().sum()
print("The percentage of positive response in data is :", round(count_1*100/totalResponse),"%")
plt.subplot(1, 2, 1)
ax1=sns.countplot(dataset['Response'], palette="cool")
plt.title("Count of response (target variable)")

total=float(len(dataset)) 

plt.subplot(1,2,2)
count = dataset['Response'].value_counts()
count.plot.pie(autopct = '%1.1f%%',colors=['c','orange'], figsize = (10,7),explode = [0,0.1],title = "pie chart of Percentage of target class")

print( "Percentage of target class\n")
print(dataset['Response'].value_counts()/len(dataset)*100)


for p in ax1.patches:
    height = p.get_height()
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")<jupyter_output>Percentage of target class

0    87.743664
1    12.256336
Name: Response, dtype: float64<jupyter_text>1. By the plot we can say that this is the problem of imbalance binary classification problem2. The indivisuals interested is 87 % as compared to the othe one.   Age variable<jupyter_code>plt.figure(figsize = (15,3))
sns.countplot(dataset['Age'], palette = 'hsv')
plt.title('Count of Age')
plt.show()

f,ax = plt.subplots(nrows=1,ncols=2,figsize=(20,5))
axx = ax.flatten()
sns.distplot(dataset['Age'] ,ax = axx[0],color='Blue')
sns.boxplot(dataset['Age'],ax = axx[1],color='Orange')
sns.FacetGrid(dataset ,hue='Response',size=5) \
 .map(sns.distplot,"Age") \
 .add_legend();
plt.show()<jupyter_output><empty_output><jupyter_text>1. Count of the indivisuals with age 24 are greater in the dataset2. Variable Age looks like right skewed3. From the boxplot we observe that here is not serious outliers in the data  * Young people below 30 are not interested in vehicle insurance. Reasons could be lack of experience, less maturity level and they don't have expensive vehicles yet.* People aged between 30-60 are more likely to be interested.* From the boxplot we can see that there no outlier in the data.   Annual_Premium  * From the distribution plot we can infer that the annual premimum variable is right skewed* From the boxplot we can observe lot of outliers in the variable<jupyter_code>plt.figure(figsize=(13,7))
plt.subplot(2,1,1)
sns.distplot(dataset['Annual_Premium'], color='green')
plt.title("Distribution of Annual premium")
plt.show()
plt.figure(figsize=(13,7))
plt.subplot(2,1,2)
sns.boxplot(dataset['Annual_Premium'])
plt.title("boxplot of Annual premium")
plt.show()
# plt.figure(figsize = (13,5))
# plt.subplot(1,2,1)
# sns.countplot(dataset['Annual_Premium'])
# plt.title("count of male and female")

# plt.subplot(1,2,2)
# sns.countplot(dataset['Annual_Premium'] , hue = dataset['Response'],palette="rocket_r")
# plt.title("Response in Male and female category")
# plt.show()<jupyter_output><empty_output><jupyter_text>Gender variable<jupyter_code>plt.figure(figsize = (13,5))
plt.subplot(1,2,1)
plt.subplots_adjust(hspace = 5)

ax1 = sns.countplot(dataset['Gender'],palette="rocket_r")
plt.title("count of male and female")
total=float(len(dataset))
total_h=[]


plt.subplot(1,2,2)
ax2 = sns.countplot(dataset['Gender'], hue = dataset['Response'],palette="rocket_r")
plt.title("Response in Male and female category")

for i,p in enumerate(ax1.patches):
    height = p.get_height()
    total_h+=[height]
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")
total_h*=2
for i,p in enumerate(ax2.patches):
    height = p.get_height()
    ax2.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total_h[i]*100), ha="center")
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>1. The gender variable in the dataset is almost equally distributed2. Male category is slightly greater than that of female and chances of buying the insurance is also little high.    Driving License<jupyter_code>print("Percentage of  Driving_License feature\n ")
print(dataset['Driving_License'].value_counts()/len(dataset)*100)
f,ax = plt.subplots(nrows=1,ncols=2,figsize=(12,6))
axx = ax.flatten()
total_h=[]

plt.title("Count plot of Driving_License vs Response")
ax1 = sns.countplot(dataset['Driving_License'],ax = axx[0],palette = 'rocket')
ax2 = sns.countplot('Driving_License', hue = 'Response',ax =axx[1],data = dataset,palette="rocket_r")

for i,p in enumerate(ax1.patches):
    height = p.get_height()
    total_h+=[height]
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")
total_h*=2

for i,p in enumerate(ax2.patches):
    height = p.get_height()
    ax2.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total_h[i]*100), ha="center")

plt.tight_layout()<jupyter_output>Percentage of  Driving_License feature
 
1    99.786938
0     0.213062
Name: Driving_License, dtype: float64<jupyter_text>1. Customers who have the DL are 99% 2. Customers who are interested in Vehicle Insurance almost all have driving license   Previously Insured<jupyter_code>print("Percentage ofPreviously_Insured feature\n ")
print(dataset['Previously_Insured'].value_counts()/len(dataset)*100)
f,ax = plt.subplots(nrows=1,ncols=2,figsize=(12,5))
axx = ax.flatten()
ax1 = sns.countplot(dataset['Previously_Insured'],ax = axx[0],palette="rocket_r")
ax2 = sns.countplot('Previously_Insured', hue = 'Response',ax =axx[1],data = dataset,palette="rocket_r")
total_h=[]

for i,p in enumerate(ax1.patches):
    height = p.get_height()
    total_h+=[height]
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")
total_h*=2

for i,p in enumerate(ax2.patches):
    height = p.get_height()
    ax2.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total_h[i]*100), ha="center")
plt.tight_layout()<jupyter_output>Percentage ofPreviously_Insured feature
 
0    54.178988
1    45.821012
Name: Previously_Insured, dtype: float64<jupyter_text>1. The variable perviosly insured almost has equal count 2. Customer who are not perviosly insured are likely to be inetrested Customers who were previously insured tend not to be interested. We can think that the reason for this is that their previous insurance agreement has not expired yet.  Vehicle Age<jupyter_code>print("Percentage of vechicle age feature\n ")
print(dataset['Vehicle_Age'].value_counts()/len(dataset)*100)
plt.figure(figsize = (13,5))
plt.subplot(1,2,1)
ax1 = sns.countplot(dataset['Vehicle_Age'],palette="rocket_r")
plt.title("Count plot of vechicle age")
plt.subplot(1,2,2)
plt.title("Plot of vechicle age vs response")
ax2 = sns.countplot('Vehicle_Age', hue = 'Response',data = dataset ,palette="rocket_r")
total_h=[]

for i,p in enumerate(ax1.patches):
    height = p.get_height()
    print(f'{height}\n')
    total_h+=[height]
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")

total_h*=2
print(f'{total_h}\n')
for i,p in enumerate(ax2.patches):
    height = p.get_height()
    print(height)
    ax2.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total_h[i]*100), ha="center")
plt.tight_layout()

dataset.groupby(['Vehicle_Age']).apply( lambda x : x['Response'].value_counts(normalize= True)*100)<jupyter_output><empty_output><jupyter_text>* There is very less number of customers with vechicle age less than 2 years* Customers with vechicle age 1-2 years are more likely to interested as compared to the other two * Customers with with Vehicle_Age <1 years have very less chance of buying Insurance<jupyter_code>#sns.countplot('Response', hue = 'Vehicle_Age',data = dataset ,palette="cool")<jupyter_output><empty_output><jupyter_text>Vechicle damage     Customers having damaged vehicle<jupyter_code>print("Percentage of vechicle damage feature\n ")
print(dataset['Vehicle_Damage'].value_counts()/len(dataset)*100)
plt.figure(figsize = (13,5))
plt.subplot(1,2,1)
ax1 = sns.countplot(dataset['Vehicle_Damage'],palette="rocket_r")
plt.title("Count plot of Vehicle_Damage")
plt.subplot(1,2,2)
plt.title("Plot of vechicle damage vs response")
ax2 = sns.countplot('Vehicle_Damage', hue = 'Response',data = dataset,palette="cool")

total_h=[]
for i,p in enumerate(ax1.patches):
    height = p.get_height()
    total_h+=[height]
    ax1.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total*100), ha="center")
total_h*=2
for i,p in enumerate(ax2.patches):
    height = p.get_height()
    ax2.text(p.get_x()+p.get_width()/2., height + 3, '{:1.2f}'.format(height/total_h[i]*100), ha="center")
plt.tight_layout()<jupyter_output>Percentage of vechicle damage feature
 
Yes    50.487656
No     49.512344
Name: Vehicle_Damage, dtype: float64<jupyter_text>* Customers with vechicle damage(Yes and NO) are equally distributed with (50.48 % , 49.51 %) * Customers with vechicle damage are more interested in Vehicle Insurance  VintageNumber of Days, Customer has been associated with the company<jupyter_code>f,ax = plt.subplots(nrows=1,ncols=2,figsize=(25,5))
axx = ax.flatten()
sns.distplot(dataset['Vintage'],ax=axx[0], color='c')
sns.boxplot (dataset['Vintage'],ax=axx[1],color='c')<jupyter_output><empty_output><jupyter_text>Region Code<jupyter_code>plt.figure(figsize = (20,10))
plt.subplot(3,1,1)
sns.countplot(dataset['Region_Code'], palette = 'hsv')
plt.title('Count of Region code')<jupyter_output><empty_output><jupyter_text>* The indivisuals with region code 28 the highest as compared to the other ones* From the box plot it looks like there is no outliers in the data - Further  we can analyze which region has highest intrested customers     Policy Sales Channel<jupyter_code>plt.figure(figsize = (20,10))
plt.subplot(3,1,2)
sns.distplot(dataset['Policy_Sales_Channel'])
plt.title('Distribution of Region code')

sns.FacetGrid(dataset,hue='Response',size=5) \
 .map(sns.distplot,"Policy_Sales_Channel") \
 .add_legend();
plt.show()<jupyter_output><empty_output><jupyter_text>The most used sales channels are 152, 26 and 124. The best channel that results in customer interest is 152.<jupyter_code>plt.figure(figsize=(5,5))
plt.title("Correlation plot")
sns.heatmap(dataset[numerical_features].corr().abs(),linewidths=5, annot=True, square=True,annot_kws={'size': 10},cmap='YlGnBu', fmt=".3f") #cmap = 'Blues'<jupyter_output><empty_output><jupyter_text>We can see that the most influencing factors for Response are Vehicle_Damage and Previously_Insured, followed by Vehicle_Age   Data Preprocessing   annual premium has the worst outliers<jupyter_code>Q1 = np.percentile(dataset['Annual_Premium'], 25, interpolation = 'midpoint') 
Q2 = np.percentile(dataset['Annual_Premium'], 50, interpolation = 'midpoint') 
Q3 = np.percentile(dataset['Annual_Premium'], 75, interpolation = 'midpoint') 
  
IQR = Q3 - Q1 

low_lim = Q1 - 1.5 * IQR
up_lim = Q3 + 1.5 * IQR
outlier =[]
for x in dataset['Annual_Premium']:
    if ((x> up_lim) or (x<low_lim)):
         outlier.append(x)
print(' Outliers in the Annual Premium :', len(outlier))
import seaborn as sns
sns.boxplot(dataset['Annual_Premium'])<jupyter_output><empty_output><jupyter_text>* From the distribution plot we can infer that the annual premimum variable is right skewed* From the boxplot we can observe lot of outliers in the variable<jupyter_code>#dataset['Annual_Premium']=dataset['Annual_Premium'].replace(outlier,up_lim)
#import seaborn as sns
#ns.boxplot(dataset['Annual_Premium'])<jupyter_output><empty_output><jupyter_text>Checking duplicate rows<jupyter_code>duplicate=dataset[dataset.duplicated()].sort_values(['Vintage','Age'])
print(duplicate)<jupyter_output><empty_output><jupyter_text>* there is no duplicated rows in the dataset the above summary shows the number of observations that have the same respose and same features  Label Encoding In machine learning, we usually deal with datasets which contains multiple labels in one or more than one columns. These labels can be in the form of words or numbers. To make the data understandable or in human readable form, the training data is often labeled in words.Label Encoding refers to converting the labels into numeric form so as to convert it into the machine-readable form. We replaced some values in the data sets with numerical values, as follows;**Vehicle Age ->*** "<1 Year" = 0* "1-2 Year" = 1* ">2 Year" = 2**Gender ->*** "Female" = 0* "Male" = 1**Vehicle Damage ->*** "No" = 0* "Yes" = 1<jupyter_code>dataset.head(5)
dataset['Vehicle_Age']=dataset['Vehicle_Age'].replace({'< 1 Year':0.5,'1-2 Year':1.5,'> 2 Years':2.5})
dataset['Gender']=dataset['Gender'].replace({'Male':1,'Female':0})
dataset['Vehicle_Damage']=dataset['Vehicle_Damage'].replace({'Yes':1,'No':0})
dataset.head(5)<jupyter_output><empty_output><jupyter_text>Feature Engineering :Handling Categorical values in Policy_Sales_Channel  and Region_Code<jupyter_code>from sklearn.model_selection import train_test_split
dataset , dataset_test = train_test_split(dataset , test_size=0.2 )
dataset.shape , dataset_test.shape<jupyter_output><empty_output><jupyter_text>target mean encodingMake this a fucniton , such that it can also be applied to incoming data ,create a data prep pipelinesuch this transform can be carried out on new incoming data also<jupyter_code>Mean_encoding_Policy_Sales_Channel = dataset.groupby(['Policy_Sales_Channel'])['Response'].mean().to_dict()

dataset['Policy_Sales_Channel_Target_encoded'] =  dataset['Policy_Sales_Channel'].map(Mean_encoding_Policy_Sales_Channel)
#dataset_test['Policy_Sales_Channel_Target_encoded'] =  dataset_test['Policy_Sales_Channel'].map(Mean_encoding_Policy_Sales_Channel)
Mean_encoding_Region_Code = dataset.groupby(['Region_Code'])['Response'].mean().to_dict()

dataset['Region_Code_Target_encoded'] =  dataset['Region_Code'].map(Mean_encoding_Region_Code)
#dataset_test['Region_Code_Target_encoded'] =  dataset_test['Region_Code'].map(Mean_encoding_Region_Code)<jupyter_output><empty_output><jupyter_text>Cleaned Data Exporting<jupyter_code>dataset.head()
dataset.info()
filepath = '/content/drive/MyDrive/AlmaBetter/Capstone2_Supervised_Classification/Code/Data/processed_data.csv'
filepath = '/content/drive/MyDrive/AlmaBetter/Capstone2_Supervised_Classification/Code/Data/processed_data_test.csv'

dataset = dataset.to_csv(filepath)
dataset_test = dataset_test.to_csv(filepath)<jupyter_output><empty_output><jupyter_text>Inferences and ConclusionWe've drawn many interesting inferences from the health_insurance-cross-sell-prediction data , here's a summary of the few of them:we use pandas library as pd numpy as np etc. * Customers of age between 30 to 60 are more likely to buy insurance.* Customers with Driving License have higher chance of buying Insurance.* Customers with Vehicle_Damage are likely to buy insurance.*  The variable such as Age, Previously_insured,Annual_premium  are more afecting the target variable.* comparing ROC curve we can see that Random Forest model preform better. Because curves closer to the top-left corner, it indicate a better performance.  What Worked?* Hyperparameter tuning* Feature Engineering such as concatenation,aggregation, binning* Data cleaning steps and most pertinent of them were removing duplicates[more reliable CV] and confusing rows* Treating Annual Premium and Vintage as categorical features in catboost What didn't Work?* Class balancing via oversampler,undersampler,TomkeLinks etc and via hardcoding class weights parameter in individual algorithm  References and Future Work**TODO** - Write some explanation here: ideas for future projects using this dataset, and links to resources you found useful.  Hypothesis test1. Chi-test: It is help to figure-out relation between features and label with **"pvalue <= 0.05"**<jupyter_code>ct = pd.crosstab(dataset['Vintage'],dataset['Response'])
from scipy.stats import chi2_contingency
stat,pvalue,dof,expected_R = chi2_contingency(ct)
print("pvalue : ",pvalue)

if pvalue <= 0.05 :
    print("Alternate Hypothesis passed. Vintage and Response have Relationship")
else:
    print("Null hypothesis passed. Vintage and Response doesnot have  Relationship")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup