<jupyter_start><jupyter_text>1. Defining the Question  a) Specifying the Data Analytic Question> To figure out how we can predict which individuals are most likely to have or use a bank account  b) Defining the Metric for Success> Our analysis will be deemed successful if we are able to get the specific indicators that we can use to accurately predict individuals most likely to hav a bank account.  c) Understanding the context> Traditionally, access to bank accounts has been regarded as an indicator of financial inclusion. Despite the proliferation of mobile money in Africa and the growth of innovative fintech solutions, banks still play a pivotal role in facilitating access to financial services. Access to bank accounts enables households to save and facilitate payments while also helping businesses build up their credit-worthiness and improve their access to other financial services. Therefore, access to bank accounts is an essential contributor to long-term economic growth.   d) Recording the Experimental Design> Our experimental design will include the following:1. Defining the question2. Reading the Data3. Checking the Data4. Tidying the dataset5. Exploratory Analysis6. Implementing the Solution7. Challenging the solution8. Follow-up questions  e) Data Relevance> The data provided is from Kenya, Tanzania, Rwanda & Uganda from 2016-2018 and contains data on features such as age, gender, education level among other, which are very relevant in answering our question. However seeing as it was colleted over 2 years ago and with the dynamism in the banking industry, the insights provided may not present the current situation.  2. Reading the Data<jupyter_code># Loading the required libraries first
import pandas as pd
import numpy as np
import seaborn as sns
import scipy.stats as stats
import matplotlib 
from matplotlib import pyplot as plt

# Loading our datasets from the urls provided
# Dataset: http://bit.ly/FinancialDataset 
#
df = pd.read_csv('http://bit.ly/FinancialDataset')
df<jupyter_output><empty_output><jupyter_text>3. Checking the Data<jupyter_code># Determining the no. of records in our dataset
#
df.shape
# Detecting our column names
#
df.columns<jupyter_output><empty_output><jupyter_text>> Our dataset has 23,524 rows and 13 columns.> The column names are 'country', 'year', 'uniqueid', 'Has a Bank account', 'Type of Location', 'Cell Phone Access', 'household_size', 'Respondent Age', 'gender_of_respondent',  'The relathip with head', 'marital_status', 'Level of Educuation', 'Type of Job'.<jupyter_code># Previewing the top of our dataset
#
df.head(10)
# Previewing the bottom of our dataset
# 
df.tail(10)
# Checking whether each column has an appropriate datatype
#
df.dtypes<jupyter_output><empty_output><jupyter_text>> We can see that most of our data consists of objects with exceptions on the year, household size & respondent age, which is appropriate for our dataframe.<jupyter_code># Checking the dataframe's information
#
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 23524 entries, 0 to 23523
Data columns (total 13 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   country                 23510 non-null  object 
 1   year                    23524 non-null  int64  
 2   uniqueid                23524 non-null  object 
 3   Has a Bank account      23488 non-null  object 
 4   Type of Location        23509 non-null  object 
 5   Cell Phone Access       23513 non-null  object 
 6   household_size          23496 non-null  float64
 7   Respondent Age          23490 non-null  float64
 8   gender_of_respondent    23490 non-null  object 
 9   The relathip with head  23520 non-null  object 
 10  marital_status          23492 non-null  object 
 11  Level of Educuation     23495 non-null  object 
 12  Type of Job             23494 non-null  object 
dtypes: float64(2), int64(1), object(10)
memory usage: 2.3+ MB<jupyter_text>4. Tidying the Dataset<jupyter_code># Checking for Outliers in our numerical columns
#
col_names = ['year', 'household_size', 'Respondent Age']

fig, ax = plt.subplots(len(col_names), figsize=(10,15))

for i, col_val in enumerate(col_names):

    sns.boxplot(x = df[col_val], ax=ax[i])
    ax[i].set_title('A boxplot on the {} column'.format(col_val), fontsize=14)
    ax[i].set_xlabel(col_val, fontsize=12)<jupyter_output><empty_output><jupyter_text>> From the 'year' column, we can see some outliers which are erroneous as the period was between 2016-2018. We shall remove those.>> On the 'household_size' column we also have some outliers, which may be accurate as ther isn't a limit on african household sizes. However, these will affect our summary statistics, thus we shall remove them.>> The  respondent age also has some outliers, which may be due to the fact that some households probably contain the grandparents and may be accurate. However, again these will affect our summary statistics thus we shall remove them<jupyter_code># Removing outliers using IQR
#
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
df = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]
df

# Check the shape of the new dataset
#
df.shape<jupyter_output><empty_output><jupyter_text>> Our new dataset now has 22,903 rows and 13 columns<jupyter_code># Identifying Missing Data
#
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>> We have some missing data in our dataset. We shall remove them.<jupyter_code># Dealing with the Missing data and checking if there is any more missing data
#
df.fillna( method ='ffill', inplace = True)
df.isnull().sum()

# Dropping irrelevant columns
#
drop_df = df[['uniqueid','The relathip with head']]
df = df.drop(drop_df, axis= 1)
# Check for missing values
#
df.isnull().sum()
# Checking final shape of our dataset
#
df.shape<jupyter_output><empty_output><jupyter_text>5. Exploratory Analysis a) Univariate Analysis:We will use this to describe our data and find the trends within the different variables.<jupyter_code># Mode
# Calculating mode across different columns

df.mode(axis=0)<jupyter_output><empty_output><jupyter_text>From the values above, we can see where most of our respondents fall in.<jupyter_code># Mean
# Calculating the mean for our numerical columns

df.mean(axis=0)<jupyter_output><empty_output><jupyter_text>Since years can only be whole numbers, we'll take our average as 2016. The average household size is 3.571890≈4 since people can only be integers. The average age across our dataset is 38years old.<jupyter_code># Median
# Calculating the median for our numerical columns

df.median(axis=0)<jupyter_output><empty_output><jupyter_text>The median year is 2017. Median household size is 3 while median age is 35.<jupyter_code># Skewness
# We will now calculate the skewness to check the assymetry of the distribution

df.skew()<jupyter_output><empty_output><jupyter_text>> All our numeric data is positively skewed showing that the tail is bigger on the right side. This also means that the mean is greater than the mode.<jupyter_code># Kurtosis
# We will now calculate the kurtosis to check the tail of our distribution

df.kurt()<jupyter_output><empty_output><jupyter_text>> All our numeric data have negative kurtosis values, signifying that they are platykurtic, thus light tailed. This might e because we have removed the outliers.<jupyter_code># Range, Standard Deviation & Variance
# We will now calculate the range, standard deviation and variance to check the spread of our data from the mean
range1 = df['Respondent Age'].max() - df['Respondent Age'].min()
range2 = df['household_size'].max() - df['household_size'].min()
std = df.std()
var = df.var()
print('Range: Respondent Age ', range1 ,'Household size ', range2)
print('Std deviation:', std)
print('Variance:', var)<jupyter_output>Range: Respondent Age  67.0 Household size  9.0
Std deviation: year               0.844533
household_size     2.079246
Respondent Age    15.801188
dtype: float64
Variance: year                0.713235
household_size      4.323264
Respondent Age    249.677555
dtype: float64<jupyter_text>> The years have a low standard deviation signifying that they are close, which is accurate as we only have 2016-2018.>> The Respondent age has a high standard deviation and variation, which shows that the data is spread over a wide range, evidenced by the range of 67.>> The household size has a medium range and standard deviation.<jupyter_code># Graphical representation of the repondents by Country
sns.countplot(df.country, color='blue',saturation=0.7)
plt.xlabel('Country')
plt.ylabel('Number of Respondents')<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>> Rwanda had the highest number of respondents compared to the rest<jupyter_code># Graphical representation of marital status
sns.countplot(df.marital_status, color='pink',saturation=0.7)
plt.xlabel('Marital_status')
plt.ylabel('Number of Respondents')
plt.title('Marital Status Distribution of Respondents', fontsize=10)<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>> Most of the respondents are Married/Living together while the least group are those who don't know.<jupyter_code># Graphical representation of household size
sns.countplot(df.household_size, color='magenta',saturation=0.7)
plt.xlabel('Household Size')
plt.ylabel('Number of Respondents')
plt.title('Household size Distribution of Respondents', fontsize=10)<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>> Our distribution is skewed to the right with most households consisting of between 1-4 people<jupyter_code># Creating a histogram to visualize the age distribution of individuals in the sample population

ages = df['Respondent Age']
plt.hist(ages, bins=10, histtype='bar', rwidth=0.9)
plt.xlabel('Number of Respondents', fontsize=10)
plt.ylabel('Age', fontsize=14, labelpad=15)
plt.title('Age Distribution of Respondents', fontsize=10)

plt.show()<jupyter_output><empty_output><jupyter_text>> The histogram is skewed to the right, indicating thus most respondents were young, thus its mean is greater than the mode.<jupyter_code># Graphical representation of number of people with bank accounts
label = df['Has a Bank account'].unique()
explode = (0.1, 0.1)
plt.pie(df['Has a Bank account'].value_counts().sort_values(),labels = label, explode = explode, autopct = '%1.1f%%')
plt.title("Respondents with bank accounts")
plt.axis('equal')<jupyter_output><empty_output><jupyter_text>> 85.8% of our respondents do not have a bank account<jupyter_code># Graphical representation of number of people with Cell phone access
label = df['Cell Phone Access'].unique()
explode = (0.1, 0.1)
plt.pie(df['Cell Phone Access'].value_counts(),labels = label, explode = explode, autopct = '%1.1f%%')
plt.title("Respondents with Cell Phone Access")
plt.axis('equal')<jupyter_output><empty_output><jupyter_text>> Most of the respondents have cell phone access<jupyter_code># Graphical representation of the type of location
label = df['Type of Location'].unique()
explode = (0.1, 0.1)
plt.pie(df['Type of Location'].value_counts(),labels = label, explode = explode, autopct = '%1.1f%%')
plt.title("Rural/Urban Distribution")
plt.axis('equal')<jupyter_output><empty_output><jupyter_text>Summary1. Most of our respondents are from Rwanda.2. Majority are married/living together.3. Most the households in the study have 1-4 people.4. Our respondents mostly consist of people aged 40 and below.5. 85.8% of our respondents do not have bank accounts.6. 74.7% of our respondents have cell phone access.7. 60.6% of our respondents are from rural b) Bivariate Analysis:We will use this now to check the relationships and measure the strengths of these relationships between 2 variables.<jupyter_code># Check for correlation in our numerical data
#
correlation = df.corr()
correlation
# Plotting a correlation heatmap
sns.heatmap(correlation, xticklabels=correlation.columns, yticklabels=correlation.columns, annot=True)<jupyter_output><empty_output><jupyter_text>> From this,we can see that household size and the respondent age are weakly negatively correlated at -0.105352. The 2 variables are also weakly negatively correlated to the year.<jupyter_code># Creating a scatterplot to check the linear regression betweeen age and household size
sns.regplot(x = df["Respondent Age"], y = df["household_size"], fit_reg=False)

plt.title("Scatter plot of Household Size vs Respondent's Age", fontsize=15, y=1.015)
plt.xlabel('Age', fontsize=14, labelpad=15)
plt.ylabel('Household Size', fontsize=14, labelpad=15)
plt.show()<jupyter_output><empty_output><jupyter_text>> Thus we cannot use the respondent's age to predict the household size<jupyter_code># Checking relationship between respondents with bank accounts and gender
gender = df.groupby('gender_of_respondent')['Has a Bank account'].value_counts(normalize=True).unstack()
colors= ['orange', 'green']
gender.plot(kind='bar', figsize=(8, 6), color=colors, stacked=True)
plt.title('Respondents with Bank Accounts by Gender', fontsize=15, y=1.015)
plt.xlabel('Gender', fontsize=14, labelpad=15)
plt.xticks(rotation = 360)
plt.ylabel('Respondents with bank accounts', fontsize=14, labelpad=15)
plt.show()<jupyter_output><empty_output><jupyter_text>> From our plot, we can see that males have a higher likelihood of having a bank account as compared to females.<jupyter_code># Checking respondents with which type of job are more likely to have a bank account
ax1 = df.groupby('Has a Bank account')['Type of Job'].value_counts(normalize=True).unstack()
ax1.plot(kind='bar', stacked='True',title=str(ax1))
int_level = df['Has a Bank account'].value_counts()<jupyter_output><empty_output><jupyter_text>> From our plot we are able to see that self employed individuals are more likely to have a bank account as compared to the other occupations. Those in informal employment are less likely.<jupyter_code># Checking whether respondents from urban or rural are more likely to have a bank account
location=df.groupby('Type of Location')['Has a Bank account'].value_counts(normalize=True).unstack()
colors= ['cyan', 'green']
location.plot(kind='bar', figsize=(8, 6), color=colors, stacked=True)
plt.title('Respondents with bank accounts by Type of Location', fontsize=15, y=1.015)
plt.xlabel('Type of Location', fontsize=14, labelpad=15)
plt.xticks(rotation = 360)
plt.ylabel('Respondents with bank accounts', fontsize=14, labelpad=15)
plt.show()<jupyter_output><empty_output><jupyter_text>> Respondents from the urban areas are more likely to have bank accounts as compared to their counterparts in the rural areas.<jupyter_code># Descriptive Statistics
!pip install researchpy
import researchpy as rp


rp.summary_cont(df.groupby("Has a Bank account")['Respondent Age'])<jupyter_output>Requirement already satisfied: researchpy in /usr/local/lib/python3.7/dist-packages (0.3.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from researchpy) (1.4.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from researchpy) (1.19.5)
Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from researchpy) (0.10.2)
Requirement already satisfied: patsy in /usr/local/lib/python3.7/dist-packages (from researchpy) (0.5.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from researchpy) (1.1.5)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from patsy->researchpy) (1.15.0)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->researchpy) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->researchpy) (2018.9)<jupyter_text>Summary1.  Household size and the respondent age are weakly negatively correlated at -0.105352.  2. We cannot use the respondent's age to predict the household size.3. Males have a higher likelihood of having a bank account as compared to females.4. Self employed individuals are more likely to have a bank account as compared to other occupations. Those in informal employment are less likely.5. Respondents from the urban areas are more likely to have bank accounts as compared to their counterparts in the rural areas. c) Multivariate Analysis: 1. Principal Component Analysis<jupyter_code># We shall label the dataset that has 2 outcomes

encode={"Has a Bank account" : {"Yes" : 1, "No": 0},"Type of Location" : {"Rural" : 1, "Urban" : 0},"Cell Phone Access" : {"Yes" : 1, "No" : 0}, "gender_of_respondent" : {"Female" : 1, "Male" : 1}}
df.replace(encode, inplace=True)
df.head()
# Drop unnecessary columns

df.drop(['country','year','marital_status','Level of Educuation','Type of Job'],axis=1,inplace=True)
# Define our variables to work with in our dimension reduction

X=df.drop('Has a Bank account',1)
y=df['Has a Bank account']
# Split dataset into test and train and define size of the test data(20%)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2,random_state=0)
# Import the Standard Scaler from Sklearn to normalise the distribution of the data 

from sklearn.preprocessing import StandardScaler

sc=StandardScaler()
X_train=sc.fit_transform(X_train)
X_test=sc.transform(X_test)
# Import the PCA for our analysis which will only be used in the train data and all the factors will e considered.

from sklearn.decomposition import PCA

pca=PCA()

X_train=pca.fit_transform(X_train)
X_test=pca.transform(X_test)
# Find out the variance for each principal component of our analysis. In our analysis we find out that the first four components account for 100% of the classification of whether one can pick a bank account or not. The first three components account for 82.9%

explained_variance=pca.explained_variance_ratio_
explained_variance
# We shall use the first component for our analysis in our prediction model

pca=PCA(n_components=1)

X_train=pca.fit_transform(X_train)
X_test=pca.transform(X_test)
# Train our model using the Random Forest Classifier to help out in the classification


from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(max_depth=0.2, random_state=0)
classifier.fit(X_train, y_train)
# Test using the data that was set apart for the test and train to find out the efficiency of th  e model.

y_pred=classifier.predict(X_test)
# Find out the first Principal Component has an 85% predictive capacity.

from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm=confusion_matrix(y_test,y_pred)
print(cm)
print('Accuracy', accuracy_score(y_test,y_pred))
# Test the model for factor analysis.

!pip install factor-analyzer
from factor_analyzer import calculate_bartlett_sphericity

chi_square_value, p_value=calculate_bartlett_sphericity(df)
chi_square_value, p_value
# The dataset has no capacity for factor analysis with nan KMO value

from factor_analyzer.factor_analyzer import calculate_kmo
kmo_all, kmo_model=calculate_kmo(df)
kmo_all, kmo_model<jupyter_output>/usr/local/lib/python3.7/dist-packages/factor_analyzer/utils.py:248: UserWarning: The inverse of the variance-covariance matrix was calculated using the Moore-Penrose generalized matrix inversion, due to its determinant being at or very close to zero.
  warnings.warn('The inverse of the variance-covariance matrix '
/usr/local/lib/python3.7/dist-packages/factor_analyzer/utils.py:205: RuntimeWarning: divide by zero encountered in true_divide
  Is = np.sqrt(1 / np.diag(m))
/usr/local/lib/python3.7/dist-packages/factor_analyzer/utils.py:206: RuntimeWarning: invalid value encountered in multiply
  retval = Is * m * np.repeat(Is, numrows).reshape(numrows, numrows)<jupyter_text>2. Linear Discriminant Analysis<jupyter_code># Divide the dataset and split the data into train and test

from sklearn.model_selection import train_test_split

x=df.iloc[:,1:6].values
Y=df.iloc[:,0].values
x_train, x_test, Y_train, Y_test=train_test_split(x,Y,test_size=0.2,random_state=0)
# Create a normal distribution of the data

from sklearn.preprocessing import StandardScaler
sc=StandardScaler()
x_train=sc.fit_transform(x_train)
x_test=sc.transform(x_test)
# Using the Random Forest Classifier in our classification model

from sklearn.ensemble import RandomForestClassifier
classifier=RandomForestClassifier(max_depth=0.2, random_state=0)
classifier.fit(x_train, Y_train)
Y_pred=classifier.predict(x_test)
# Finding out the predictive capacity of our Linear Discriminant model.

from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm=confusion_matrix(Y_test,Y_pred)
print(cm)
print('Our accuracy is ' + str(accuracy_score(Y_test,Y_pred)))<jupyter_output>[[3934    0]
 [ 647    0]]
Our accuracy is 0.8587644619078804<jupyter_text>7. Challenging the solution > The easy solution is nice because it is, well, easy, but you should never allow those results to hold the day. You should always be thinking of ways to challenge the results, especially if those results comport with your prior expectation.<jupyter_code># Reviewing the Solution 
#<jupyter_output><empty_output>