# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="4fsDADOlVKmV"
# #ML 101
#
# You all have seen datasets. Sometimes they are small, but often at times, they are tremendously large in size. It becomes very challenging to process the datasets which are very large, at least significant enough to cause a processing bottleneck.
#
# So, what makes these datasets this large? Well, it's features. The more the number of features the larger the datasets will be. Well, not always. You will find datasets where the number of features is very much, but they do not contain that many instances. But that is not the point of discussion here. So, you might wonder with a commodity computer in hand how to process these type of datasets without beating the bush.
#
# Often, in a high dimensional dataset, there remain some entirely irrelevant, insignificant and unimportant features. It has been seen that the contribution of these types of features is often less towards predictive modeling as compared to the critical features. They may have zero contribution as well. These features cause a number of problems which in turn prevents the process of efficient predictive modeling -
#
# >Unnecessary resource allocation for these features.These features act as a noise for which the machine learning model can perform terribly poorly. The machine model takes more time to get trained.
#
# So, what's the solution here? The most economical solution is **Feature Selection**.
#
# Feature Selection is the process of selecting out the most significant features from a given dataset. In many of the cases, Feature Selection can enhance the performance of a machine learning model as well.
#
# ## Introduction to feature selection
#
# Feature selection is also known as Variable selection or Attribute selection.
#
# Essentially, it is the process of selecting the most important/relevant. Features of a dataset.
#
# ## Understanding the importance of feature selection
#
# The importance of feature selection can best be recognized when you are dealing with a dataset that contains a vast number of features. This type of dataset is often referred to as a high dimensional dataset. Now, with this high dimensionality, comes a lot of problems such as - this high dimensionality will significantly increase the training time of your machine learning model, it can make your model very complicated which in turn may lead to Overfitting.
#
# Often in a high dimensional feature set, there remain several features which are redundant meaning these features are nothing but extensions of the other essential features. These redundant features do not effectively contribute to the model training as well. So, clearly, there is a need to extract the most important and the most relevant features for a dataset in order to get the most effective predictive modeling performance.
#
# >"The objective of variable selection is three-fold: improving the prediction performance of the predictors, providing faster and more cost-effective predictors, and providing a better understanding of the underlying process that generated the data."
#
# Now let's understand the difference between dimensionality reduction and feature selection.
#
# Sometimes, feature selection is mistaken with **dimensionality reduction**. But they are different. Feature selection is different from dimensionality reduction. Both methods tend to reduce the number of attributes in the dataset, but a dimensionality reduction method does so by creating new combinations of attributes (sometimes known as feature transformation), whereas feature selection methods include and exclude attributes present in the data without changing them.
#
# Some examples of dimensionality reduction methods are Principal Component Analysis, Singular Value Decomposition, Linear Discriminant Analysis, etc.
#
# ## Filter methods
#
# The following image best describes filter-based feature selection methods:
#
# ![fs00](https://media.githubusercontent.com/media/mariolpantunes/ml101/main/figs/fs00.webp)
#
# Filter method relies on the general uniqueness of the data to be evaluated and pick feature subset, not including any mining algorithm. Filter method uses the exact assessment criterion which includes distance, information, dependency, and consistency. The filter method uses the principal criteria of ranking technique and uses the rank ordering method for variable selection. The reason for using the ranking method is simplicity, produce excellent and relevant features. The ranking method will filter out irrelevant features before classification process starts.
#
# Filter methods are generally used as a data preprocessing step. The selection of features is independent of any machine learning algorithm. Features give rank on the basis of statistical scores which tend to determine the features' correlation with the outcome variable. Correlation is a heavily contextual term, and it varies from work to work. You can refer to the following table for defining correlation coefficients for different types of data (in this case continuous and categorical).
#
# ## Wrapper methods
#
# Like filter methods, let me give you a same kind of info-graphic which will help you to understand wrapper methods better:
#
# ![fs01](https://media.githubusercontent.com/media/mariolpantunes/ml101/main/figs/fs01.webp)
#
# As you can see in the above image, a wrapper method needs one machine learning algorithm and uses its performance as evaluation criteria. This method searches for a feature which is best-suited for the machine learning algorithm and aims to improve the mining performance. To evaluate the features, the predictive accuracy used for classification tasks and goodness of cluster is evaluated using clustering.
#
# Some typical examples of wrapper methods are forward feature selection, backward feature elimination, recursive feature elimination, etc.
#
# - **Forward Selection**: The procedure starts with an empty set of features [reduced set]. The best of the original features is determined and added to the reduced set. At each subsequent iteration, the best of the remaining original attributes is added to the set.
#
# - **Backward Elimination**: The procedure starts with the full set of attributes. At each step, it removes the worst attribute remaining in the set.
#
# - **Combination of forward selection and backward elimination**: The stepwise forward selection and backward elimination methods can be combined so that, at each step, the procedure selects the best attribute and removes the worst from among the remaining attributes.
#
# - **Recursive Feature elimination**: Recursive feature elimination performs a greedy search to find the best performing feature subset. It iteratively creates models and determines the best or the worst performing feature at each iteration. It constructs the subsequent models with the left features until all the features are explored. It then ranks the features based on the order of their elimination. In the worst case, if a dataset contains $N$ number of features RFE will do a greedy search for $2^N$ combinations of features.
#
# ## Embedded methods
#
# Embedded methods are iterative in a sense that takes care of each iteration of the model training process and carefully extract those features which contribute the most to the training for a particular iteration. Regularization methods are the most commonly used embedded methods which penalize a feature given a coefficient threshold.
#
# This is why Regularization methods are also called penalization methods that introduce additional constraints into the optimization of a predictive algorithm (such as a regression algorithm) that bias the model toward lower complexity (fewer coefficients).
#
# Examples of regularization algorithms are the LASSO, Elastic Net, Ridge Regression, etc.
#
# ## Difference between filter and wrapper methods
#
# Well, it might get confusing at times to differentiate between filter methods and wrapper methods in terms of their functionalities. Let's take a look at what points they differ from each other.
#
#  - Filter methods do not incorporate a machine learning model in order to determine if a feature is good or bad whereas wrapper methods use a machine learning model and train it the feature to decide if it is essential or not.
#  - Filter methods are much faster compared to wrapper methods as they do not involve training the models. On the other hand, wrapper methods are computationally costly, and in the case of massive datasets, wrapper methods are not the most effective feature selection method to consider.
#  - Filter methods may fail to find the best subset of features in situations when there is not enough data to model the statistical correlation of the features, but wrapper methods can always provide the best subset of features because of their exhaustive nature.
#  - Using features from wrapper methods in your final machine learning model can lead to overfitting as wrapper methods already train machine learning models with the features and it affects the true power of learning. But the features from filter methods will not lead to overfitting in most of the cases
#
# So far you have studied the importance of feature selection, understood its difference with dimensionality reduction. You also covered various types of feature selection methods. So far, so good!
#

# + id="6VuPeDYWVGwX"
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# + id="JYyMhTz-eYXC"
url = "https://media.githubusercontent.com/media/mariolpantunes/ml101/main/datasets/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pd.read_csv(url, names=names)
dataframe.head()

# + id="ZT3lWCMRelar"
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]

print(X)

# + [markdown] id="on8VUd8oeqpm"
# First, you will implement a Chi-Squared statistical test for non-negative features to select 4 of the best features from the dataset. You have already seen Chi-Squared test belongs the class of filter methods. If anyone's curious about knowing the internals of Chi-Squared, this [video](https://www.youtube.com/watch?v=VskmMgXmkMQ) does an excellent job.
#
# The scikit-learn library provides the **SelectKBest** class that can be used with a suite of different statistical tests to select a specific number of features, in this case, it is Chi-Squared.

# + id="7hwPCgKdeom4"
# Import the necessary libraries first
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
from sklearn.feature_selection import mutual_info_classif

# Feature extraction
test = SelectKBest(score_func=mutual_info_classif, k=4)
fit = test.fit(X, Y)

# Summarize scores
np.set_printoptions(precision=8)
print(fit.scores_)

features = fit.transform(X)
# Summarize selected features
print(features[0:5,:])

# + [markdown] id="1_9GgqLWe8W0"
# ## Interpretation:
#
# You can see the scores for each attribute and the 4 attributes chosen (those with the highest scores): plas, test, mass, and age. This scores will help you further in determining the best features for training your model.
#
# **P.S.**: The first row denotes the names of the features. For preprocessing of the dataset, the names have been numerically encoded.

# + [markdown] id="uY3ZiEmBmJfQ"
# The second filter method will be the Pearson correlation.
# Here we will first plot the Pearson correlation heatmap and see the correlation of independent variables with the output variable MEDV. We will only select features which has correlation of above 0.5 (taking absolute value) with the output variable.

# + id="YJk2EMyVmE3y"
#Using Pearson Correlation
plt.figure(figsize=(12,10))
cor = dataframe.corr()
sns.heatmap(cor, annot=True, cmap=plt.cm.Reds)
plt.show()

# + id="0oKVmcIwmgdD"
#Correlation with output variable
cor_target = abs(cor["class"])#Selecting highly correlated features
relevant_features = cor_target[cor_target>0.20]
relevant_features

# + [markdown] id="hueq6gyQfSq2"
# Next, you will implement Recursive Feature Elimination which is a type of wrapper feature selection method.
#
# The Recursive Feature Elimination (or RFE) works by recursively removing attributes and building a model on those attributes that remain.
#
# It uses the model accuracy to identify which attributes (and combination of attributes) contribute the most to predicting the target attribute.
#
# You can learn more about the **RFE** class in the scikit-learn [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE).
#
# You will use RFE with the Logistic Regression classifier to select the top 3 features. The choice of algorithm does not matter too much as long as it is skillful and consistent.

# + id="psALnu9cfizG"
# Import your necessary dependencies
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

# Feature extraction
#model = LogisticRegression(max_iter=1000)
model = RandomForestClassifier(random_state=0)
rfe = RFE(model, n_features_to_select=3)
fit = rfe.fit(X, Y)
print("Num Features: %s" % (fit.n_features_))
print("Selected Features: %s" % (fit.support_))
print("Feature Ranking: %s" % (fit.ranking_))

# + [markdown] id="mKveCXOqf7js"
# You can see that RFE chose the top 3 features as preg, mass, and pedi.
#
# These are marked True in the support array and marked with a choice “1” in the ranking array. This, in turn, indicates the strength of these features.

# + [markdown] id="OZVCt1yUgETC"
# Next up you will use Ridge regression which is basically a regularization technique and an embedded feature selection techniques as well.
#
# This [article](https://www.analyticsvidhya.com/blog/2016/01/ridge-lasso-regression-python-complete-tutorial/#three) gives you an excellent explanation on Ridge regression. Be sure to check it out.

# + id="JXL5pBr0gDrY"
# First things first
from sklearn.linear_model import Ridge

ridge = Ridge(alpha=1.0)
ridge.fit(X, Y)


# + [markdown] id="cA9G0_5agX3s"
# In order to better understand the results of Ridge regression, you will implement a little helper function that will help you to print the results in a better so that you can interpret them easily.

# + id="SdP0yWA5gYq9"
# A helper method for pretty-printing the coefficients
def pretty_print_coefs(coefs, names = None, sort = False):
    if names == None:
        names = ["X%s" % x for x in range(len(coefs))]
    lst = zip(coefs, names)
    if sort:
        lst = sorted(lst,  key = lambda x:-np.abs(x[0]))
    return " + ".join("%s * %s" % (round(coef, 3), name)
                                   for coef, name in lst)
  
print ("Ridge model:", pretty_print_coefs(ridge.coef_))

# + [markdown] id="C0AloSh_geli"
# You can spot all the coefficient terms appended with the feature variables. It will again help you to choose the most essential features. Below are some points that you should keep in mind while applying Ridge regression:
#  - It is also known as L2-Regularization.
#  - For correlated features, it means that they tend to get similar coefficients.
#  - Feature having negative coefficients don't contribute that much. But in a more complex scenario where you are dealing with lots of features, then this score will definitely help you in the ultimate feature selection decision-making process.
#

StarcoderdataJupyterScriptsDedupFiltered