# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# # Feature Selection Methods
#
# This notebook is dedicated to performing feature selection using the best model found during autoML. The following approaches are considered:
#    - SelectKBest: will use ANOVA to keep only the most relevant features
#    - SequentialForwardSelection: will greedily build an estimator from an empty subset of features
#     
# The first method is a filtering method, meaning that it does not consider interactions between variables. while fast, this family of methods does not guarantee the generation of the most performing feature subset. Sequential Forward Floating Selection on the other hand is a wrapper method, that will consider interaction between the features, but it is very slow when the number of features is significant.

#For development
#Reload the library when a change is detected in one of the imported libraries
# %load_ext autoreload 
# %autoreload 2

import numpy as np
import pandas as pd

from src import feature_selection as fs

# # Setting up
#
# Load the gold dataframe (the one read to be ingested by the machine learning algorithms, where missing values are removed).

df = pd.read_csv('data/Train_gold.csv')
df.head()

target = 'Flow_label'
features = df.columns.drop(target)

# Split features from labels

X, y = df[features], df[target].values.ravel()

# # Univariate Feature Selection
# The next statistical test is actually to select the best K perfomring features. This is done using the ANOVA test for the variables in the dataframe. Note that using this type of filter does not guarantee generating a list of the best meaningfull variables, since the interactions are not taken into account. Nonetheless is a good test to perform. For the time being we select 80% of the features, of course this is a choice dicated by a random number I had in mind when creating this notebook.

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_classif, mutual_info_classif

feat_to_keep = round(0.8*len(X.columns))
univariate_model = SelectKBest(score_func=f_classif, k=feat_to_keep)
univariate_model.fit(X, y)

# ## Plot the ANOVA Feature Importance
# Once we have determined the best performing features, the next step is to actually plot their importance

X.columns = [r'$\theta$', r'$Re_{L}$', r'$Re_{G}$', 
             r'$Fr_{L}$', r'$Fr_{G}$', r'$N_{L}$', 
             r'$X_{LM}$', r'$X_{LM}^2$', r'$Y$', 
             r'$We$', r'$Eo$', r'$\alpha_{L}$', 
             r'$K_{G}$', r'$T_{TB}$']

model = f_classif(X, y)
ANOVA_data = {'Features': X.columns, 'F_score': model[0]}
imp_f_classif = pd.DataFrame(ANOVA_data).sort_values(by='F_score', ascending=False)
fs.plot_ANOVA(imp_f_classif)

# ## Step Forward Feature Selection
# Here we perfrom the step forward feature selection. This technique will use a greedy strategy to build the best performing model. Note that every feature kept will never be discarded in future steps unless the floating method is set to true.

# +
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
from imblearn.over_sampling import SMOTE

import lightgbm as lgbm
# -

estimator = lgbm.LGBMClassifier()
sfs=SFS(estimator, k_features=6, forward=True, 
        floating=True, scoring='balanced_accuracy', 
        verbose=0, cv=5, n_jobs=-1)

sfs.fit(X, y)

info_di = sfs.get_metric_dict()
l = [list(info_di[x]['feature_names']) for x in info_di.keys()]
scores = fs.generate_SFFSinfo(X, y, l, balance_method=SMOTE())

fs.plot_SFFS(scores)





StarcoderdataJupyterScriptsDedupFiltered