<jupyter_start><jupyter_text>"EDA and Prediction of Mushroom Edibility using Select ML Algorithms"- toc: true- badges: true- comments: true  Introduction Today, we'll work on a classification problem. The dataset we have chosen is the [mushroom-classification](https://www.kaggle.com/datasets/uciml/mushroom-classification) dataset available on Kaggle. This dataset was provided by UCI Machine Learning repository nearly 3 decades ago. The dataset includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family Mushroom drawn from The Audubon Society Field Guide to North American Mushrooms (1981). Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended.Our task making successful predictions begins first by the setup of the system for training.  Setup<jupyter_code># Import the required libraries and get the file path

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # data visualization
import seaborn as sns # data visualization
from sklearn.model_selection import train_test_split, cross_validate, GridSearchCV # validation
from sklearn.preprocessing import OneHotEncoder, LabelEncoder # data preparation
from sklearn.metrics import ConfusionMatrixDisplay, RocCurveDisplay, precision_score, recall_score, accuracy_score, f1_score # metrics
from sklearn.pipeline import make_pipeline # build pipeline

# ML models
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression, RidgeClassifier
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from xgboost import XGBClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# ignore warnings
import warnings
warnings.filterwarnings("ignore")

# get file path
import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
file_dir = "/kaggle/input/mushroom-classification/mushrooms.csv"

# inspect file size
!ls -lh {file_dir}

# inspect first 5 rows of dataset
!head {file_dir}<jupyter_output>class,cap-shape,cap-surface,cap-color,bruises,odor,gill-attachment,gill-spacing,gill-size,gill-color,stalk-shape,stalk-root,stalk-surface-above-ring,stalk-surface-below-ring,stalk-color-above-ring,stalk-color-below-ring,veil-type,veil-color,ring-number,ring-type,spore-print-color,population,habitat
p,x,s,n,t,p,f,c,n,k,e,e,s,s,w,w,p,w,o,p,k,s,u
e,x,s,y,t,a,f,c,b,k,e,c,s,s,w,w,p,w,o,p,n,n,g
e,b,s,w,t,l,f,c,b,n,e,c,s,s,w,w,p,w,o,p,n,n,m
p,x,y,w,t,p,f,c,n,n,e,e,s,s,w,w,p,w,o,p,k,s,u
e,x,s,g,f,n,f,w,b,k,t,e,s,s,w,w,p,w,o,e,n,a,g
e,x,y,y,t,a,f,c,b,n,e,c,s,s,w,w,p,w,o,p,k,n,g
e,b,s,w,t,a,f,c,b,g,e,c,s,s,w,w,p,w,o,p,k,n,m
e,b,y,w,t,l,f,c,b,n,e,c,s,s,w,w,p,w,o,p,n,s,m
p,x,y,w,t,p,f,c,n,p,e,e,s,s,w,w,p,w,o,p,k,v,g<jupyter_text>Observations* The dataset file size is 366 KB.* It will be safe to import the whole dataset.* The prediction class is the first column.* There appears to be no index column<jupyter_code># read file
df = pd.read_csv(file_dir)

# view all columns
pd.set_option("display.max_columns", None)

df.head()<jupyter_output><empty_output><jupyter_text>EDA and Data Preparation<jupyter_code># split datasets for training and testing
X = df.copy()
y = X.pop("class")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)<jupyter_output><empty_output><jupyter_text>Preliminary Analysis<jupyter_code>X_train.shape
X_train.info()
X_train.describe()<jupyter_output><empty_output><jupyter_text>Observations- The dataset has 22 features.- There are 6499 enries. - All features are categorical in nature.- Most of the features have unique values less than 10.  Target Class<jupyter_code># target classes and their distribution

class_vc = df["class"].value_counts()
class_vc
sns.barplot(x = class_vc.index, y = class_vc)<jupyter_output><empty_output><jupyter_text>Observations- The class counts are not much imbalanced.  Distribution of Features<jupyter_code>for i, cols in enumerate(df):
    feature_vc = df[cols].value_counts()
    print(feature_vc, "\n_________\n")
    
    plt.figure(i)
    sns.barplot(x = feature_vc.index, y = feature_vc)<jupyter_output>e    4208
p    3916
Name: class, dtype: int64 
_________

x    3656
f    3152
k     828
b     452
s      32
c       4
Name: cap-shape, dtype: int64 
_________

y    3244
s    2556
f    2320
g       4
Name: cap-surface, dtype: int64 
_________

n    2284
g    1840
e    1500
y    1072
w    1040
b     168
p     144
c      44
u      16
r      16
Name: cap-color, dtype: int64 
_________

f    4748
t    3376
Name: bruises, dtype: int64 
_________

n    3528
f    2160
y     576
s     576
a     400
l     400
p     256
c     192
m      36
Name: odor, dtype: int64 
_________

f    7914
a     210
Name: gill-attachment, dtype: int64 
_________

c    6812
w    1312
Name: gill-spacing, dtype: int64 
_________

b    5612
n    2512
Name: gill-size, dtype: int64 
_________

b    1728
p    1492
w    1202
n    1048
g     752
h     732
u     492
k     408
e      96
y      86
o      64
r      24
Name: gill-color, dtype: int64 
_________

t    4608
e    3516
Name: stalk-shape, dtype: int64 
_________

b    [...]<jupyter_text>Observations- All the feautures are categorical in nature.- All classes are anonymized.- We'll need to OneHotEncode the data.- Because the number of no feature classes is too large, we can OneHotEncode each feature class. - Feature `vei-type` has only one class. Thus it'll need to be dropped.  Feature Distribution against Target Class<jupyter_code>for i, feature in enumerate(X_train):
    plt.figure(i)
    sns.countplot(x = feature, hue = y_train, data = X_train)<jupyter_output><empty_output><jupyter_text>Observations- Allmost all the features differ and distinguish between the two target classes.- Their distributions are different for the target classes.It means that models may also offer incredible scores in classifying the two target classes.  Null ValuesNow, we'll look at the number of null values in the data.<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Observations- There are no null values in the dataset. - It will need no imputation or transformation  Data PreparationThere isn't much to prepare in the data. We just need to OneHotEncode all the categorical features. And we'll also encode the target class.<jupyter_code># OneHotEncoding Categorical features
ohe = OneHotEncoder(drop = "first", handle_unknown = "ignore", sparse = False)
ohe_train_data = ohe.fit_transform(X_train)
ohe_test_data = ohe.transform(X_test)
ohe_train_data

# Encode the target in train dataset
le = LabelEncoder()
y_train_le = le.fit_transform(y_train)

# Encode target in test dataset
y_test_le = le.transform(y_test)<jupyter_output><empty_output><jupyter_text>So, we can now move on to training ML models.  Train Hardcoded Model and Evaluate ResultsFirst we'll train a hardcoded model, which will just predict the most frequent target class which is 'edible'. This will help us in giving us a baseline score which our future models should at least beat. It helps to find errors in training.<jupyter_code>mf_model = DummyClassifier(random_state = 42, strategy = "most_frequent")
mf_cross_val = cross_validate(mf_model, ohe_train_data, y_train_le, scoring = ["accuracy", "precision", "recall", "f1", "roc_auc"])
mf_cross_val<jupyter_output><empty_output><jupyter_text>Observations- The test roc_auc score is 0.5 in all validations, which means the model is not able to distinguish between the two target classes in any way. - This model also gives a test accuracy of 0.52.We'll need to at least improve on these scores.  Model SelectionIn this section, we'll train multiple ML models and compare their performance after cross validation. Then we'll choose the best performing one for final training.<jupyter_code>models = {"LogisticRegression": LogisticRegression(random_state = 42),
         "RidgeClassification": RidgeClassifier(random_state = 42),
         "GaussianNB": GaussianNB(),
         "RandomForestClassifier": RandomForestClassifier(n_estimators = 70, random_state = 42),
         "XGBClassifier": XGBClassifier(n_estimators = 70, objective = "binary:logistic", learning_rate = 0.05, n_jobs = -1, scoring = "auc", random_state = 42)}
model_scores = {}

# cross validate all models
for model_name, model in models.items():
    
    cross_val = cross_validate(model, ohe_train_data, y_train_le, n_jobs = -1, scoring = ["accuracy", "precision", "recall", "f1", "roc_auc"])
    del cross_val["fit_time"]
    del cross_val["score_time"]
    
    model_scores[model_name] = cross_val

# put results into a dataframe
model_scores_df = pd.DataFrame.from_dict(model_scores)
model_scores_df = model_scores_df.applymap(np.mean)
model_scores_df<jupyter_output><empty_output><jupyter_text>Observations- Surprisingly, all of the models performed incredibly well and fit almost perfectly to the training dataset. - Out of these, RandomForestClassifier performed the best with a perfect score of 1.0 in all scores.Thus, we'll train the RandomForestClassifier to make final predictions.  Train Final Model and make predictions<jupyter_code># train model
forest_clas = RandomForestClassifier(random_state = 42)

forest_clas.fit(ohe_train_data, y_train_le)

# make predictions
preds = forest_clas.predict(ohe_test_data)

# decode predictions into their original labels
preds_in = le.inverse_transform(preds) 
preds_in

# plot results
ConfusionMatrixDisplay.from_predictions(y_test, preds_in)
accuracy_score(y_test, preds_in)
recall_score(y_test_le, preds)
f1_score(y_test_le, preds)
RocCurveDisplay.from_estimator(forest_clas, ohe_test_data, y_test_le)<jupyter_output><empty_output>