<jupyter_start><jupyter_text>Introduction to scikit-learn (sklearn)This notebook demostrates some of the most useful functions of the beautiful scikit-learn libraryWhat we're going to cover:0. An end-to-end scikit-learn workflow1. Getting the data ready2. Choose the right estimator/algorithm for our problems3. Fit the model/algorithm and use it to make predictions on our data4. Evaluating a model5. Improve a model6. Save and load a trained model7. Putting it all together!  0. An end-to-end scikit-learn workflow<jupyter_code>import numpy as np
# 1. Get the data ready
import pandas as pd
heart_disease=pd.read_csv("heart-disease.csv")
heart_disease
# create X (features matrix)
X=heart_disease.drop("target",axis=1)

# create y (labels)
y=heart_disease["target"]
pip install sklearn
# choose the right model and hyperparameters
from sklearn.ensemble import RandomForestClassifier
clf=RandomForestClassifier(n_estimators=100)

# we'll keep the default hyperparameters 
clf.get_params()
# 3. Fit the model to the  training data
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)
import sklearn
sklearn.show_versions();
clf.fit(X_train,y_train);
X_train
X_test
# make a prediction 
y_pred=clf.predict(X_test)
y_pred
y_test
# 4. Evaluate the model om the training data and test data 
clf.score(X_train, y_train)
clf.score(X_test,y_test)
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

print(classification_report(y_test,y_pred))
confusion_matrix(y_test,y_pred)
accuracy_score(y_test,y_pred)
# 5. Improve a model
# Try different amount of n_estimators 
np.random.seed(42)
for i in range(10,100,10):
    print(f"Trying model with {i} estimators..")
    clf=RandomForestClassifier(n_estimators=i).fit(X_train,y_train)
    print(f"Model accuracy on test set: {clf.score(X_test,y_test)*100:.2f}%")
    print("")
# 6. Save a model and load it
import pickle
pickle.dump(clf,open("random_forest_model_1.plk","wb"))
loaded_model=pickle.load(open("random_forest_model_1.plk","rb"))
loaded_model.score(X_test,y_test)
import warnings
warnings.filterwarnings("default")
# Standard imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>1. Getting our data ready to be used with machine learningThree main things we have to do:     1. Split the data into features and labels (usually 'X' & 'Y')    2. Filling (also called imputing) or disregarding missing values    3. Converting non-numerical values to numerical values (also called feature encoding)<jupyter_code>heart_disease=pd.read_csv("heart-disease.csv")
heart_disease
X=heart_disease.drop("target",axis=1)
X.head()
y=heart_disease["target"]
y.head()
# Split the data into training and test sets
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)
X_train
X_train.shape, X_test.shape, y_train.shape, y_test.shape
X.shape[0]*0.8
len(heart_disease)
242.4+61<jupyter_output><empty_output><jupyter_text>1.1 Make sure it's all numerical<jupyter_code>car_sales=pd.read_csv("car-sales-extended.csv")
car_sales.head()
len(car_sales)
car_sales.dtypes
# split into X/y
X=car_sales.drop("Price",axis=1)
y=car_sales["Price"]

# split into training and test
X_train, X_test, y_train, y_test= train_test_split(X,
                                                  y, 
                                                  test_size=0.2)
# Build machine learning model
from sklearn.ensemble import RandomForestRegressor

model=RandomForestRegressor()
model.fit(X_train,y_train)
model.score(X_test,y_test)
# Turn the categories into numbers

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

categorical_features=["Make","Colour","Doors"]
one_hot=OneHotEncoder()
transformer=ColumnTransformer([("one_hot",
                               one_hot,
                               categorical_features)],
                             remainder="passthrough")
transformed_X= transformer.fit_transform(X)
transformed_X
X.head()
pd.DataFrame(transformed_X)
dummies=pd.get_dummies(car_sales[["Make","Colour","Doors"]])
dummies
# Let's refit the model
np.random.seed(42)
X_train, X_test, y_train, y_test=train_test_split(transformed_X,
                                                 y,
                                                 test_size=0.2)

model.fit(X_train,y_train)
model.score(X_test,y_test)<jupyter_output><empty_output><jupyter_text>1.2 What if there were missing values?1. Fill them with some value (also known as imputation).2. Remove the samples with missing data altogether.<jupyter_code># import car sales missing data
car_sales_missing=pd.read_csv("car-sales-extended-missing-data.csv")
car_sales_missing.head()
car_sales_missing
car_sales_missing["Doors"].value_counts()
car_sales_missing.isna().sum()
# Create X & y
X=car_sales_missing.drop("Price",axis=1)
y=car_sales_missing["Price"]
# Let's try and convert our data to numbers 

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

categorical_features=["Make","Colour","Doors"]
one_hot=OneHotEncoder()
transformer=ColumnTransformer([("one_hot",
                               one_hot,
                               categorical_features)],
                             remainder="passthrough")
transformed_X= transformer.fit_transform(X)
transformed_X<jupyter_output><empty_output><jupyter_text>Option 1: Fill missing data with Pandas<jupyter_code># Fill the "Make" column
car_sales_missing["Make"].fillna("missing", inplace=True)

# Fill the "colour" column
car_sales_missing["Colour"].fillna("missing",inplace=True)

# Fill the "Odometer (KM)" column
car_sales_missing["Odometer (KM)"].fillna(car_sales_missing["Odometer (KM)"].mean(),inplace=True)

# Fill the "Door" column 
car_sales_missing["Doors"].fillna(4,inplace=True)
# Check our dataframe again
car_sales_missing.isna().sum()
# Remove rows with missing Price value
car_sales_missing.dropna(inplace=True)
car_sales_missing.isna().sum()
len(car_sales_missing)
X=car_sales_missing.drop("Price",axis=1)
y=car_sales_missing["Price"] 
# Let's try and convert our data to numbers 
# Turn the categories into numbers 

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

categorical_features=["Make","Colour","Doors"]
one_hot=OneHotEncoder()
transformer=ColumnTransformer([("one_hot",
                               one_hot,
                               categorical_features)],
                             remainder="passthrough")
transformed_X= transformer.fit_transform(car_sales_missing)
transformed_X<jupyter_output><empty_output><jupyter_text>Option 2: Fill missing values with Scikit-learn<jupyter_code>import pandas as pd
car_sales_missing=pd.read_csv("car-sales-extended-missing-data.csv")
car_sales_missing.head()
car_sales_missing.isna().sum()
car_sales_missing.dropna(subset=["Price"],inplace=True)
car_sales_missing.isna().sum()
# Split into X & Y
X=car_sales_missing.drop("Price",axis=1)
y=car_sales_missing["Price"]
X
y
# Fill missing values with Scikit-Learn 
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer

# Fill categorical values with 'missing' and numerical values with mean
cat_imputer=SimpleImputer(strategy="constant",fill_value="missing")
door_imputer=SimpleImputer(strategy="constant",fill_value=4)
num_imputer=SimpleImputer(strategy="mean")

#Define columns
cat_features=["Make","Colour"]
door_features=["Doors"]
num_features=["Odometer (KM)"]

#Create an imputer (something that fills missing data)
imputer=ColumnTransformer([(
"cat_imputer",cat_imputer,cat_features),
("door_imputer",door_imputer,door_features),
("num_imputer",num_imputer,num_features)])

#Transform the data 
filled_X=imputer.fit_transform(X)
filled_X
car_sales_filled=pd.DataFrame(filled_X,
                             columns=["Make","Colour","Doors","Odometer (KM)"])
car_sales_filled.head()
car_sales_filled.isna().sum()
#Turn the categories into numbers
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

categorical_features=["Make","Colour","Doors"]

one_hot=OneHotEncoder()
transformer=ColumnTransformer([("one_hot",
                       one_hot,
                       categorical_features)],
                      remainder="passthrough")
transformed_X=transformer.fit_transform(car_sales_filled)
transformed_X

# Now we've got our data as numbers and filled (no missing values)
# Let's fit a model 
import numpy as np

np.random.seed(42)
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test=train_test_split(transformed_X,
                                                 y,
                                                 test_size=0.2)

model=RandomForestRegressor(n_estimators=100)
model.fit(X_train,y_train)
model.score(X_test,y_test)


car_sales=pd.read_csv("car-sales-extended.csv")
len(car_sales_filled),len(car_sales)<jupyter_output><empty_output><jupyter_text>2. Chosing the right estimator/algorithm for your problemSome things to note:* Sklearn refers to machine learning model, algorithms as estimators* Classification problem - predicting a category (heart disease or not)    * Sometimes you'll see 'clf' (short for classifier) used as a classification estimators* Regression problem - predicting a number (selling price of a car)if you're working on a machine learning problem and looking to use Sklearn and not sure what model yo should use, refer to the sklearn machine learning map: https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html  2.1 Picking a machine learning model for a regression problem Lets's use the California Housing dataset.<jupyter_code># Get California Housing dataset 
from sklearn.datasets import fetch_california_housing
housing=fetch_california_housing() 
housing_df=pd.DataFrame(housing["data"],columns=housing["feature_names"])
housing_df.head(100)
housing_df["target"]=housing["target"]
housing_df.head()
housing_df=housing_df.drop("MedHouseVal",axis=1)
housing_df.head(10)
# import algorithm/estimator
from sklearn.linear_model import Ridge

# setup random seed
np.random.seed(42)

# Create the data
X=housing_df.drop("target",axis=1)
y=housing_df["target"]  #median house price in $100,000s

# split into train and test sets

X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# instantiate and fit the model (on the training set)
model=Ridge()
model.fit(X_train,y_train)
model.score(X_test,y_test)<jupyter_output><empty_output><jupyter_text>What if Ridge didn't work our the score didn't fit our needs?well, we could always try different model...how about we try an ensemble model (an ensemble is combination of smaller model to try and make better predictions than just a single model)sklearn's ensemble models can be found here: https://scikit-learn.org/stable/modules/ensemble.html<jupyter_code># import the RandomForestRegressor model class from the ensemble module 
from sklearn.ensemble import RandomForestRegressor

# setup random seed
np.random.seed(42)

#create the data 
X=housing_df.drop("target",axis=1)
y=housing_df["target"]

#Split into train and test sets
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# create random forest model
model=RandomForestRegressor(n_estimators=100)
model.fit(X_train,y_train)

# check the score of the model (on the test set)
model.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>2.2 Picking a machine learning model for a classification problem<jupyter_code>heart_disease=pd.read_csv("heart-disease.csv")
heart_disease.head()
len(heart_disease)<jupyter_output><empty_output><jupyter_text>Consulting the map and it says to try 'LinearSVC'.<jupyter_code># Import the linearSVC estimator class
from sklearn.svm import LinearSVC

# Setup random seed
np.random.seed(42)

# make the data
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

# split the data 
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# instantiate LineaSVC
clf=LinearSVC()
clf.fit(X_train, y_train)

# evaluate the linearSVC
clf.score(X_test, y_test)
heart_disease["target"].value_counts()
# Import the RandomForestClassifier estimator class
from sklearn.ensemble import RandomForestClassifier

# Setup random seed
np.random.seed(42)

# make the data
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

# split the data 
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# instantiate RandomForestClassifier
clf=RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)

# evaluate the RandomForestClassifier
clf.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Tidbit:    1. if you have structured data, use ensemble methods    2. if you have unstructured data, use deep learning or transfer learning<jupyter_code>heart_disease.head()<jupyter_output><empty_output><jupyter_text>3. Fit the model/algorithm on our data and use it to make predictions 3.1 Fitting the model to the dataDifferent names for: * X= features, features variables, data* y=labels, targets, target variables<jupyter_code># Import the RandomForestClassifier estimator class
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Setup random seed
np.random.seed(42)

# make the data
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

# split the data 
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# instantiate RandomForestClassifier
clf=RandomForestClassifier(n_estimators=100)

#Fit the model to the data (training the machine learning model)
clf.fit(X_train, y_train)

# evaluate the RandomForestClassifier (use the patterns the model has learned)
clf.score(X_test, y_test)
X.head()
y.tail()<jupyter_output><empty_output><jupyter_text>3.2 Make predictions using a machine learning model2 ways to make predictions1. predict()2. predict_proba()<jupyter_code># Use a trained model to make predictions
clf.predict(np.array([1,7,8,3,4])) #this doesn't work..
X_test.head()
X_test.shape
clf.predict(X_test)
y_test
np.array(y_test)
# Compare predictions to truth labels to evaluate  the model
y_preds=clf.predict(X_test)
np.mean(y_preds==y_test)
clf.score(X_test,y_test)
from sklearn.metrics import accuracy_score
accuracy_score(y_test,y_preds)
# predict_proba() returns probabilities of a classufication label
# make predictions with predict_proba()
clf.predict_proba(X_test[:5])
# Let's predict() on the same data 
# index del valor mayor
clf.predict(X_test[:5])
X_test[:5]
heart_disease["target"].value_counts()<jupyter_output><empty_output><jupyter_text>predict() can also be used for regression models.<jupyter_code>housing_df.head()
from sklearn.ensemble import RandomForestRegressor

np.random.seed(42)

# Create the data
X=housing_df.drop("target",axis=1)
y=housing_df["target"]

# split into training and test sets
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# create  model instance 
model= RandomForestRegressor(n_estimators=100)

# fit the model to the data
model.fit(X_train,y_train)

# make predictions 
y_preds=model.predict(X_test)
y_preds[:10]
np.array(y_test[:10])
# Compare the predictions to the truth
from sklearn.metrics import mean_absolute_error
mean_absolute_error(y_test,y_preds)
housing_df["target"].head()<jupyter_output><empty_output><jupyter_text>4. Evaluating a machine learning modelThree ways to evaluate Scikit-learn models/estimators:    1. Estimators's built-in 'score()' method    2. The 'scoring' parameter    3. Problem specific metric functions   4.1 evaluating a model with 'score' method<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np

heart_disease=pd.read_csv("heart-disease.csv")
np.random.seed(42)
heart_disease.head()
# Create X & y
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

#  Create train/test
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# Instantied Random Forest Classifier
clf=RandomForestClassifier(n_estimators=100)

# fit the model to the data (training the machine learning model)
clf.fit(X_train,y_train)
# The highest value from .score() method is 1.0, the lowest is 0.0
clf.score(X_train,y_train)
clf.score(X_test,y_test)<jupyter_output><empty_output><jupyter_text>Let's use the .score() method on our regresion ploblem...<jupyter_code># Get California Housing dataset 
from sklearn.datasets import fetch_california_housing
housing=fetch_california_housing()
housing_df=pd.DataFrame(housing["data"],columns=housing["feature_names"])
housing_df.head()
np.random.seed(42)

housing_df["target"]=housing["target"]
X=housing_df.drop("target",axis=1)
y=housing_df["target"]

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

model=RandomForestRegressor(n_estimators=100)

model.fit(X_train,y_train)
# the default score() evaluation metric is r_squared for regression algorithms 
# the highest =1.0 the lowest 0.0
model.score(X_test,y_test)
model.score(X_test,y_test)
housing_df.head()
y_test.head()
y_test.mean()<jupyter_output><empty_output><jupyter_text>4.2 Evaluating a model using a 'scoring' parameter<jupyter_code>from sklearn.model_selection import cross_val_score

np.random.seed(42)

X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

#  Create train/test
X_train, X_test, y_train, y_test=train_test_split(X,y,test_size=0.2)

# Instantied Random Forest Classifier
clf=RandomForestClassifier(n_estimators=100)

# fit the model to the data (training the machine learning model)
clf.fit(X_train,y_train);
clf.score(X_test,y_test)
cross_val_score(clf,X,y,cv=5)
cross_val_score(clf,X,y,cv=10)
np.random.seed(42)
# single training and test split score
clf_single_score=clf.score(X_test,y_test)

# take the mean of 5-fold cross validation score
clf_cross_val_score=np.mean(cross_val_score(clf,X,y,cv=5))

#compare two
clf_single_score,clf_cross_val_score
# default scoring parameter of classifier = mean accuracy
clf.score()
# scoring parameter set to None for default 
cross_val_score(clf,X,y,cv=5,scoring=None)<jupyter_output><empty_output><jupyter_text>Classifications model evaluation metrics    1. Accuracy    2. Area under ROC curve    3. Confusion matrix    4. Classification report<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

np.random.seed(42)

X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

clf=RandomForestClassifier(n_estimators=100)
cross_val_score=cross_val_score(clf,X,y,cv=5)
np.mean(cross_val_score)
print(f"heart disease Classifier Cross-Validated Accuracy: {np.mean(cross_val_score)*100:.2f}%")<jupyter_output>heart disease Classifier Cross-Validated Accuracy: 82.48%<jupyter_text>**Area under the receiver operating characteristic curve (AUC/ROC)*** Area under curve (AUC)* ROC curve ROC curves are a comparison of a model's true positive rate (tpr)  versus a models false positive rate (fpr).* True positive = model predicts 1 when truth is 1* False positive = model predicts 1 when truth is 0* True negative = model predicts 0 when truth is 0* False negative = model predicts 0 when truth is 1<jupyter_code># create X_test...

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)
from sklearn.metrics import roc_curve

#fit the classifier

clf.fit(X_train,y_train)


# make predictions with probabilities
y_probs=clf.predict_proba(X_test)

y_probs[:10],len(y_probs)
y_probs_positive=y_probs[:,1]
y_probs_positive[:10]
# calcular fpr, ttpr and thresholds
fpr, tpr, thresholds=roc_curve(y_test,y_probs_positive)

# check the false positive
fpr
# create a function for plotting ROC curve

import matplotlib.pyplot as plt

def plot_roc_curve(fpr,tpr):
    """
        Plot a ROC curver given the false positive rate (fpr)
        and true positive rate (tpr) of a model. 
    """
    
    # plot roc curve
    plt.plot(fpr,tpr,color="orange",  label="ROC")
    # plot line with not predictive power (baseline)
    plt.plot([0,1],[0,1],color="darkblue",linestyle="--",label="Guessing")
    
    # customize the plot 
    plt.xlabel("False positive rate (fpr)")
    plt.ylabel("True positive rate (tpr)")
    plt.title("Receiver operating characteristic (ROC) curve")
    plt.legend()
    plt.show()

plot_roc_curve(fpr,tpr)
from sklearn.metrics import roc_auc_score

roc_auc_score(y_test,y_probs_positive)
# plot perfect roc curve and AUC score
fpr,tpr,thresholds=roc_curve(y_test,y_test)
plot_roc_curve(fpr,tpr)
# perfect AUC score 
roc_auc_score(y_test,y_test)<jupyter_output><empty_output><jupyter_text>**Confusing matrix**A confusing matrix is a quick way to compare the labels a model predicts and the actuals labels it was a supposed to predict. In essence, giving you and idea of where the model is getting confused.<jupyter_code>from sklearn.metrics import confusion_matrix

y_preds=clf.predict(X_test)

confusion_matrix(y_test,y_preds)
# visualize confusing matrix with pd.crosstab()

pd.crosstab(y_test,y_preds,rownames=["Actual labels"],colnames=["Predicted labels"])
24+5+3+29
len(X_test)
# make confusion matrix more visual with Seaborn's heatmap()

import seaborn as sns

# set the font scale 
sns.set(font_scale=1.5)

# create a confusion matrix 
conf_mat=confusion_matrix(y_test,y_preds)

# plot it using seaborn
sns.heatmap(conf_mat)<jupyter_output><empty_output><jupyter_text>Creating a confusion matrix using Scikit-Learn<jupyter_code>import sklearn
sklearn.__version__
clf
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_estimator(estimator=clf,X=X,y=y);
ConfusionMatrixDisplay.from_predictions(y_true=y_test,y_pred=y_preds);<jupyter_output><empty_output><jupyter_text>**Classification report**<jupyter_code>from sklearn.metrics import classification_report

print(classification_report(y_test,y_preds))
# where precision and recall become valuable
disease_true=np.zeros(10000)
disease_true[0]=1 # only one positive case

disease_preds=np.zeros(10000) #models predict every case as 0

pd.DataFrame(classification_report(disease_true,disease_preds,output_dict=True))<jupyter_output>c:\users\ulisesj.000\appdata\local\programs\python\python38-32\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
c:\users\ulisesj.000\appdata\local\programs\python\python38-32\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
c:\users\ulisesj.000\appdata\local\programs\python\python38-32\lib\site-packages\sklearn\metrics\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavi[...]<jupyter_text>4.2.2 Regression model evaluation metricsThe one we're goin to cover are:1. R^2 (pronounced r-squared) or coefficient of determination2. Mean absolute error (MAE)3. Mean squared error (MSE)<jupyter_code>from sklearn.ensemble import RandomForestRegressor

np.random.seed(42)

X=housing_df.drop("target",axis=1)
y=housing_df["target"]

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

model=RandomForestRegressor(n_estimators=100)
model.fit(X_train,y_train)
model.score(X_test,y_test)
housing_df.head()
y_test[:5]
y_test.mean()
from sklearn.metrics import r2_score

# fill an array with y_test mean
y_test_mean=np.full(len(y_test),y_test.mean())
y_test_mean[:10]
r2_score(y_true=y_test,
        y_pred=y_test_mean)
r2_score(y_true=y_test,
        y_pred=y_test)<jupyter_output><empty_output><jupyter_text>**Mean absolute error (MAE)**MAE is the average of the absolute differences between predictions and actual values.It gives you idea of how wrong your models predictions are.<jupyter_code># MAE 
from sklearn.metrics import mean_absolute_error

y_preds=model.predict(X_test)
mae=mean_absolute_error(y_test,y_preds)
mae
y_preds
y_test[:20]
df=pd.DataFrame(data={"actual values":y_test,
               "predicted values":y_preds})
df["differences"]=df["predicted values"] -df["actual values"]
df.head()
df["differences"].mean()
#MAE using formulas and differences
np.abs(df.differences).mean()
# Get California Housing dataset 
from sklearn.datasets import fetch_california_housing
housing=fetch_california_housing() 
import pandas as pd
housing_df=pd.DataFrame(housing["data"],columns=housing["feature_names"])
from sklearn.ensemble import RandomForestRegressor
import numpy as np
from sklearn.model_selection import train_test_split

np.random.seed(42)
housing_df["target"]=housing["target"]

X=housing_df.drop("target",axis=1)
y=housing_df["target"]

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

model=RandomForestRegressor(n_estimators=100)
model.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>**Mean squared error (MSE)**MSE is the mean of the squared of the errors between actual and predicted values.<jupyter_code># Mean squared error
from sklearn.metrics import mean_squared_error

y_preds=model.predict(X_test)
mse=mean_squared_error(y_test,y_preds)
mse
df["squared differences"]=np.square(df["differences"])
df.head()
# calculate msi by hand
squared=np.square(df["differences"])
squared.mean()
df.iloc[0]["squared differences"].mean()<jupyter_output><empty_output><jupyter_text>4.2.3 Finally using the scoring parameter<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

heart_disease=pd.read_csv("heart-disease.csv")
np.random.seed(42)
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

clf=RandomForestClassifier(n_estimators=100)

np.random.seed(42)

#cross validation accuracy
cv_acc=cross_val_score(clf,X,y,cv=5,scoring=None) #if scoring =None, estimator's default scoring evaluating metric is used(accuracy for classification models)
cv_acc
# cross validation accuracy 
print(f"The cross validated accuracy is: {np.mean(cv_acc)*100:.2f}%")
np.random.seed(42)
cv_acc=cross_val_score(clf,X,y,cv=5, scoring="accuracy")
cv_acc<jupyter_output><empty_output><jupyter_text>Precision<jupyter_code>np.random.seed(42)
cv_precision=cross_val_score(clf,X,y,cv=5, scoring="precision")
cv_precision
print(f"The cross-validated precision is: {np.mean(cv_precision)}")<jupyter_output>The cross-validated precision is: 0.8329547346025924<jupyter_text>Recall<jupyter_code>np.random.seed(42)
cv_recall=cross_val_score(clf,X,y,cv=5,scoring="recall")
cv_recall
 print(f"The cross-validated recall is: {np.mean(cv_recall)}")<jupyter_output>The cross-validated recall is: 0.8545454545454545<jupyter_text>Let's see the scoring parameter being using for a regression problem...<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestRegressor

np.random.seed(42)
X=housing_df.drop("target",axis=1)
y=housing_df["target"]

model=RandomForestRegressor(n_estimators=100)
np.random.seed(42)
cv_r2=cross_val_score(model,X,y,cv=3,scoring=None)
np.mean(cv_r2)
cv_r2
# Mean squared error
cv_mse=cross_val_score(model,X,y,cv=3,scoring="neg_mean_squared_error")
np.mean(cv_mse)
cv_mse
# Mean absolute error
cv_mae=cross_val_score(model,X,y,cv=3,scoring="neg_mean_absolute_error")
np.mean(cv_mae)
 cv_mae<jupyter_output><empty_output><jupyter_text>4.3 Using different evaluation metrics as Scikit-learn functionsThe 3rd way to evaluate scikit learn machine learning models/estimators is to using 'sklearn.metrics' module<jupyter_code>from sklearn.metrics import accuracy_score,precision_score,recall_score,f1_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import numpy as np

np.random.seed(42)
#create X and y
heart_disease=pd.read_csv("heart-disease.csv")

X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

#split the data 
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

#create model 
clf=RandomForestClassifier() 

#fit the model 
clf.fit(X_train,y_train)

#make predictions
y_preds=clf.predict(X_test)

#evaluate model using evaluation functions 
print("Classifier metrics on the test set")
print(f"Accuracy: {accuracy_score(y_test,y_preds)*100:.2f}%")
print(f"Precison: {precision_score(y_test,y_preds)*100:.2f}%")
print(f"Recall: {recall_score(y_test,y_preds)*100:.2f}%")
print(f"F1: {f1_score(y_test,y_preds)*100:.2f}%")
from sklearn.metrics import r2_score,mean_absolute_error,mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

np.random.seed(42)

#create X and y
X=heart_disease.drop("target",axis=1)
y=heart_disease["target"]

#split the data 
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

#create model 
model=RandomForestRegressor()

#fit the model
model.fit(X_train,y_train) 

#make predictions
y_preds=model.predict(X_test)

#evaluate model using evaluation functions 
print("Regression metrics on the test set")
print(f"R2 score: {r2_score(y_test,y_preds)}")
print(f"MAE: {mean_absolute_error(y_test,y_preds)}")
print(f"MSE: {mean_squared_error(y_test,y_preds)}")<jupyter_output>Regression metrics on the test set
R2 score: 0.5106393318965518
MAE: 0.23754098360655734
MSE: 0.12204426229508197<jupyter_text>5. Improving a modelFirst predictions=baseline predictions.First model = baseline model.From a data perspective* Could we collect more data? (generally, the more data, the better)* Could we improve our data? From a model perspective:* Is there a better model we could use?* Could we improve the current model?Hyperparameters vs. parameters* Parameters = model find these patterns on data * Hyperparameters = setting from on a model you can adjust to (potencially) improve its ability to find pattern Three ways to adjust hyperparameters:1. By hand2. Randomly with RandomSearchCV3. Exhaustively with GridSearchCV<jupyter_code>from sklearn.ensemble import RandomForestClassifier
clf=RandomForestClassifier()
clf.get_params()<jupyter_output><empty_output><jupyter_text>5.1 Turning hyperparameters by handLet's make 3 sets, training, validation and test.<jupyter_code>clf.get_params()<jupyter_output><empty_output><jupyter_text>We're going to try and adjust.* `max_depth`* `max_features`* `min_samples_leaf`* `min_samples_split`* `n_estimators`<jupyter_code>from sklearn.metrics import accuracy_score,precision_score,recall_score,f1_score
def evaluate_preds(y_true,y_preds):
    """
    Performs evaluation comparison on y_true labels vs. y_pred labels
    """
    accuracy=accuracy_score(y_true,y_preds)
    precision=precision_score(y_true,y_preds)
    recall=recall_score(y_true,y_preds)
    f1=f1_score(y_true,y_preds)
    metric_dict={"accuracy":round(accuracy,2),
                "precision":round(precision,2),
                "recall":round(recall,2),
                "f1":round(f1,2)}
    print(f"Acc:{accuracy*100:.2f}%")
    print(f"Precision: {precision*100:.2f}")
    print(f"Recall: {recall:.2f}")
    print(f"F1 score: {f1:.2f}")
    
    return metric_dict
from sklearn.model_selection import train_test_split
import pandas as pd

heart_disease=pd.read_csv("heart-disease.csv")
heart_disease.head()
from sklearn.ensemble import RandomForestClassifier
import numpy as np

np.random.seed(42)

# shuffle the data 
heart_disease_shuffled=heart_disease.sample(frac=1)

# split into X & y
X=heart_disease_shuffled.drop("target",axis=1)
y=heart_disease_shuffled["target"]

# split the data into train, validation
train_split=round(0.7*len(heart_disease_shuffled)) # 70% of data
valid_split=round(train_split+0.15*len(heart_disease_shuffled)) # 15% of data
X_train,y_train=X[:train_split],y[:train_split]
X_valid,y_valid=X[train_split:valid_split],y[train_split:valid_split]
X_test,y_test=X[valid_split:],y[valid_split:]

len(X_train),len(X_valid),len(X_test)

clf=RandomForestClassifier()
clf.fit(X_train,y_train)

# Make baseline predictions
y_preds= clf.predict(X_valid)

# Evaluate the classifier on validation set
baseline_metrics=evaluate_preds(y_valid,y_preds)
baseline_metrics
np.random.seed(42)
clf_2=RandomForestClassifier(n_estimators=100)
clf_2.fit(X_train,y_train)

# make predicitions
y_preds2=clf_2.predict(X_valid)

# evaluate the second classifier
clf_2_metrics=evaluate_preds(y_valid,y_preds2)<jupyter_output>Acc:82.22%
Precision: 84.00
Recall: 0.84
F1 score: 0.84<jupyter_text>5.2 Hyperparameters tuning with RandomizedSearchCV<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
import numpy as np

grid={"n_estimators":[10,100,200,500,1000,1200],
     "max_depth":[None,5,10,20,30],
     "max_features":["auto","sqrt"],
     "min_samples_split":[2,4,6],
      "min_samples_leaf":[1,2,6]}
np.random.seed(42) 

# split into X & y
X=heart_disease_shuffled.drop("target",axis=1)
y=heart_disease_shuffled["target"]

# split into train and test sets 
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

# Instantiate RandomForestClassifier
clf=RandomForestClassifier(n_jobs=1)

# setup RandomizedSearchCV
rs_clf=RandomizedSearchCV(estimator=clf, 
                        param_distributions=grid,
                         n_iter=10,
                         cv=5,
                         verbose=2)

# fit the RandomizedSearchCV version of clf
rs_clf.fit(X_train,y_train)
rs_clf.best_params_
# make predictions with the best hyperparameters
rs_y_preds=rs_clf.predict(X_test)

# EVALUATE The predictions
rs_metrics=evaluate_preds(y_test,rs_y_preds)<jupyter_output>Acc:81.97%
Precision: 77.42
Recall: 0.86
F1 score: 0.81<jupyter_text>5.3 Hyperparameters tuning with GridSearchCV<jupyter_code>grid
grid_2={"n_estimators":[100,200,500],
     "max_depth":[None],
     "max_features":["auto","sqrt"],
     "min_samples_split":[6],
     "min_samples_leaf":[1,2]}
from sklearn.model_selection import GridSearchCV, train_test_split

np.random.seed(42) 

# split into X & y
X=heart_disease_shuffled.drop("target",axis=1)
y=heart_disease_shuffled["target"]

# split into train and test sets 
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

# Instantiate RandomForestClassifier
clf=RandomForestClassifier(n_jobs=1)

# setup GridSearchCV
gs_clf=GridSearchCV(estimator=clf, 
                        param_grid=grid_2,
                         cv=5,
                         verbose=2)

# fit the GridSearchCV version of clf
gs_clf.fit(X_train,y_train)

gs_clf.best_params_
gs_y_preds=gs_clf.predict(X_test)

# evaluate the predictions
gs_metrics=evaluate_preds(y_test,gs_y_preds)<jupyter_output>Acc:78.69%
Precision: 74.19
Recall: 0.82
F1 score: 0.78<jupyter_text>Let's compare our different models metric<jupyter_code>import matplotlib.pyplot as plt
compare_metrics=pd.DataFrame({"baseline":baseline_metrics,
                            "clf_2":clf_2_metrics,
                            "random search":rs_metrics,
                            "grid search":gs_metrics})
compare_metrics.plot.bar(figsize=(10,8));<jupyter_output><empty_output><jupyter_text>6 Saving and loading machine learning models Two ways to save and load machine learning models:1. With python's `pickle` module2. With the `joblib` module**Pickle**<jupyter_code>import pickle

# save an existing model to file
pickle.dump(gs_clf,open("gs_random_random_forest_model_1.plk","wb"))
# laod a save model
loaded_pickle_model=pickle.load(open("gs_random_random_forest_model_1.plk","rb"))
# make some predictions
pickle_y_preds=loaded_pickle_model.predict(X_test)
evaluate_preds(y_test,pickle_y_preds)<jupyter_output>Acc:78.69%
Precision: 74.19
Recall: 0.82
F1 score: 0.78<jupyter_text>**Joblib**<jupyter_code>from joblib import dump,load

# save model to file
dump(gs_clf,filename="gs_random_forest_model_1.joblib")
# import a save joblib model
loaded_job_model=load(filename="gs_random_forest_model_1.joblib")
# make and evaluate joblib predictions
joblib_y_preds=loaded_job_model.predict(X_test)
evaluate_preds(y_test,joblib_y_preds)<jupyter_output>Acc:78.69%
Precision: 74.19
Recall: 0.82
F1 score: 0.78<jupyter_text>7. Putting all together!<jupyter_code>data=pd.read_csv("car-sales-extended-missing-data.csv")
data.head()
data.dtypes
data.isna().sum()<jupyter_output><empty_output><jupyter_text>Steps we want to do (all in one cell):1. Fill missing data2. Convert data to numbers3. Build a model on data<jupyter_code>#  getting data ready
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

# modelling
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split,GridSearchCV

# setup random seed
import numpy as np
np.random.seed(42)

# import data and drop rows with the missing labels
data=pd.read_csv("car-sales-extended-missing-data.csv")
data.dropna(subset=["Price"],inplace=True)

# define different features on transformer pipeline 
categorical_features=["Make","Colour"]
categorical_transformer=Pipeline(steps=[
    ("imputer",SimpleImputer(strategy="constant",fill_value="missing")),
    ("onehot",OneHotEncoder(handle_unknown="ignore"))])

door_feature=["Doors"]
door_transformer=Pipeline(steps=[
    ("imputer",SimpleImputer(strategy="constant",fill_value=4))
])

numerical_features=["Odometer (KM)"]
numerical_transformer=Pipeline(steps=[
    ("imputer",SimpleImputer(strategy="mean"))
])

# setup the preprocessing steps(fill missing values, then convert to numbers)
preprocessor=ColumnTransformer( transformers=[
    ("cat",categorical_transformer,categorical_features),
    ("door",door_transformer,door_feature),
    ("num",numerical_transformer,numerical_features)
])

# create a preprocessing and modelling pipeline
model=Pipeline(steps=[("preprocessor",preprocessor),
                     ("model",RandomForestRegressor())])

# split data 
X = data.drop("Price",axis=1)
y = data["Price"]
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2)

# fit and score model
model.fit(X_train,y_train)
model.score(X_test,y_test)
data.isna().sum()<jupyter_output><empty_output><jupyter_text>it's also possible to use `GridSearchCV` or `RandomizedSearchCV` with our `Pipeline`<jupyter_code># use GridSearchCV with our regression Pipeline
from sklearn.model_selection import GridSearchCV

pipe_grid={
    "preprocessor__num__imputer__strategy":["mean","median"],
    "model__n_estimators":[100,1000],
    "model__max_depth":[None,5],
    "model__max_features":["auto"],
    "model__min_samples_split":[2,4]
}

gs_model=GridSearchCV(model,pipe_grid,cv=5,verbose=2)
gs_model.fit(X_train,y_train)
gs_model.score(X_test,y_test)
import pickle

# save an existing model to file
pickle.dump(gs_model,open("final_example.plk","wb"))<jupyter_output><empty_output>