<jupyter_start><jupyter_text>K Nearest Neighbor A movie-ing target  Friend inventory Use handout![bullseye3](img/bullseye2.png)  Friend Inventory What would each person choose?![movies](img/movies.png)  Friend Inventory Decision majority by ringIf you just polled the inner ring of people, what movie would you end up seeing?- How about if you polled the first *and* second ring?- The first three rings?- All of it?Share with your neighbor what movie you'd end up seeing.  Friend Inventory What did you end up seeing?Who's movie choices changed based on how many people you polled?![movies](img/movies.png)  Friend Inventory What's the "algorithm" we used for this process, in normal words?  How does this relate to K nearest neighbor(knn)? ![annotate](img/bullseye-annotate.png)  Starting OffWe are going to revisit the pima native Americans diabetes dataset. Can we use knn to classify people correctly and then predict if someone will have diabetes?  Implementing a K-Nearest Neighbors Model  Objective: - To be able to implement a KNN classification model, and utilize a visualization to identify the best value of K.  Agenda- What is the **K-nearest neighbors** classification model?- How do you use the sklearn grid search function?  K-nearest neighbors (KNN) classification **Lazy learning** or **Instance-based (IB)** learning methods simply store the training examples and postpone the generalization (building a model) until a new instance must be classified or prediction made. **Non-parametric models** assume that the data distribution cannot be defined interms of such a finite set of parameters.  How does the KNN algorithm work?  What should the grey point be?  KNN has the following basic steps:  VotingHow to break ties:1. When doing a binary classification, often use a odd K to avoid ties.2. Multiple approaches for Multiclass problems:    - Reduce the K by 1 to see who wins.    - Weight the votes based on the distance of the neighbors  Example training dataThis example uses a multi-class problem and each color represents a different class.   KNN classification map (K=1)![1NN classification map](img/04_1nn_map.png)  KNN classification map (K=5)![5NN classification map](img/04_5nn_map.png)  A bit more math Euclidean Distance**Euclidean distance** refers to the distance between two points. These points can be in different dimensional space and are represented by different forms of coordinates. In one-dimensional space, the points are just on a straight number line.  A bit more math Measuring distance in a 2-d SpaceIn two-dimensional space, the coordinates are given as points on the x- and y-axes![alt text](img/euclidean-distance.png)  A bit more math Measuring distance in a 3-d SpaceIn three-dimensional space, x-, y- and z-axes are used. $$\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2 +  (z_1-z_2)^2}$$![alt text](img/vectorgraph.jpg)  A bit more math Euclidean Distance Equation![alt text](img/euclidean-equation.png)The source of this formula is in the Pythagorean theorem.   Implementing the KNN Classifier with SKlearn  Reviewing the Pima dataset  Loading the data[following this analysis for part of this section](https://www.kaggle.com/shrutimechlearn/step-by-step-diabetes-classification-knn-detailed)The Question creeping out of this summary- Can minimum value of below listed columns be zero (0)?- On these columns, a value of zero does not make sense and thus indicates missing value.Following columns or variables have an invalid zero value:- Glucose- BloodPressure- SkinThickness- Insulin- BMI<jupyter_code>import pandas as pd

diabetes = pd.read_csv('diabetes.csv')
# print the shapes of X and y
print(X.shape)
print(y.shape)
# Use x and y variables to split the training data into train and test set

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)<jupyter_output><empty_output><jupyter_text>Importance of ScalingCompare how the different the data looks when it is scaled versus non-scaled  Should we use a Standard Scaler or Min-Max Scaler?https://sebastianraschka.com/Articles/2014_about_feature_scaling.htmlhttp://datareality.blogspot.com/2016/11/scaling-normalizing-standardizing-which.html<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler

scaler = StandardScaler()  
scaler.fit(X_train)

X_train = scaler.transform(X_train)  
X_test = scaler.transform(X_test)<jupyter_output>/Users/swilson5/anaconda3/lib/python3.6/site-packages/sklearn/preprocessing/data.py:625: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  return self.partial_fit(X, y)
/Users/swilson5/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  import sys
/Users/swilson5/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:8: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.<jupyter_text>scikit-learn 4-step modeling pattern ![steps](img/sklearnsteps.png) **Step 1:** Import the class you plan to use<jupyter_code>from sklearn.neighbors import KNeighborsClassifier<jupyter_output><empty_output><jupyter_text>**Step 2:** "Instantiate" the "estimator"- "Estimator" is scikit-learn's term for model- "Instantiate" means "make an instance of"<jupyter_code>knn = KNeighborsClassifier(n_neighbors=1)<jupyter_output><empty_output><jupyter_text>**Class specifications**- Name of the object does not matter- Can specify tuning parameters (aka "hyperparameters") during this step- All parameters not specified are set to their defaults<jupyter_code>print(knn)<jupyter_output>KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           metric_params=None, n_jobs=None, n_neighbors=1, p=2,
           weights='uniform')<jupyter_text>**Step 3:** Fit the model with data (aka "model training")- Model is learning the relationship between X and y- Occurs in-place<jupyter_code>knn.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>**Step 4:** Predict the response for a new observation- New observations are called "out-of-sample" data- Uses the information it learned during the model training process<jupyter_code># make class predictions for the testing set
y_pred_class = knn.predict(X_test)
# calculate accuracy
from sklearn import metrics
print('Accuracy:' + str(metrics.accuracy_score(y_test, y_pred_class)))
print('F1: ' + str(metrics.f1_score(y_test, y_pred_class)))
import matplotlib.pyplot as plt
import itertools
import numpy as np
%matplotlib inline
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
plt.rcParams["figure.figsize"] = [10,5]
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion Matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test,y_pred_class)
classes = ['Perished', 'Survived']
plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues)<jupyter_output><empty_output><jupyter_text>Using a different value for K<jupyter_code># instantiate the model (using the value K=5)
knn = KNeighborsClassifier(n_neighbors=5)

# fit the model with data
knn.fit(X_train, y_train)

# make class predictions for the testing set
y_pred_class = knn.predict(X_test)
print('Accuracy:' + str(metrics.accuracy_score(y_test, y_pred_class)))
print('F1: ' + str(metrics.f1_score(y_test, y_pred_class)))
from sklearn.metrics import classification_report  

print(classification_report(y_test, y_pred_class))
cm = confusion_matrix(y_test,y_pred_class)
classes = ['Perished', 'Survived']
plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues)<jupyter_output><empty_output><jupyter_text>Search for an optimal value of K for KNN<jupyter_code>k_range = list(range(1, 4))
k_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    y_predict = knn.predict(X_test)
    score = metrics.f1_score(y_test, y_predict, average='weighted')
    k_scores.append( score)
print(k_scores)<jupyter_output><empty_output><jupyter_text>Visual comparison of different $K$sThis is not an ROC curve<jupyter_code>import matplotlib.pyplot as plt  

plt.figure(figsize=(12, 6))  
plt.plot(k_range, k_scores, color='red', linestyle='dashed', marker='o',  
         markerfacecolor='blue', markersize=10)
plt.title('F1 score by K Value')  
plt.xlabel('K Value')  
plt.ylabel('F1 Score') 
plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup