<jupyter_start><jupyter_text>Подготовка обучающей и тестовой выборки, кросс-валидация и подбор гиперпараметров на примере метода ближайших соседей.   Задание1. Выберите набор данных (датасет) для решения задачи классификации или регресии.2. В случае необходимости проведите удаление или заполнение пропусков и кодирование категориальных признаков.3. С использованием метода train_test_split разделите выборку на обучающую и тестовую.4. Обучите модель ближайших соседей для произвольно заданного гиперпараметра K. Оцените качество модели с помощью трех подходящих для задачи метрик.5. Постройте модель и оцените качество модели с использованием кросс-валидации. Проведите эксперименты с тремя различными стратегиями кросс-валидации.6. Произведите подбор гиперпараметра K с использованием GridSearchCV и кросс-валидации.7. Повторите пункт 4 для найденного оптимального значения гиперпараметра K. Сравните качество полученной модели с качеством модели, полученной в пункте 4.8. Постройте кривые обучения и валидации.<jupyter_code>import pandas as pd
from enum import Enum
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from operator import itemgetter
import matplotlib.pyplot as plt
import numpy as np
import sklearn.metrics
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error as MSE, median_absolute_error as MedAE, r2_score as R2
from sklearn.model_selection import cross_val_score, cross_validate
from sklearn.model_selection import KFold, RepeatedKFold, LeaveOneOut, LeavePOut, ShuffleSplit, StratifiedKFold
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import learning_curve, validation_curve
data = pd.read_csv("train.csv",sep=",",encoding = 'cp1251')
data.head()
data.shape<jupyter_output><empty_output><jupyter_text>Проверим наличие пропущенных значениений<jupyter_code>nd = data.columns[data.isnull().any()]

print(len(nd))<jupyter_output>0<jupyter_text>Как мы видим пропущенных значений нет  Проверим наличие категориальных значений<jupyter_code>cats = [col for col in data.columns if data[col].dtype=="object"]
print(len(cats))<jupyter_output>12<jupyter_text>Имеем 12 столбцов с категориальными признаками Заменим категориальные признаки числовыми используя Label encoding<jupyter_code>le = LabelEncoder()
for col in cats:
    data[col]=le.fit_transform(data[col])
newcats = [col for col in data.columns if data[col].dtype=="object"]
print(len(newcats))
y=data["value"]
y.unique()
x=data.drop("value",axis=1)
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)
# Размер тренировочной выборки
print(X_train.shape, y_train.shape)
# Размер тестовой выборки
print(X_test.shape, y_test.shape)<jupyter_output>(6980, 55) (6980,)
(1746, 55) (1746,)<jupyter_text>Обучим модель используя метод k близжайших значений<jupyter_code>KNN = KNeighborsRegressor()
KNN.fit(X_train,y_train)
a= KNN.predict(X_test)<jupyter_output><empty_output><jupyter_text>Проверим модель при помощи метрик R2, MSE, MedAE<jupyter_code>R2(y_test, a)
MSE(y_test, a)
MedAE(y_test, a)<jupyter_output><empty_output><jupyter_text>Обучим модель при помощи кросс валидации тремя стратегиями и подберем гиперпараметры<jupyter_code>ll = [1,2,3,4,5,10,25,40,50,100,250,500,1000]<jupyter_output><empty_output><jupyter_text>1. KFold<jupyter_code>scores = cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,cv=KFold(n_splits=5), scoring = 'r2')
print(np.mean(scores))
scores = -1*cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,cv=KFold(n_splits=5), scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1*cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,cv=KFold(n_splits=5), scoring = 'neg_median_absolute_error')
print(np.mean(scores))
random_search_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'r2', 
                             cv= KFold(n_splits=5))
random_search_kFold.fit(x,y)
random_search_kFold.best_params_
random_search_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_mean_squared_error', 
                             cv= KFold(n_splits=5))
random_search_kFold.fit(x,y)
random_search_kFold.best_params_
random_search_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_median_absolute_error', 
                             cv= KFold(n_splits=5))
random_search_kFold.fit(x,y)
random_search_kFold
scores = cross_val_score(KNeighborsRegressor(n_neighbors=1000),x, y,
                         cv=KFold(n_splits=5), scoring = 'r2')
print(np.mean(scores))
scores = -1*cross_val_score(KNeighborsRegressor(n_neighbors=1000),x, y,
                            cv=KFold(n_splits=5), scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1*cross_val_score(KNeighborsRegressor(n_neighbors=10),x, y,
                            cv=KFold(n_splits=5), scoring = 'neg_median_absolute_error')
print(np.mean(scores))<jupyter_output>163.45470000000003<jupyter_text>2. ShuffleSplit<jupyter_code>scores = cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,
                         cv=ShuffleSplit(n_splits=5, test_size=0.25), scoring = 'r2')
print(np.mean(scores))
scores = -1 * cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,
                              cv=ShuffleSplit(n_splits=5, test_size=0.25), 
                              scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1* cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,
                             cv=ShuffleSplit(n_splits=5, test_size=0.25), 
                             scoring = 'neg_median_absolute_error')
print(np.mean(scores))
random_search_Shuffled = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'r2', 
                             cv= ShuffleSplit(n_splits=5, test_size=0.25))
random_search_Shuffled.fit(x,y)
random_search_Shuffled.best_params_
random_search_Shuffled = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_mean_squared_error', 
                             cv= ShuffleSplit(n_splits=5, test_size=0.25))
random_search_Shuffled.fit(x,y)
random_search_Shuffled.best_params_
random_search_Shuffled = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_median_absolute_error', 
                             cv= ShuffleSplit(n_splits=5, test_size=0.25))
random_search_Shuffled.fit(x,y)
random_search_Shuffled.best_params_
scores = cross_val_score(KNeighborsRegressor(n_neighbors=3),x, y,
                         cv=ShuffleSplit(n_splits=5, test_size=0.25), 
                         scoring = 'r2')
print(np.mean(scores))
scores = -1 * cross_val_score(KNeighborsRegressor(n_neighbors=3),x, y,
                              cv=ShuffleSplit(n_splits=5, test_size=0.25), 
                              scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1* cross_val_score(KNeighborsRegressor(n_neighbors=3),x, y,
                             cv=ShuffleSplit(n_splits=5, test_size=0.25), 
                             scoring = 'neg_median_absolute_error')
print(np.mean(scores))<jupyter_output>81.00933333333333<jupyter_text>3. RepeatedKFold<jupyter_code>scores = cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,
                         cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                         scoring = 'r2')
print(np.mean(scores))
scores = -1 * cross_val_score(KNeighborsRegressor(n_neighbors=5),x, y,
                              cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                              scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1* cross_val_score(KNeighborsRegressor(n_neighbors=5),x, 
                             y,cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                             scoring = 'neg_median_absolute_error')
print(np.mean(scores))
random_search_n_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'r2', 
                             cv= RepeatedKFold(n_splits=5, n_repeats=5))
random_search_n_kFold.fit(x, y)
random_search_n_kFold.best_params_
random_search_n_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_mean_squared_error', 
                             cv= RepeatedKFold(n_splits=5, n_repeats=5))
random_search_n_kFold.fit(x, y)
random_search_n_kFold.best_params_
random_search_n_kFold = GridSearchCV(estimator= KNeighborsRegressor(),
                             param_grid= {'n_neighbors': ll},
                             scoring= 'neg_median_absolute_error', 
                             cv= RepeatedKFold(n_splits=5, n_repeats=5))
random_search_n_kFold.fit(x, y)
random_search_n_kFold.best_params_
scores = cross_val_score(KNeighborsRegressor(n_neighbors=3),x, y,
                         cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                         scoring = 'r2')
print(np.mean(scores))
scores = -1 * cross_val_score(KNeighborsRegressor(n_neighbors=3),x, y,
                              cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                              scoring = 'neg_mean_squared_error')
print(np.mean(scores))
scores = -1* cross_val_score(KNeighborsRegressor(n_neighbors=2),x, y,
                             cv=RepeatedKFold(n_splits=5, n_repeats=5), 
                             scoring = 'neg_median_absolute_error')
print(np.mean(scores))
def plot_validation_curve(estimator, title, X, y, 
                          param_name, param_range, cv, 
                          scoring='r2'):
                                                   
    train_scores, test_scores = validation_curve(
        estimator, X, y, param_name=param_name, param_range=param_range,
        cv=cv, scoring=scoring, n_jobs=1)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)

    plt.title(title)
    plt.xlabel(param_name)
    plt.ylabel("Score")
    plt.ylim(0.0, 1.1)
    lw = 2
    plt.plot(param_range, train_scores_mean, label="Training score",
                 color="darkorange", lw=lw)
    plt.fill_between(param_range, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.2,
                     color="darkorange", lw=lw)
    plt.plot(param_range, test_scores_mean, label="Cross-validation score",
                 color="navy", lw=lw)
    plt.fill_between(param_range, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.2,
                     color="navy", lw=lw)
    plt.legend(loc="best")
    return plt
n_range = range(3,33,3)
plot_validation_curve(KNeighborsRegressor(), 'knn', 
                      x, y, 
                      param_name='n_neighbors', param_range=n_range, 
                      cv=RepeatedKFold(n_splits=5, n_repeats=5), scoring="r2")<jupyter_output><empty_output>