<jupyter_start><jupyter_text>Data Mining and Machine Learning <NAME> February 2021 Mean/median/mode and k-nn imputation Datasets: Titanic, Breastw y Segment<jupyter_code>import pandas as pd
import numpy as np
from sklearn.preprocessing  import LabelEncoder<jupyter_output><empty_output><jupyter_text>Ejemplo1. Leyendo los datos de Titanic<jupyter_code>titanic=pd.read_csv('https://raw.githubusercontent.com/eacunafer/DataAnalysiswithPython3/master/Datasets/titanic.csv',header=0,sep=',',na_values='')
titanic.info()
titanic.head(39)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  1309 non-null   int64  
 1   Pclass       1309 non-null   int64  
 2   Name         1309 non-null   object 
 3   Sex          1309 non-null   object 
 4   Age          1046 non-null   float64
 5   SibSp        1309 non-null   int64  
 6   Parch        1309 non-null   int64  
 7   Ticket       1309 non-null   object 
 8   Fare         1308 non-null   float64
 9   Cabin        295 non-null    object 
 10  Embarked     1307 non-null   object 
 11  Survived     1309 non-null   int64  
dtypes: float64(2), int64(5), object(5)
memory usage: 122.8+ KB<jupyter_text>Deleting columns: PassengerID, Name and cabin(it has a lot of missing values)<jupyter_code>titanic1=titanic.iloc[:,[1,3,4,5,6,8,10]]
titanic1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Pclass    1309 non-null   int64  
 1   Sex       1309 non-null   object 
 2   Age       1046 non-null   float64
 3   SibSp     1309 non-null   int64  
 4   Parch     1309 non-null   int64  
 5   Fare      1308 non-null   float64
 6   Embarked  1307 non-null   object 
dtypes: float64(2), int64(3), object(2)
memory usage: 71.7+ KB<jupyter_text>Hay missings en las variables Age, Fare que son continuas y en Embarked que es categorica<jupyter_code>#computing the values to  be used in the imputation
t3=titanic1['Age'].mean()
t6=titanic1['Fare'].mean()
t7=titanic1['Embarked'].mode().iloc[0]
#Performing the imputacion
values = {'Age': t3, 'Fare': t6, 'Embarked': t7}
titanic2=titanic1.fillna(value=values)
titanic2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Pclass    1309 non-null   int64  
 1   Sex       1309 non-null   object 
 2   Age       1309 non-null   float64
 3   SibSp     1309 non-null   int64  
 4   Parch     1309 non-null   int64  
 5   Fare      1309 non-null   float64
 6   Embarked  1309 non-null   object 
dtypes: float64(2), int64(3), object(2)
memory usage: 71.7+ KB<jupyter_text>Imputando los missing values por la media si el atributo es continuo y por la moda si el atributo es nominal<jupyter_code>#Imputando los missing values por la media si el atributo es continuo
#y por la moda si el atributo es nominal
from sklearn.impute import SimpleImputer
from sklearn.base import TransformerMixin

class DataFrameImputer(TransformerMixin):

    def fit(self, X, y=None):

        self.fill = pd.Series([X[c].value_counts().index[0]
            if X[c].dtype == np.dtype('O') else X[c].mean() for c in X],
            index=X.columns)

        return self

    def transform(self, X, y=None):
        return X.fillna(self.fill)
X=titanic1
xt = DataFrameImputer().fit_transform(X)
xt.info()
#Imputando los missing values por la mediana si el atributo es continuo
#y por la moda si el atributo es nominal

class DataFrameImputer(TransformerMixin):

    def fit(self, X, y=None):

        self.fill = pd.Series([X[c].value_counts().index[0]
            if X[c].dtype == np.dtype('O') else X[c].median() for c in X],
            index=X.columns)

        return self

    def transform(self, X, y=None):
        return X.fillna(self.fill)
X=titanic1
xt = DataFrameImputer().fit_transform(X)
xt.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Pclass    1309 non-null   int64  
 1   Sex       1309 non-null   object 
 2   Age       1309 non-null   float64
 3   SibSp     1309 non-null   int64  
 4   Parch     1309 non-null   int64  
 5   Fare      1309 non-null   float64
 6   Embarked  1309 non-null   object 
dtypes: float64(2), int64(3), object(2)
memory usage: 71.7+ KB<jupyter_text>IMPUTACION k-nn<jupyter_code>import numpy as np
from sklearn.impute import KNNImputer
X = [[4, 5, 6,np.nan], [5, 1, 1,4], [7, 9,2, 5], [8, 2,8, 5],[6,4,2,6]]
#imputer = KNNImputer(n_neighbors=3)
#imputer.fit_transform(X)
KNNImputer(n_neighbors=3).fit_transform(X)<jupyter_output><empty_output><jupyter_text>Ejemplo 2. k-nn imputation applied to the  Breast-wisconsin dataset (todas las features son continuas)<jupyter_code>#Reading the data from the UCI
breastdf=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data",header=None, sep=",",na_values=['?'])
#breastdf=pd.read_csv("c://PW-PR/breast-cancer-wisconsin.data",header=None, sep=",",na_values=['?'])
breastdf.columns=['idx','v1','v2','v3','v4','v5','v6','v7','v8','v9','class']
breastdf.info()
breastdf.head()
breast1=breastdf.iloc[:,1:11]
breast1.head()
b1=np.array(breast1)
imputer = KNNImputer(n_neighbors=5)
data_imp=imputer.fit_transform(b1)
completo5=pd.DataFrame(data_imp)
completo5.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 699 entries, 0 to 698
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   0       699 non-null    float64
 1   1       699 non-null    float64
 2   2       699 non-null    float64
 3   3       699 non-null    float64
 4   4       699 non-null    float64
 5   5       699 non-null    float64
 6   6       699 non-null    float64
 7   7       699 non-null    float64
 8   8       699 non-null    float64
 9   9       699 non-null    float64
dtypes: float64(10)
memory usage: 54.7 KB<jupyter_text>Ejemplo 3. Imputacion k-nn a Titanic. Hay que convertir las variables categoticas a numericas porque knn-imputacion solo esta implementada para variables numericas  Convirtiendo las variables nominales en numericas, debido a que la imputacion knn de python no trabaja con variables mezcladas Debe hacerse con cuidado porque el atributo categorico contiene missing value y el LabelEncoder puede asignarle un valor en lugar  de ignorarlo. LabelEncoder codifica los valores categoricas en numero enteros desde 0 hasta el numero de categorias-1. Similar resultado se puede conseguir con cat.codes de Pandas<jupyter_code>#Guardando las posiciones donde estan los missings
titanic3=titanic1.copy()
mask = titanic3.isnull()
print(mask)
#Convirtiendo a numero los atributos categoricos
titanic1=titanic1.astype(str).apply(LabelEncoder().fit_transform)
#Sustituyendo con NaN las posiciones donde los atributos categoricos tienen missings 
titanic4=titanic1.where(~mask, titanic3).copy()
titanic4.tail(7)
titanic4.info()
#Convirtiendo en Float las variables categoricas donde ahi missings
titanic4['Age'] = titanic4['Age'].astype('float64')
titanic4['Fare'] = titanic4['Fare'].astype('float64')
titanic4['Embarked'] = titanic4['Embarked'].astype('float64')
b2=np.array(titanic4)
imputer = KNNImputer(n_neighbors=5)
titanic_knnimp=imputer.fit_transform(b2)
completo=pd.DataFrame(titanic_knnimp)
completo.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 7 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   0       1309 non-null   float64
 1   1       1309 non-null   float64
 2   2       1309 non-null   float64
 3   3       1309 non-null   float64
 4   4       1309 non-null   float64
 5   5       1309 non-null   float64
 6   6       1309 non-null   float64
dtypes: float64(7)
memory usage: 71.7 KB<jupyter_text>Example 4. Simulating missing values and  imputing them. Dataset: segment from the UCIThis dataset is an image segmentation database similar to a database already present in the repository (Image segmentation database) but in a slightly different form. The instances were drawn randomly from a database of 7 outdoor images. The images were handsegmented to create a classification for every pixel. Each instance is a 3x3 region.<jupyter_code>seg=pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/statlog/segment/segment.dat',header=None, delim_whitespace=True)
#seg=seg=pd.read_table('c://PW-PR/segment.dat',header=None, delim_whitespace=True)
seg.columns=['v1','v2','v3','v4','v5','v6','v7','v8','v9','v10','v11','v12','v13','v14','v15','v16','v17','v18','v19','class']
seg.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2310 entries, 0 to 2309
Data columns (total 20 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   v1      2310 non-null   float64
 1   v2      2310 non-null   float64
 2   v3      2310 non-null   int64  
 3   v4      2310 non-null   float64
 4   v5      2310 non-null   float64
 5   v6      2310 non-null   float64
 6   v7      2310 non-null   float64
 7   v8      2310 non-null   float64
 8   v9      2310 non-null   float64
 9   v10     2310 non-null   float64
 10  v11     2310 non-null   float64
 11  v12     2310 non-null   float64
 12  v13     2310 non-null   float64
 13  v14     2310 non-null   float64
 14  v15     2310 non-null   float64
 15  v16     2310 non-null   float64
 16  v17     2310 non-null   float64
 17  v18     2310 non-null   float64
 18  v19     2310 non-null   float64
 19  class   2310 non-null   int64  
dtypes: float64(18), int64(2)
memory usage: 361.1 KB<jupyter_text>Todos las variables de segment son cuantitativas.<jupyter_code>#simulando un 5% de celdas missing
import random
seg5=seg.copy()
seg5= seg5.stack().sample(frac=0.95).unstack().reindex(index=seg5.index, columns=seg5.columns)

seg5.head()
seg5.info()
#imputando por la media
seg5.mean=seg5.apply(lambda x: x.fillna(x.mean()),axis=0)
#Imputando por la mediana
seg5.median=seg5.apply(lambda x: x.fillna(x.median()),axis=0)
b1=np.array(seg5)
imputer = KNNImputer(n_neighbors=5)
data_nimp=imputer.fit_transform(b1)
completoknn5=pd.DataFrame(data_nimp)
completoknn5.info()
from sklearn.metrics import mean_squared_error
#Calculando el cuadrado medio del error para imputacion por la media
completo5mean=pd.DataFrame(seg5.mean)
mean_squared_error(seg,completo5mean)
#Calculando el cuadrado medio del error para imputacion por la mediana
completo5median=pd.DataFrame(seg5.median)
mean_squared_error(seg,completo5median)
#Calculando el cuadrado medio del error para imputacion knn
mean_squared_error(seg,completoknn5)<jupyter_output><empty_output><jupyter_text>Notice that the best imputation method was k-nn<jupyter_code>#simulando un 20% de celdas missing
seg20=seg.copy()
seg20= seg20.stack().sample(frac=0.80).unstack().reindex(index=seg20.index, columns=seg20.columns)
seg20.info()
#hallando el numero de filas que tienen missings
sum(seg20.apply(lambda x: sum(x.isnull().values), axis = 1)>0)
#imputando por la media al 20%
seg20.mean=seg20.apply(lambda x: x.fillna(x.mean()),axis=0)
completo20mean=pd.DataFrame(seg20.mean)
mean_squared_error(seg,completo20mean)
#imputando por la mediana al 20%
seg20.median=seg20.apply(lambda x: x.fillna(x.median()),axis=0)
completo20median=pd.DataFrame(seg20.median)
mean_squared_error(seg,completo20median)
b2=np.array(seg20)
imputer = KNNImputer(n_neighbors=5)
data_imp=imputer.fit_transform(b2)
completoknn20=pd.DataFrame(data_imp)
mean_squared_error(seg,completoknn20)<jupyter_output><empty_output>