<jupyter_start><jupyter_text>https://www.analyticsvidhya.com/blog/2018/08/dimensionality-reduction-techniques-python/<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# get data 
from google.colab import files
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>3.1 Missing Value Ratio<jupyter_code>train = pd.read_csv("Train_UWu5bXk.csv")
# nuls ?
train.isnull().sum()/len(train)*100 # bo %
nuls = train.isnull().sum()/len(train)*100
vars = train.columns
var = []
for i in range(0,12):
    if (nuls[i]<=20):
        var.append(vars[i])
var<jupyter_output><empty_output><jupyter_text>So the variables to be used are stored in “variable”, which contains only those features where the missing values are less than 20%.  3.2 Low Variance Filter Let’s first impute the missing values in the Item_Weight column using the median value of the known Item_Weight observations<jupyter_code>train['Item_Weight'].fillna(train['Item_Weight'].median(), inplace=True)
train['Outlet_Size'].fillna(train['Outlet_Size'].mode()[0], inplace=True)
train.isnull().sum()/len(train)*100<jupyter_output><empty_output><jupyter_text>calculate all varible variance :<jupyter_code>train.var()<jupyter_output><empty_output><jupyter_text>As the above output shows, the variance of Item_Visibility is very less as compared to the other variables.<jupyter_code># get numeric columns
num = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
numeric = train.select_dtypes(include=num)
var = numeric.var()
numeric = numeric.columns
vars = []
for i in range(0, len(var)):
    if (var[i] >= 10):
        vars.append(numeric[i])
vars<jupyter_output><empty_output><jupyter_text>3.3 High Correlation filter High correlation between two variables means they have similar trends and are likely to carry similar information. This can bring down the performance of some models drastically (linear and logistic regression models, for instance). We can calculate the correlation between independent numerical variables that are numerical in nature. If the correlation coefficient crosses a certain threshold value, we can drop one of the variables (dropping a variable is highly subjective and should always be done keeping the domain in mind).<jupyter_code>train.corr()<jupyter_output><empty_output><jupyter_text>**As a general guideline, we should keep those variables which show a decent or high correlation with the target variable.** Wonderful, we don’t have any variables with a high correlation in our dataset. Generally, if the correlation between a pair of variables is greater than 0.5-0.6, we should seriously consider dropping one of those variables.  3.4 Random Forest Random Forest is one of the most widely used algorithms for feature selection. It comes packaged with in-built feature importance so you don’t need to program that separately. This helps us select a smaller subset of features.We need to convert the data into numeric form by applying one hot encoding, as Random Forest (Scikit-Learn Implementation) takes only numeric inputs. Let’s also drop the ID variables (Item_Identifier and Outlet_Identifier) as these are just unique numbers and hold no significant importance for us currently.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
df=train.drop('Item_Outlet_Sales', 1)
df = df.drop(['Item_Identifier', 'Outlet_Identifier'], axis=1)
model = RandomForestRegressor(random_state=1, max_depth=10)
# one hot encoding
df = pd.get_dummies(df)
model.fit(df, train.Item_Outlet_Sales)
features = df.columns
importances = model.feature_importances_
indices = np.argsort(importances)[-9:]  # top 10 features
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on the above graph, we can hand pick the top-most features to reduce the dimensionality in our dataset. Alernatively, we can use the SelectFromModel of sklearn to do so. It selects the features based on the importance of their weights.<jupyter_code>from sklearn.feature_selection import SelectFromModel
feature = SelectFromModel(model)
fit = feature.fit_transform(df, train.Item_Outlet_Sales)<jupyter_output><empty_output><jupyter_text>3.5 Backward Feature Elimination     -We first take all the n variables present in our dataset and train the model using them    -We then calculate the performance of the model    -Now, we compute the performance of the model after eliminating each variable (n times), i.e., we drop one variable every time and train the model on the remaining n-1 variables    -We identify the variable whose removal has produced the smallest (or no) change in the performance of the model, and then drop that variable    -Repeat this process until no variable can be dropped<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.feature_selection import RFE
from sklearn import datasets
lreg = LinearRegression()
rfe = RFE(lreg, 10)
rfe = rfe.fit_transform(df, train.Item_Outlet_Sales)
rfe<jupyter_output><empty_output><jupyter_text>3.6 Forward Feature Selection     -We start with a single feature. Essentially, we train the model n number of times using each feature separately    -The variable giving the best performance is selected as the starting variable    -Then we repeat this process and add one variable at a time. The variable that produces the highest increase in performance is retained    -We repeat this process until no significant improvement is seen in the model’s performance<jupyter_code>from sklearn.feature_selection import f_regression
ffs = f_regression(df,train.Item_Outlet_Sales )
ffs<jupyter_output><empty_output><jupyter_text>This returns an array containing the F-values of the variables and the p-values corresponding to each F value. For our purpose, we will select the variables having F-value greater than 1<jupyter_code>variable = [ ]
for i in range(0,len(df.columns)-1):
    if ffs[0][i] >=10:
       variable.append(df.columns[i])<jupyter_output><empty_output><jupyter_text>**NOTE : Both Backward Feature Elimination and Forward Feature Selection are time consuming and computationally expensive.They are practically only used on datasets that have a small number of input variables.**  3.7 Factor Analysis Suppose we have two variables: Income and Education. These variables will potentially have a high correlation as people with a higher education level tend to have significantly higher income, and vice versa.In the Factor Analysis technique, variables are grouped by their correlations, i.e., all variables in a particular group will have a high correlation among themselves, but a low correlation with variables of other group(s). Here, each group is known as a factor. These factors are small in number as compared to the original dimensions of the data. However, these factors are difficult to observe.<jupyter_code>import pandas as pd
import numpy as np
from glob import glob
import cv2
import tensorflow as tf
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train.shape
image = []
for i in range(0,60000):
    img = x_train[i].flatten()
    image.append(img)
x_train = np.array(image)
x_train.shape
y_train.shape
feat_cols = [ 'pixel'+str(i) for i in range(x_train.shape[1]) ]
df = pd.DataFrame(x_train, columns=feat_cols)
df['label'] = y_train
from sklearn.decomposition import FactorAnalysis
FA = FactorAnalysis(n_components = 3).fit_transform(df[feat_cols].values)<jupyter_output><empty_output><jupyter_text>Here, n_components will decide the number of factors in the transformed data. After transforming the data, it’s time to visualize<jupyter_code>import matplotlib.pyplot as plt
plt.figure(figsize=(12,8))
plt.title('Factor Analysis Components')
plt.scatter(FA[:,0], FA[:,1])
plt.scatter(FA[:,1], FA[:,2])
plt.scatter(FA[:,2],FA[:,0])<jupyter_output><empty_output><jupyter_text>3.8 Principal Component Analysis (PCA) PCA is a technique which helps us in extracting a new set of variables from an existing large set of variables. These newly extracted variables are called Principal Components.    -A principal component is a linear combination of the original variables    -Principal components are extracted in such a way that the first principal component explains maximum variance in the dataset    -Second principal component tries to explain the remaining variance in the dataset and is uncorrelated to the first principal component    -Third principal component tries to explain the variance which is not explained by the first two principal components and so on<jupyter_code>rndperm = np.random.permutation(df.shape[0])
plt.gray()
fig = plt.figure(figsize=(20,10))
for i in range(0,15):
    ax = fig.add_subplot(3,5,i+1)
    ax.matshow(df.loc[rndperm[i],feat_cols].values.reshape((28,28)).astype(float))
from sklearn.decomposition import PCA
pca = PCA(n_components=50)
pca_result = pca.fit_transform(df[feat_cols].values)<jupyter_output><empty_output><jupyter_text>In this case, n_components will decide the number of principal components in the transformed data. Let’s visualize how much variance has been explained using these 10 components.<jupyter_code>plt.plot(range(50), pca.explained_variance_ratio_)
plt.plot(range(50), np.cumsum(pca.explained_variance_ratio_))
plt.title("Component-wise and Cumulative Explained Variance")<jupyter_output><empty_output><jupyter_text>In the above graph, the blue line represents component-wise explained variance while the orange line represents the cumulative explained variance. We are able to explain around 80% variance in the dataset using just 50 components. Each additional dimension we add to the PCA technique captures less and less of the variance in the model. The first component is the most important one, followed by the second, then the third, and so on. **SDV** We can also use Singular Value Decomposition (SVD) to decompose our original dataset into its constituents, resulting in dimensionality reduction.SVD decomposes the original variables into three constituent matrices. It is essentially used to remove redundant features from the dataset. It uses the concept of Eigenvalues and Eigenvectors to determine those three matrices.<jupyter_code>from sklearn.decomposition import TruncatedSVD 
svd = TruncatedSVD(n_components=3,
                   random_state=42).fit_transform(df[feat_cols].values)
plt.figure(figsize=(12,8))
plt.title('SVD Components')
plt.scatter(svd[:,0], svd[:,1])
plt.scatter(svd[:,1], svd[:,2])
plt.scatter(svd[:,2],svd[:,0])<jupyter_output><empty_output><jupyter_text>The above scatter plot shows us the decomposed components very neatly. As described earlier, there is not much correlation between these components.  3.9 Independent Component Analysis Independent Component Analysis (ICA) is based on information-theory and is also one of the most widely used dimensionality reduction techniques. The major difference between PCA and ICA is that PCA looks for uncorrelated factors while ICA looks for independent factors. If two variables are uncorrelated, it means there is no linear relation between them. If they are independent, it means they are not dependent on other variables. For example, the age of a person is independent of what that person eats, or how much television he/she watches. This algorithm assumes that the given variables are linear mixtures of some unknown latent variables. It also assumes that these latent variables are mutually independent, i.e., they are not dependent on other variables and hence they are called the independent components of the observed data.<jupyter_code>from sklearn.decomposition import FastICA 
ICA = FastICA(n_components=3, random_state=12) 
X=ICA.fit_transform(df[feat_cols].values)
plt.figure(figsize=(12,8))
plt.title('ICA Components')
plt.scatter(X[:,0], X[:,1])
plt.scatter(X[:,1], X[:,2])
plt.scatter(X[:,2], X[:,0])<jupyter_output><empty_output><jupyter_text>The data has been separated into different independent components which can be seen very clearly in the above image. X-axis and Y-axis represent the value of decomposed independent components.  3.10 Methods Based on Projections In projection techniques, multi-dimensional data is represented by projecting its points onto a lower-dimensional space. We will perform non-linear dimensionality reduction through Isometric Mapping. For visualization, we will only take a subset of our dataset as running it on the entire dataset will require a lot of time.<jupyter_code>from sklearn import manifold 
trans_data = manifold.Isomap(n_neighbors=5,
                             n_components=3,
                             n_jobs=-1).fit_transform(df[feat_cols][:6000].values)<jupyter_output><empty_output><jupyter_text>n_neighbors decides the number of neighbors for each point    n_components decides the number of coordinates for manifold    n_jobs = -1 will use all the CPU cores available<jupyter_code>plt.figure(figsize=(12,8))
plt.title('Decomposition using ISOMAP')
plt.scatter(trans_data[:,0], trans_data[:,1])
plt.scatter(trans_data[:,1], trans_data[:,2])
plt.scatter(trans_data[:,2], trans_data[:,0])<jupyter_output><empty_output><jupyter_text>3.11 t- Distributed Stochastic Neighbor Embedding (t-SNE)<jupyter_code>from sklearn.manifold import TSNE 
tsne = TSNE(n_components=3,
            n_iter=300).fit_transform(df[feat_cols][:6000].values)
plt.figure(figsize=(12,8))
plt.title('t-SNE components')
plt.scatter(tsne[:,0], tsne[:,1])
plt.scatter(tsne[:,1], tsne[:,2])
plt.scatter(tsne[:,2], tsne[:,0])<jupyter_output><empty_output><jupyter_text>3.12 UMAP<jupyter_code>import umap
umap_data = umap.UMAP(n_neighbors=5,
                      min_dist=0.3,
                      n_components=3).fit_transform(df[feat_cols][:6000].values)
plt.figure(figsize=(12,8))
plt.title('Decomposition using UMAP')
plt.scatter(umap_data[:,0], umap_data[:,1])
plt.scatter(umap_data[:,1], umap_data[:,2])
plt.scatter(umap_data[:,2], umap_data[:,0])<jupyter_output><empty_output><jupyter_text>As mentioned in UMAP’s GitHub repository, it often performs better at preserving aspects of the global structure of the data than t-SNE. This means that it can often provide a better “big picture” view of the data as well as preserving local neighbor relations.<jupyter_code><jupyter_output><empty_output>