<jupyter_start><jupyter_text>KNN ON AMAZON FINE FOOD REVIEWS DATASET Data Source **[https://www.kaggle.com/snap/amazon-fine-food-reviews](https://www.kaggle.com/snap/amazon-fine-food-reviews)** The Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.It consist of data collected from past many years. This dataset consist of approx 550k reviews.  ![Knn Model on Amazon Review Dataset ](http://houseofbots.com/images/news/3197/cover.png)   SNIPPET1. Converted the reviews using NLP techniques i.e BOW, tf-IDF, Word2Vec and tf-IDF Word2Vec.2. Applied Knn on the dataset with both Techniques i.e KD-Tree and Bruteforce. 3. Calculated Train Error, CV Error and Test Error to determine the performance and to ensure best fit.4. Compared performance of each model using accuracy.5. Made confusion matrix between predicted and tested data.6. Conclusion based on the obtained results.  DATA INFORMATION* Number of reviews: 568,454* Number of users: 256,059* Number of products: 74,258* Timespan: Oct 1999 - Oct 2012* Number of Attributes/Columns in data: 10   ATTRIBUTE INFORMATION1.  Id2.  ProductId - unique identifier for the product3.  UserId - unqiue identifier for the user4.  ProfileName5.  HelpfulnessNumerator - number of users who found the review helpful6.  HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not7.  Score - rating between 1 and 58.  Time - timestamp for the review9.  Summary - brief summary of the review10. Text - text of the review  OBJECTIVEPredict the polarity of the review using Knn and Compare all models to find the best accuracy and ensure that the model is neither overfitting nor underfitting.     LOADING<jupyter_code>import time
import sqlite3 
import pandas as pd
conn=sqlite3.connect('./final.sqlite') # making a connection with sqlite
data=pd.read_sql_query("""SELECT * FROM Reviews""",conn)
data.head(3)<jupyter_output><empty_output><jupyter_text>MAPPING<jupyter_code># function to map the polarity as 0 or 1
def sign(x):
    if x=='positive':
        return 1
    else:
        return 0

data['Score']=data['Score'].map(sign)
# Dimension
print(data.shape)
# Frequency of data.
data['Score'].value_counts()<jupyter_output><empty_output><jupyter_text>SAMPLING<jupyter_code># Taking a Random Sample of 20k points.
Data=data.sample(20000)
Data['Score'].value_counts()<jupyter_output><empty_output><jupyter_text>SORTING<jupyter_code># Sorting the data according to Time.
Data.sort_values('Time',inplace=True)<jupyter_output><empty_output><jupyter_text>IMPORTING<jupyter_code>import re
import gensim
import pickle
import numpy as np
import seaborn as sns
from scipy import sparse
from prettytable import PrettyTable
from sklearn.metrics import accuracy_score
import statistics as s
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import preprocessing
from sklearn.model_selection import TimeSeriesSplit
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
import matplotlib.pyplot as plt
from gensim.models import Word2Vec
from gensim.models import KeyedVectors
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>FUNCTIONS   1. SPLIT FUNCTION<jupyter_code>'''
This function is used to split that data into train and test.
It uses the function to split it into 70-30 %.
It does not shuffle so the data is distributed sequentially.
'''

def Split(d1,d2):
    a,b,c,d= train_test_split(d1,d2,test_size=0.3,shuffle=False) # Splitting it in 70-30 without shuffling.
    return a,b,c,d<jupyter_output><empty_output><jupyter_text>2. Knn  FUNCTION<jupyter_code>'''
This function takes training data and lgorithm as input and gives execution time, accuracy and the optimal value of k
on that data.
It uses TimeSeriessplit CV.
It also calculates accuracy in training data and CV data.
'''

def KNN(x_train,y_train,algo):
    start = time.time()
    cv_acc=[]
    train_acc=[]
    tscv = TimeSeriesSplit(n_splits=5)         # Using 5 cross valiadtions.
    for n in range(1,30,2):
        l1=[]
        l2=[]
        for train,cv in tscv.split(x_train):
            knn = KNeighborsClassifier(n_neighbors=n,algorithm=algo,n_jobs=-1)
            knn.fit(x_train[train],y_train[train])
            pred_cv = knn.predict(x_train[cv])
            pred_train = knn.predict(x_train[train])
            acc_cv = accuracy_score(y_train[cv],pred_cv, normalize=True) * float(100)
            acc_train = accuracy_score(y_train[train],pred_train, normalize=True) * float(100)
            l1.append(acc_cv)
            l2.append(acc_train)
        cv_acc.append(s.mean(l1))
        train_acc.append(s.mean(l2))
    end = time.time()
    t=end-start
    neigh=list(np.arange(1,30,2))
    opt=neigh[cv_acc.index(max(cv_acc))]
    return cv_acc,train_acc,t,opt<jupyter_output><empty_output><jupyter_text>3. K vs ACCURACY PLOT<jupyter_code>'''
This function takes Accuarcy and plots the graph for accuracy vs k.
'''

def Accplot(acu,nlp,algo):
    sns.set_style("darkgrid")
    plt.plot(np.arange(1,30,2),acu,'b--')
    plt.xlabel("K Nearest Neighbours",fontsize=15, color='black')
    plt.ylabel("Accuracy",fontsize=15, color='black')
    plt.title("Accuracy -" + nlp + "- KNN - " + algo,fontsize=15, color='black')
    plt.show()
    return plt.show()<jupyter_output><empty_output><jupyter_text>4. K vs ERROR PLOT<jupyter_code>'''
This function takes the CV accuracy and Training accuracy.
Output is train error and CV error.
It also plots the graph between K vs Errors.
'''

def Trainplot(cv_acc,train_acc,algo):
    a = [100 - x for x in cv_acc]
    b = [100 - x for x in train_acc]
    k=np.arange(1,30,2)
    plt.plot(k, a, '-b', label='CV Error')
    plt.plot(k, b, '-r', label='Train Error')
    plt.legend(loc='lower right')
    plt.xlabel("K Nearest Neighbours",fontsize=15, color='black')
    plt.ylabel("Train Error & Cv Error",fontsize=15, color='black')
    plt.title("Train Error vs Cv Error on " + algo,fontsize=15, color='black')
    #plt.plot(k, a, 'r--', k, b, 'b--')
    plt.show()
    print("The Train Error is -: ",round(s.mean(b),3),"%\n")
    print("The CV Error is -: ",round(s.mean(a),3),"%\n")<jupyter_output><empty_output><jupyter_text>5. PREDICT FUNCTION<jupyter_code>'''
It runs the desired algorithm on the optimal value of k we get from training part.
It also returns accuracy and test error.
'''

def Test(x_train,y_train,x_test,y_test,opt,algo):
    knn = KNeighborsClassifier(n_neighbors=opt,algorithm=algo)
    knn.fit(x_train,y_train)
    pred = knn.predict(x_test)
    acc = accuracy_score(y_test,pred, normalize=True) * float(100)
    test_err=100-acc
    print("The Accuracy is -: ",round(acc,3),"%\n")
    print("The Test Error is -: ",round(test_err,3),"%\n")
    return pred<jupyter_output><empty_output><jupyter_text>6. CONFUSION MATRIX<jupyter_code>'''
It gives confusion matrix between actual and predicted values.
'''

def conf(test,pre):
    cf = confusion_matrix(test,pre)
    df =pd.DataFrame(cf,index=[0,1],columns=[0,1])
    sns.set(font_scale=1.5)
    sns.heatmap(df,annot=True,annot_kws={"size" :20},fmt='g')
    return plt.show()<jupyter_output><empty_output><jupyter_text>Knn Model on BOW ( Bag Of Words )![](https://cdn-images-1.medium.com/max/1600/0*JpqZhCNsQ_OGaRkB.jpg)   SPLITTING INTO TRAIN AND TEST<jupyter_code>x_train, x_test, y_train, y_test = Split(Data['CleanedText'].values,Data['Score'].values)
print("-----------------------TRAIN DATA------------------------------------")
print(x_train.shape)
print(y_train.shape)
print("---------------------------------------------------------------------")
print("\n-----------------------TEST DATA-------------------------------------")
print(x_test.shape)
print(y_test.shape)<jupyter_output>-----------------------TRAIN DATA------------------------------------
(14000,)
(14000,)
---------------------------------------------------------------------

-----------------------TEST DATA-------------------------------------
(6000,)
(6000,)<jupyter_text>CONVERTING REVIEWS INTO VECTORS USING BOW<jupyter_code>'''
Here we are fitting it on training data and then transforming the test data with that vocabulary so that the test data 
is not seen by the training phase and generalization is possible.
'''
count = CountVectorizer() 
x_train = count.fit_transform(x_train)
x_test = count.transform(x_test)
print("Train Dataset Shape -: ",x_train.shape)
print("Test Dataset Shape -: ",x_test.shape)<jupyter_output>Train Dataset Shape -:  (14000, 15029)
Test Dataset Shape -:  (6000, 15029)<jupyter_text>NORMALIZING THE DATA<jupyter_code>x_train = preprocessing.normalize(x_train)
x_test = preprocessing.normalize(x_test)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH BRUTEFORCE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(x_train,y_train,'brute')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")  # Accuracy on CV dataset.<jupyter_output>Time taken to complete -:  284.2675817012787 sec

Optimal_k -:  11 

Accuracy -:  85.521 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'BOW','Bruteforce')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'BOW')<jupyter_output><empty_output><jupyter_text>From the given plot we can analyse that the optimal_k is 11 and it tends to go towards overfitting but it is not exactly overfitting as i have not plotted it till 50 or 100k so we can't be sure about it but i think that  a range between 13-17 will be a good one to declare optimal_k.   PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(x_train,y_train,x_test,y_test,opt,'brute')<jupyter_output>The Accuracy is -:  83.8 %

The Test Error is -:  16.2 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>CONVERTING FROM SPARSE TO DENSE MATRIX<jupyter_code>d_train = x_train.todense(order=None, out=None)
d_test = x_test.todense(order=None, out=None)
print(d_train.shape)
print(d_test.shape)<jupyter_output>(14000, 15029)
(6000, 15029)<jupyter_text>CALLING Knn FUNCTION WITH KD-Tree ALGORITHM<jupyter_code>cv,train,t,opt=KNN(d_train,y_train,'kd_tree')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  15291.153882265091 sec

Optimal_k -:  11 

Accuracy -:  85.512 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'BOW','KD-Tree')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'BOW')<jupyter_output><empty_output><jupyter_text>This case is also similar to previous case as the difference in Train Error and Test Error is same.   PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(d_train,y_train,d_test,y_test,opt,'kd_tree')<jupyter_output>The Accuracy is -:  83.75 %

The Test Error is -:  16.25 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>Knn Model on tf-IDF![](https://lphinternetservices.com/images/TF-IDF-define.jpg)   SPLITTING INTO TRAIN AND TEST<jupyter_code>x_train, x_test, y_train, y_test = Split(Data['CleanedText'].values,Data['Score'].values)<jupyter_output><empty_output><jupyter_text>CONVERTING REVIEWS INTO VECTORS USING tf-IDF<jupyter_code>tf_idf_vect = TfidfVectorizer(ngram_range=(1,1)) 
x_train = tf_idf_vect.fit_transform(x_train)
x_test = tf_idf_vect.transform(x_test)
print("Train Dataset Shape -: ",x_train.shape)
print("Test Dataset Shape -: ",x_test.shape)<jupyter_output>Train Dataset Shape -:  (14000, 15029)
Test Dataset Shape -:  (6000, 15029)<jupyter_text>NORMALIZING THE DATA<jupyter_code>x_train = preprocessing.normalize(x_train)
x_test = preprocessing.normalize(x_test)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH BRUTEFORCE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(x_train,y_train,'brute')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  285.0983760356903 sec

Optimal_k -:  9 

Accuracy -:  85.632 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'BOW','Bruteforce')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'BOW')<jupyter_output><empty_output><jupyter_text>In this case the difference between the Train and Test Error is less than BOW i.e 4.189 % but by amalysing the plot we can infer that optimal_k tends towards overfitting.   PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(x_train,y_train,x_test,y_test,opt,'brute')<jupyter_output>The Accuracy is -:  83.6 %

The Test Error is -:  16.4 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>CONVERTING FROM SPARSE TO DENSE MATRIX<jupyter_code>d_train = x_train.todense(order=None, out=None)
d_test = x_test.todense(order=None, out=None)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH KD-TREE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(d_train,y_train,'kd_tree')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  16632.4118578434 sec

Optimal_k -:  9 

Accuracy -:  85.461 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'tf-IDF','KD-Tree')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'tf-IDF')<jupyter_output><empty_output><jupyter_text>It is similar to previous case in terms of difference between test and train error and about optimal_k tending to overfit.   PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(d_train,y_train,d_test,y_test,opt,'kd_tree')<jupyter_output>The Accuracy is -:  84.15 %

The Test Error is -:  15.85 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>Knn on Avg Word2Vec![](https://adriancolyer.files.wordpress.com/2016/04/word2vec-gender-relation.png?w=596)  SPLIT DATA INTO TRAIN AND TEST<jupyter_code>'''
Here we are taking two lists and putting the data separate as Test in l4 and Train in l5.
'''

l4=[]
for i in range(14000):
    l1=Data['CleanedText'].values[i]
    l2=str(l1)
    l4.append(l2)

l5=[]
for i in range(14000,20000,1):
    l1=Data['CleanedText'].values[i]
    l2=str(l1)
    l5.append(l2)<jupyter_output><empty_output><jupyter_text>MAKING LIST OF WORDS<jupyter_code>def cleanpunc(sentence): #function to clean the word of any punctuation or special characters
    cleaned = re.sub(r'[?|!|\'|"|#]',r'',sentence)
    cleaned = re.sub(r'[.|,|)|(|\|/]',r' ',cleaned)
    return  cleaned

#------------------------------------------ TRAIN DATASET ---------------------------------------------------------------------
# making a list of words for each review.
i=0
list_of_sent=[] # list to store all the lists.
for sent in l4:
    filtered_sentence=[] # list to store each review.
    for w in sent.split():
        for cleaned_words in cleanpunc(w).split():
            if(cleaned_words.isalpha()):    
                filtered_sentence.append(cleaned_words.lower())
            else:
                continue 
    list_of_sent.append(filtered_sentence)

# ------------------------------------------- TEST DATASET ---------------------------------------------------------------------    
    
# making a list of words for each review.
i=0
list_of_sent1=[] # list to store all the lists.
for sent in l5:
    filtered_sentence=[] # list to store each review.
    for w in sent.split():
        for cleaned_words in cleanpunc(w).split():
            if(cleaned_words.isalpha()):    
                filtered_sentence.append(cleaned_words.lower())
            else:
                continue 
    list_of_sent1.append(filtered_sentence)
print("____________________________________ TRAINING DATA ___________________________________________________")
print(Data['CleanedText'].values[3])

print("\n" + "********************Converted these sentences into a list of words each***********************\n ")

# First Review with breaking into words. 
print(list_of_sent[3])

#-------------------------------------------------------------------------------------------------------------------
print("--------------------------------------------------------------------------------------------------------------")

print("____________________________________ TEST DATA ___________________________________________________")

print(Data['CleanedText'].values[-1])

print("\n" + "********************Converted these sentences into a list of words each***********************\n ")

# First Review with breaking into words. 
print(list_of_sent1[-1])<jupyter_output>____________________________________ TRAINING DATA ___________________________________________________
b'love sauc put everyth'

********************Converted these sentences into a list of words each***********************
 
['blove', 'sauc', 'put', 'everyth']
--------------------------------------------------------------------------------------------------------------
____________________________________ TEST DATA ___________________________________________________
b'like product price point flavor veri strong overpow perhap figur name'

********************Converted these sentences into a list of words each***********************
 
['blike', 'product', 'price', 'point', 'flavor', 'veri', 'strong', 'overpow', 'perhap', 'figur', 'name']<jupyter_text>TRAINING THE MODEL ON TRAIN DATA<jupyter_code>'''
Trained our own model on the training data only with feature size or dimension = 100 with min_count = 2 this means that 
if a word comes atleast 2 times only then consider it otherwise leave it.
'''
w2v_model=gensim.models.Word2Vec(list_of_sent,min_count=2,size=100, workers=4)<jupyter_output><empty_output><jupyter_text>CONVERTING REVIEWS INTO VECTORS USING AVG WORD2VEC<jupyter_code>'''
Converting the reviews into vectors by using the above trained model.
'''
#-------------------------------------- TRAIN DATASET ------------------------------------------------------------------
sent_vectors = []; # the avg-w2v for each sentence/review is stored in this list
for sent in list_of_sent: # for each review/sentence
    sent_vec = np.zeros(100) # as word vectors are of zero length
    cnt_words =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        try:
            vec = w2v_model.wv[word]
            sent_vec += vec
            cnt_words += 1
        except:
            pass
    sent_vec /= cnt_words
    sent_vectors.append(sent_vec)

#------------------------------------- TEST DATASET --------------------------------------------------------------------    
'''
Here we are converting reviews of test data using the vocabulary of training data to make the concept to generalization 
meaningful and fruitful.
'''
sent_vectors1 = []; # the avg-w2v for each sentence/review is stored in this list
for sent in list_of_sent1: # for each review/sentence
    sent_vec = np.zeros(100) # as word vectors are of zero length
    cnt_words =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        try:
            vec = w2v_model.wv[word]
            sent_vec += vec
            cnt_words += 1
        except:
            pass
    sent_vec /= cnt_words
    sent_vectors1.append(sent_vec)<jupyter_output><empty_output><jupyter_text>CHECKING THE Nan VALUE<jupyter_code>'''
Here we are checking the Nan values as these creates a lot of problem and it occurs when we divide any value by 0 this
means a value of high range i.e infinity.
'''
sent_vectors=np.array(sent_vectors)
print(np.isnan(sent_vectors).any())
sent_vectors1=np.array(sent_vectors1)
print(np.isnan(sent_vectors1).any())
print(sent_vectors.shape)
print(sent_vectors1.shape)<jupyter_output>(14000, 100)
(6000, 100)<jupyter_text>NORMALIZING THE DATA<jupyter_code>sent_vectors=preprocessing.normalize(sent_vectors)
sent_vectors1=preprocessing.normalize(sent_vectors1)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH BRUTEFORCE ALGORITHM<jupyter_code>x_train, x_test, y_train, y_test = Split(Data['CleanedText'].values,Data['Score'].values)
cv,train,t,opt=KNN(sent_vectors, y_train, 'brute')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  126.74465680122375 sec

Optimal_k -:  13 

Accuracy -:  85.315 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'Avg Word2Vec','Bruteforce')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'Avg Word2Vec')<jupyter_output><empty_output><jupyter_text>By analyzing this we can say that neither it is overfitting nor underfitting and the optimal_k is between the 2 i.e 13 but we can't be so sure as we have not plotted it on whole dataset and moreover we had not taken k values till 50 or 100 maybe then the picture will become clear.   PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(sent_vectors, y_train, sent_vectors1, y_test, opt, 'brute')<jupyter_output>The Accuracy is -:  84.283 %

The Test Error is -:  15.717 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>CONVERTING FROM SPARSE TO DENSE MATRIX<jupyter_code>b1=sparse.csr_matrix(sent_vectors)
b2=sparse.csr_matrix(sent_vectors1)
d_train = b1.todense(order=None, out=None)
d_test = b2.todense(order=None, out=None)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH KD-TREE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(d_train, y_train, 'kd_tree')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  992.0894522666931 sec

Optimal_k -:  13 

Accuracy -:  85.315 %<jupyter_text>The KD-TREE is giving the same results as bruteforce but is taking long to compute as the data given to the algorithm is dense.   K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'Avg Word2Vec','KD-Tree')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'Avg Word2Vec')<jupyter_output><empty_output><jupyter_text>PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(d_train, y_train, d_test, y_test, opt, 'kd_tree')<jupyter_output>The Accuracy is -:  84.283 %

The Test Error is -:  15.717 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>Knn Model On tf-IDF Word2Vec  NOTE : I forgot to save it so have to do it again.<jupyter_code>x_train, x_test, y_train, y_test = Split(Data['CleanedText'].values,Data['Score'].values)
tf_idf_vect = TfidfVectorizer(ngram_range=(1,1)) 
x_train = tf_idf_vect.fit_transform(x_train)
x_test = tf_idf_vect.transform(x_test)<jupyter_output><empty_output><jupyter_text>TRAINING OWN MODEL ON TRAIN DATASET<jupyter_code>model=gensim.models.Word2Vec(list_of_sent,min_count=2,size=100, workers=4)<jupyter_output><empty_output><jupyter_text>CONVERTING REVIEWS INTO VECTORS USING tf-IDF WORD2VEC<jupyter_code>#--------------------------------------- TRAIN DATASET ---------------------------------------------------------------- 

tfidf_feat = tf_idf_vect.get_feature_names()

tfidf_sent_vectors_train = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in list_of_sent: # for each review/sentence
    sent_vec = np.zeros(100) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        try:
            vec = model.wv[word]
            # obtain the tf_idfidf of a word in a sentence/review
            tf_idf = x_train[row, tfidf_feat.index(word)]
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
        except:
            pass
    sent_vec /= weight_sum
    tfidf_sent_vectors_train.append(sent_vec)
    row += 1

    
#----------------------------------------- TEST DATASET ---------------------------------------------------------------

tfidf_sent_vectors_test = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in list_of_sent1: # for each review/sentence
    sent_vec = np.zeros(100) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        try:
            vec = model.wv[word]
            # obtain the tf_idfidf of a word in a sentence/review
            tf_idf = x_test[row, tfidf_feat.index(word)]
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
        except:
            pass
    sent_vec /= weight_sum
    tfidf_sent_vectors_test.append(sent_vec)
    row += 1<jupyter_output><empty_output><jupyter_text>CHECKING THE Nan VALUE<jupyter_code>train = np.array(tfidf_sent_vectors_train)
test = np.array(tfidf_sent_vectors_test)
print(np.isnan(train).any())
print(np.isnan(test).any())<jupyter_output>False
False<jupyter_text>NORMALIZING THE DATA<jupyter_code>train_ = preprocessing.normalize(train)
test_ = preprocessing.normalize(test)<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH BRUTEFORCE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(train_, y_train, 'brute')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  113.40949702262878 sec

Optimal_k -:  11 

Accuracy -:  85.152 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'tf-IDF Word2Vec','BruteForce')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'tf-IDF Word2Vec')<jupyter_output><empty_output><jupyter_text>PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(train_, y_train, test_, y_test, opt, 'brute')<jupyter_output>The Accuracy is -:  83.0 %

The Test Error is -:  17.0 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>CONVERTING FROM SPARSE TO DENSE MATRIX<jupyter_code>b1=sparse.csr_matrix(train_)
b2=sparse.csr_matrix(test_)
training = b1.todense()
testing = b2.todense()<jupyter_output><empty_output><jupyter_text>CALLING Knn FUNCTION WITH KD-TREE ALGORITHM<jupyter_code>cv,train,t,opt=KNN(training, y_train, 'kd_tree')
print("Time taken to complete -: ",t,"sec\n")
print("Optimal_k -: ",opt,"\n")
print("Accuracy -: ",round(max(cv),3),"%")<jupyter_output>Time taken to complete -:  978.9853284358978 sec

Optimal_k -:  11 

Accuracy -:  85.152 %<jupyter_text>K  VS ACCURACY PLOT<jupyter_code>Accplot(cv,'tf-IDF Word2Vec','KD-Tree')<jupyter_output><empty_output><jupyter_text>K VS TRAIN & CV ERROR PLOT<jupyter_code>Trainplot(cv,train,'tf-IDF Word2Vec')<jupyter_output><empty_output><jupyter_text>PREDICTING ON OPTIMAL K<jupyter_code>pred = Test(train_, y_train, test_, y_test, opt, 'kd_tree')<jupyter_output>The Accuracy is -:  83.0 %

The Test Error is -:  17.0 %<jupyter_text>CONFUSION MATRIX BETWEEN ACTUAL AND PREDICTED CLASS LABELS<jupyter_code>conf(y_test,pred)<jupyter_output><empty_output><jupyter_text>CONCLUSION<jupyter_code>x = PrettyTable()

x.field_names = ["NLP Technique", "Algorithm", "Accuracy(%)", "Hyperparameter", "Train Error(%)", "Test Error(%)", "Time(in sec)"]

x.add_row(["BOW", "BruteForce", 83.80, 11, 10.675, 16.20, 284])
x.add_row(["BOW", "KD-Tree", 83.75, 11, 10.683, 16.25, 15291])
x.add_row(["tf-IDF", "BruteForce", 83.60, 9, 10.901, 16.40, 285])
x.add_row(["tf-IDF", "KD-Tree", 84.15, 9, 11.059, 15.85, 16632])
x.add_row(["Avg Word2Vec", "BruteForce", 84.283, 13, 10.615, 15.717, 126])
x.add_row(["Avg Word2Vec", "KD-TREE", 84.283, 13, 10.615, 15.717, 992])
x.add_row(["tf-IDF Word2Vec", "BruteForce", 83.0, 11, 11.031, 17.0, 113])
x.add_row(["tf-IDF Word2Vec", "KD-TREE", 83.0, 11, 11.019, 17.0, 978])

print(x)<jupyter_output>+-----------------+------------+-------------+----------------+----------------+---------------+--------------+
|  NLP Technique  | Algorithm  | Accuracy(%) | Hyperparameter | Train Error(%) | Test Error(%) | Time(in sec) |
+-----------------+------------+-------------+----------------+----------------+---------------+--------------+
|       BOW       | BruteForce |     83.8    |       11       |     10.675     |      16.2     |     284      |
|       BOW       |  KD-Tree   |    83.75    |       11       |     10.683     |     16.25     |    15291     |
|      tf-IDF     | BruteForce |     83.6    |       9        |     10.901     |      16.4     |     285      |
|      tf-IDF     |  KD-Tree   |    84.15    |       9        |     11.059     |     15.85     |    16632     |
|   Avg Word2Vec  | BruteForce |    84.283   |       13       |     10.615     |     15.717    |     126      |
|   Avg Word2Vec  |  KD-TREE   |    84.283   |       13       |     10.615     |     15.717    |     992[...]
StarcoderdataJupyterStructuredCleanDedup