<jupyter_start><jupyter_text>Marketa Analytics - E-Commerce data  ContentThis is a transactional data set which contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail.The company mainly sells unique all-occasion gifts. Many customers of the company are wholesalers.  This dataframe contains 8 variables that correspond to:InvoiceNo: Invoice number. Nominal, a 6-digit integral number uniquely assigned to each transaction. If this code starts with letter 'c', it indicates a cancellation.StockCode: Product (item) code. Nominal, a 5-digit integral number uniquely assigned to each distinct product.Description: Product (item) name. Nominal.Quantity: The quantities of each product (item) per transaction. Numeric.InvoiceDate: Invice Date and time. Numeric, the day and time when each transaction was generated.UnitPrice: Unit price. Numeric, Product price per unit in sterling.CustomerID: Customer number. Nominal, a 5-digit integral number uniquely assigned to each customer.Country: Country name. Nominal, the name of the country where each customer resides.  Importing and Data Cleaning<jupyter_code>#importing all important package..

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

import warnings
warnings.filterwarnings("ignore")
#load data into pandas dataframe..
df = pd.read_csv('C:/Users/harini/Desktop/NEUCourses/ADM/Assignment 1/data.csv',encoding = "ISO-8859-1")
df.head()
#information of dataset..
df.info()
df.Country.value_counts().head(5)

df = df[df.Country == 'United Kingdom']
df.Quantity.describe()
#Quantity can not be negative so remove negative values..
df = df[df['Quantity']>0]
df.Quantity.describe()
df = df[df['UnitPrice']>0]
df.UnitPrice.describe()
#checking null values in all columns in dataset
null_values = pd.DataFrame(df.isnull().sum(),columns=['count_value'])
ax = sns.barplot(null_values.count_value,null_values.index)<jupyter_output><empty_output><jupyter_text>Removing Missing Value<jupyter_code>df.dropna(subset=['CustomerID'],how='all',inplace=True)
df.isnull().sum()
#last date available in our dataset
df['InvoiceDate'].max()
#use latest date in our data as current date..

import datetime as dt
now = dt.date(2011,12,9)
df['date'] = pd.DatetimeIndex(df.InvoiceDate).date
df['month'] = pd.DatetimeIndex(df.InvoiceDate).month_name(locale = 'English')
df['year'] = pd.DatetimeIndex(df.InvoiceDate).year
df.head()<jupyter_output><empty_output><jupyter_text>RFM Analysis<jupyter_code>#group by customer by last date they purchased...

recency_df = df.groupby(['CustomerID'],as_index=False)['date'].max()
recency_df.columns = ['CustomerID','LastPurchaseDate']
recency_df.head()
#calculate how often he is purchasing with reference to latest date in days..

recency_df['Recency'] = recency_df.LastPurchaseDate.apply(lambda x : (now - x).days)
recency_df.drop(columns=['LastPurchaseDate'],inplace=True)
#check frequency of customer means how many transaction has been done..

frequency_df = df.copy()
frequency_df.drop_duplicates(subset=['CustomerID','InvoiceNo'], keep="first", inplace=True) 
frequency_df = frequency_df.groupby('CustomerID',as_index=False)['InvoiceNo'].count()
frequency_df.columns = ['CustomerID','Frequency']
frequency_df.head()
#calculate how much a customer spend in the each transaction...

df['Total_cost'] = df['UnitPrice'] * df['Quantity']
df.columns
#check summed up spend of a customer with respect to latest date..

monetary_df=df.groupby(['CustomerID'],as_index=False)['Total_cost'].sum()
monetary_df.columns = ['CustomerID','Monetary']
monetary_df.head()
#Combine all together all dataframe in so we have recency, frequency and monetary values together..

#combine first recency and frequency..
rf = recency_df.merge(frequency_df,left_on='CustomerID',right_on='CustomerID')

#combibe rf frame with monetary values..

rfm = rf.merge(monetary_df,left_on='CustomerID',right_on='CustomerID')

#rfm.set_index('CustomerID',inplace=True)
rfm.head()
#checking correctness of output..

df[df.CustomerID == 12346.0]
(now - dt.date(2011,1,18)).days == 325
#bring all the quartile value in a single dataframe

rfm_segmentation = rfm.copy()<jupyter_output><empty_output><jupyter_text>Customer Segmentation using RFM Analysis<jupyter_code>from sklearn.cluster import KMeans
# get right number of cluster for K-means so we neeed to loop from 1 to 20 number of cluster and check score.
#Elbow method is used to represnt that. 
Nc = range(1, 20)
kmeans = [KMeans(n_clusters=i) for i in Nc]
score = [kmeans[i].fit(rfm_segmentation).score(rfm_segmentation) for i in range(len(kmeans))]
plt.plot(Nc,score)
plt.xlabel('Number of Clusters')
plt.ylabel('Score')
plt.title('Elbow Curve')
plt.show()
#fitting data in Kmeans theorem.
kmeans = KMeans(n_clusters=3, random_state=0).fit(rfm_segmentation)
# this creates a new column called cluster which has cluster number for each row respectively.
rfm_segmentation['cluster'] = kmeans.labels_
#check our hypothesis

rfm_segmentation[rfm_segmentation.cluster == 0].head(10)
rfm_segmentation[rfm_segmentation.cluster == 1].head(5)
rfm_segmentation[rfm_segmentation.cluster == 2].head(10)
rfm_segmentation.head()
rfm_segmentation = rfm_segmentation.reset_index(drop = 1)
rfm_segmentation['Status'] = 0
for i in range(0,len(rfm_segmentation)):
    if(rfm_segmentation['cluster'][i] == 0):
        rfm_segmentation['Status'][i] = "Silver"
    elif(rfm_segmentation['cluster'][i] == 1):
        rfm_segmentation['Status'][i]= "Gold"
    else:
        rfm_segmentation['Status'][i] ="Platinum"
rfm_segmentation.head(10)
'''
cluster 0 have high recency rate which is bad. cluster 1 and cluster 2 having low so they are in race of platinum
and gold customer.
'''
sns.boxplot(rfm_segmentation.cluster,rfm_segmentation.Recency)
'''
cluster 0 have low frequency rate which is bad. cluster 1 and cluster 2 having high so they are in 
race of platinum and gold customer.
'''
sns.boxplot(rfm_segmentation.cluster,rfm_segmentation.Frequency)
'''
cluster 0 have low Monetary rate which is bad. cluster 1 have highest Montary (money spend) platinum where as 
cluster 2 have medium level(Gold) and cluster 0 is silver customer.
'''

sns.boxplot(rfm_segmentation.cluster,rfm_segmentation.Monetary)
seg_data = df.merge(rfm_segmentation,left_on='CustomerID',right_on='CustomerID')
seg_data.columns
seg_data.head()<jupyter_output><empty_output><jupyter_text>Creating Discounts<jupyter_code>grouped = seg_data[['Description','UnitPrice']].groupby(['Description'])
seg_data['discount'] = 0
for i in range(0,len(seg_data)):
    if(i%50000 == 0):
        print(i)
    des = seg_data['Description'][i]
    if seg_data['UnitPrice'][i] < grouped.get_group(des).max()[1]:
        seg_data['discount'][i] = 1
platinum = seg_data[seg_data['Status'] == 'Platinum']
gold = seg_data[seg_data['Status'] == 'Gold']
silver = seg_data[seg_data['Status'] == 'Silver']
print(platinum.shape,gold.shape,silver.shape)
seg_data.head(10)<jupyter_output><empty_output><jupyter_text>Product Segmentation<jupyter_code>import pickle
import nltk, warnings
from nltk.tokenize import sent_tokenize, word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from string import digits, punctuation

from scipy.stats import chi2_contingency

from sklearn.preprocessing import LabelEncoder, StandardScaler, Normalizer
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score
from sklearn import preprocessing, model_selection, metrics, feature_selection
from sklearn.model_selection import GridSearchCV, learning_curve
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
from sklearn import neighbors, linear_model, svm, tree, ensemble
from sklearn.decomposition import PCA, TruncatedSVD
from sklearn.manifold import TSNE
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

from wordcloud import WordCloud, STOPWORDS
X = seg_data1["Description"].unique()

stemmer = nltk.stem.porter.PorterStemmer()
stopword = nltk.corpus.stopwords.words('english')

def stem_and_filter(doc):
    tokens = [stemmer.stem(w) for w in analyzer(doc)]
    return [token for token in tokens if token.isalpha()]

analyzer = TfidfVectorizer().build_analyzer()
CV = TfidfVectorizer(lowercase=True, stop_words="english", analyzer=stem_and_filter, min_df=0.00, max_df=0.3)  # we remove words if it appears in more than 30 % of the corpus (not found stopwords like Box, Christmas and so on)
TF_IDF_matrix = CV.fit_transform(X)
print("TF_IDF_matrix :", TF_IDF_matrix.shape, "of", TF_IDF_matrix.dtype)
svd = TruncatedSVD(n_components = 100)
normalizer = Normalizer(copy=False)

TF_IDF_embedded = svd.fit_transform(TF_IDF_matrix)
TF_IDF_embedded = normalizer.fit_transform(TF_IDF_embedded)
print("TF_IDF_embedded :", TF_IDF_embedded.shape, "of", TF_IDF_embedded.dtype)
score_tfidf = []

x = list(range(5, 155, 10))

for n_clusters in x:
    kmeans = KMeans(init='k-means++', n_clusters = n_clusters, n_init=10)
    kmeans.fit(TF_IDF_embedded)
    clusters = kmeans.predict(TF_IDF_embedded)
    silhouette_avg = silhouette_score(TF_IDF_embedded, clusters)

    rep = np.histogram(clusters, bins = n_clusters-1)[0]
    score_tfidf.append(silhouette_avg)
plt.figure(figsize=(20,16))

plt.subplot(2, 1, 1)
plt.plot(x, score_tfidf, label="TF-IDF matrix")
plt.title("Evolution of the Silhouette Score")
plt.legend()<jupyter_output><empty_output><jupyter_text>The highest value for the silhouette score is when there are 135 clusters. So we'll chose this value.<jupyter_code>n_clusters = 135

kmeans = KMeans(init='k-means++', n_clusters = n_clusters, n_init=30, random_state=0)
proj = kmeans.fit_transform(TF_IDF_embedded)
clusters = kmeans.predict(TF_IDF_embedded)
plt.figure(figsize=(10,10))
plt.scatter(proj[:,0], proj[:,1], c=clusters)
plt.title("ACP with 135 clusters", fontsize="20")
X = list(X)
clusters = list(clusters)
prod_seg = pd.DataFrame(list(zip(X, clusters)), 
               columns =['Description', 'prod_cluster']) 
prod_seg.to_csv('C:/Users/harini/Desktop/NEUCourses/ADM/Assignment 1/prod_seg.csv')
Y = [0,1,2,3,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,
     41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,
     78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,
     111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134]

cluster_name = ['STATIONARY',' PURSE','ZINC','SETS','VINTAGE','BOXES',' METAL',' BEADS','CUISHON',' ROSE',' BAG',
                ' HEART',' EGG',' FRAME',' RETROSPOT',' DECORATION',' GARDEN',' HOME',' CAKE',' NECKLACE',' ART',
                ' EASTER',' CANDEL',' LOLA',' LIST',' SILVER',' DOOR',' CARD',' PACK',' RIBBON',' PENCIL',' POLKADOT',
                ' BRACELET',' PAPER',' FLOWER',' HOLDER',' WRAP',' TILE',' TRAY',' MINI',' BUTTERFLY',' STAND',' JAR',
                ' BLACK',' DRAWER',' HEART',' TEA',' HOUSEHOLD',' FELT',' LUGGAGE',' DOLLY',' WITH',' T LIGHT',' PAISLEY',
                ' VINTAGE',' HANGING',' BIRD',' STAR',' IN',' BOX',' IVORY',' KEY RING',' STICKER',' CERAMIC, STRAWBERRY',
                ' EARRING',' BAG',' CHOCOLATE MOULD',' HOOK, PANTRY',' KIT',' WHITE',' BOWL',' WOODEN',' HEART',' NOTEBOOK',
                ' SET',' MUG',' MIRROR',' BLUE',' BIN, BASKET, TUMBLER',' POT',' HOLDER',' MAGIC',' SKULL',' SPACEBOY',
                ' CLOCK',' SET',' GARLAND',' BIRD',' SET',' BELL',' SMALL',' CHRISTMAS',' DESIGN',' GLASS',' SPOT',' BOX',
                ' FELTCRAFT',' BRACELET',' VINTAGE BAG',' TIN',' TOY',' SWEETHEART',' ASSORTED',' TRADITIONAL',' ZINC',
                ' PARTY',' CHRISTMAS',' BOWL',' WOOD',' GREEEN',' PAPER',' LIGHT',' LARGE',' MUG',' PURPLE',' ENAMEL',
                ' JUMBO BAG',' RED',' DOORMAT',' LOVE',' CREAM',' PINK',' GOLD',' APRON',' POLYESTER',' BAG',' ACRYLIC',
                ' CHARM',' SET',' INCENSE',' NAPKIN',' FLOWER',' BOTTLE']
prod_seg_name = pd.DataFrame(list(zip(Y, cluster_name)), 
               columns =['prod_cluster', 'Cluster']) 
prod_seg_name.head()
seg_data.columns
merge_1_1 = seg_data1.merge(prod_seg,left_on='Description',right_on='Description')
merge_2_2 = merge_1.merge(prod_seg_name, left_on = 'prod_cluster',right_on='prod_cluster')
merge_2.columns
final_data_nonuk = merge_2[['InvoiceNo', 'StockCode', 'Description', 'Quantity', 'InvoiceDate',
       'UnitPrice', 'CustomerID', 'Country', 'date', 'Total_cost','Status', 'discount','Cluster']]
final_data['month'] = pd.DatetimeIndex(final_data.InvoiceDate).month_name(locale = 'English')
final_data['year'] = pd.DatetimeIndex(final_data.InvoiceDate).year
final_data_nonuk.to_csv('C:/Users/harini/Desktop/NEUCourses/ADM/Assignment 1/final_data_nonuk.csv')
platinum = final_data[final_data['Status'] == 'Platinum']
gold = final_data[final_data['Status'] == 'Gold']
silver = final_data[final_data['Status'] == 'Silver']<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup