<jupyter_start><jupyter_text>Starbucks Capstone Challenge IntroductionThis data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks. Not all users receive the same offer, and that is the challenge to solve with this data set.Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer. Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer. ExampleTo give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer. CleaningThis makes data cleaning especially important and tricky.You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers. Final AdviceBecause this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A).  Data SetsThe data is contained in three files:* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)* profile.json - demographic data for each customer* transcript.json - records for transactions, offers received, offers viewed, and offers completedHere is the schema and explanation of each variable in the files:**portfolio.json*** id (string) - offer id* offer_type (string) - type of offer ie BOGO, discount, informational* difficulty (int) - minimum required spend to complete an offer* reward (int) - reward given for completing an offer* duration (int) - time for offer to be open, in days* channels (list of strings)**profile.json*** age (int) - age of the customer * became_member_on (int) - date when customer created an app account* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)* id (str) - customer id* income (float) - customer's income**transcript.json*** event (str) - record description (ie transaction, offer received, offer viewed, etc.)* person (str) - customer id* time (int) - time in hours since start of test. The data begins at time t=0* value - (dict of strings) - either an offer id or transaction amount depending on the record**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.  You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:Then you will want to run the above command:Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors.  Project OverviewThis project is part of the Udacity Data Scientist program. This project used to create the data simulates how people make purchasing decisions and how those decisions are influenced by promotional offers. The given simulated data for the sake of testing algorithms, not simulated data for the sake of mimicking real people.  Project StatementThe goal of this project is to build a model to see whether the people who receive the offer will complete the offer.The approach to solving these two questions are as follows:- Data Exploration- Data Preprocessing(cleaning)- Questions and Visualization- Modeling- Conclusion- Improvements  Metrics - Supervised machine learning model is used (random forest classifier)- Unsupervised machine learning model is used (artificial nueral network)- using models to predict whether user will complete offer.  Data Exploration<jupyter_code>import pandas as pd
import numpy as np
import math
import json
%matplotlib inline
import seaborn as sns
import datetime
import matplotlib.pyplot as plt

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestRegressor

# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)<jupyter_output>C:\Users\User\Anaconda3\lib\site-packages\numpy\_distributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\User\Anaconda3\lib\site-packages\numpy\.libs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\User\Anaconda3\lib\site-packages\numpy\.libs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)<jupyter_text>Portfolio dataset<jupyter_code>portfolio.head()
portfolio.info()
portfolio.shape
portfolio.isna().sum()
portfolio['offer_type'].value_counts()
portfolio['id'].nunique()
portfolio.columns.duplicated().sum()<jupyter_output><empty_output><jupyter_text>From the portfolio dataset, following things are observed:1.This dataset has no missing values nor duplicates.2.There are 10 unique offers.3.There are 4 Discount offers, 4 Bogo Offers and 2 informational offers.  Profile dataset<jupyter_code>profile.head()
profile.shape
profile.isna().sum()
profile[profile.isnull().any(axis=1)]
profile.info()
profile['id'].nunique()
profile['gender'].unique()
profile.columns.duplicated().sum()
plt.hist(profile['age'], bins=10);
profile.age.value_counts()
profile.gender.value_counts()
sns.countplot(profile['gender'])
profile['age'].describe()
profile['income'].describe()
plt.hist(profile['age'], bins=10);<jupyter_output><empty_output><jupyter_text>Following things are observed from the profile dataset:1.This dataset has no duplicates.2.This dataset has 2175 missing values on 'gender', 'income'.3.With above exploration and visualizaton with users age=118, I think it is the mobile app default setting for users who doesn't give his/her basic info for the gender and income variables.4.Male users are more than Female users than Others.  Transcript dataset<jupyter_code>transcript.head()
transcript.shape
transcript.info()
transcript.isna().sum()
transcript.columns.duplicated().sum()
transcript.event.unique()
transcript.event.value_counts()<jupyter_output><empty_output><jupyter_text>From the transcript dataset, following things are observed:1.There are 306534 rows in total.2.There is no missing value nor duplicates.3.There are 4 types of events in this dataset: 'transaction', 'offer received', 'offer viewed' and 'offer completed'.  Data Preprocessing  Portfolio dataset<jupyter_code># create a copy from the dataset
portfolio_clean = portfolio.copy()
portfolio_clean.head()
# rename 'id' column to 'offer_id'
portfolio_clean.rename(columns={'id':'offer_id'},inplace=True)
portfolio_clean['channels']<jupyter_output><empty_output><jupyter_text>Since channels is a categorical column, I will create dummy variables for it.<jupyter_code>portfolio_clean['channel_email'] = portfolio_clean['channels'].apply(lambda x: 1 if 'email' in x else 0)
portfolio_clean['channel_mobile'] = portfolio_clean['channels'].apply(lambda x: 1 if 'mobile' in x else 0)
portfolio_clean['channel_social'] = portfolio_clean['channels'].apply(lambda x: 1 if 'social' in x else 0)
portfolio_clean['channel_web'] = portfolio_clean['channels'].apply(lambda x: 1 if 'web' in x else 0)
portfolio_clean[['channels','channel_email','channel_mobile','channel_web','channel_social']].head()
# drop the 'channels'column
portfolio_clean.drop('channels', axis=1, inplace=True)
portfolio_clean.head()<jupyter_output><empty_output><jupyter_text>Since offer_type is a catergorical column, I will create dummy variables for it.<jupyter_code>portfolio_clean['offer_type_bogo'] = portfolio_clean['offer_type'].apply(lambda x: 1 if 'bogo' in x else 0)
portfolio_clean['offer_type_informational'] = portfolio_clean['offer_type'].apply(lambda x: 1 if 'informational' in x else 0)
portfolio_clean['offer_type_discount'] = portfolio_clean['offer_type'].apply(lambda x: 1 if 'social' in x else 0)
portfolio_clean.head()<jupyter_output><empty_output><jupyter_text>Profile dataset<jupyter_code>profile_clean = profile.copy()
profile_clean.head()
# renaming 'id' column name to ''user_id'.
profile_clean.rename(columns={'id':'user_id'},inplace=True)<jupyter_output><empty_output><jupyter_text>Extract year & month info from became_member_on column.<jupyter_code>profile_clean['became_member_on'] = pd.to_datetime(profile_clean['became_member_on'], format = '%Y%m%d')
profile_clean['membership_year'] = profile_clean['became_member_on'].dt.year
profile_clean['membership_month'] = profile_clean['became_member_on'].dt.month
profile_clean['membership_date'] = profile_clean['became_member_on'].dt.day
profile_clean.head()
# replacing 'age = 118' by NaN value
profile_clean['age'] = profile_clean['age'].apply(lambda x: np.nan if x == 118 else x)
# dropping rows with NaNs in 'age', 'gender' and 'income' columns
profile_clean.dropna(inplace=True)
profile_clean.isna().sum()<jupyter_output><empty_output><jupyter_text>Since gender is a categorical column, I will create the dummy variables for it.<jupyter_code>profile_clean['gender_M'] = profile_clean['gender'].apply(lambda x: 1 if 'M' in x else 0)
profile_clean['gender_F'] = profile_clean['gender'].apply(lambda x: 1 if 'F' in x else 0)
profile_clean['gender_O'] = profile_clean['gender'].apply(lambda x: 1 if 'O' in x else 0)
profile_clean.drop(['became_member_on'], axis=1, inplace=True)
profile_clean.head()<jupyter_output><empty_output><jupyter_text>Transcript dataset<jupyter_code>transcript_clean = transcript.copy()
# rename 'person' to 'user_id'
transcript_clean.rename(columns={'person': 'user_id'}, inplace=True)
transcript_clean.head()
# Extract each key that exist in 'value' column to a seperate column.
# getting the different keys  that exists in the 'value' column
keys = []
for idx, row in transcript_clean.iterrows():
    for k in row['value']:
        if k in keys:
            continue
        else:
            keys.append(k)
keys
#create columns and specify the datatype of each of them
transcript_clean['offer_id'] = '' # datatype : string
transcript_clean['amount'] = 0  # datatype : integer
transcript_clean['reward'] = 0  # datatype : integer
# Iterating over transcript_clean dataset and checking 'value' column
# then updating it and using the values to fill in the columns created above
for idx, row in transcript_clean.iterrows():
    for k in row['value']:
        if k == 'offer_id' or k == 'offer id': # b/c 'offer_id' and 'offer id' are representing the same thing 
            transcript_clean.at[idx, 'offer_id'] = row['value'][k]
        if k == 'amount':
            transcript_clean.at[idx, 'amount'] = row['value'][k]
        if k == 'reward':
            transcript_clean.at[idx, 'reward'] = row['value'][k]
# filling all the NaNs in the 'offer_id' column with 'N/A' values
transcript_clean['offer_id'] = transcript_clean['offer_id'].apply(lambda x: 'N/A' if x == '' else x)

# drop the 'value' column 
transcript_clean.drop(columns=['value','amount','reward'], axis=1, inplace=True)
transcript_clean.head()
# check the unique values in 'event' column
transcript['event'].unique()
# create 'offer received', 'offer viewed', 'transaction', 'offer completed' columns in the dataset
transcript_clean['offer_received'] = transcript_clean['event'].apply(lambda x: 1 if 'offer received' in x else 0)
transcript_clean['offer_viewed'] = transcript_clean['event'].apply(lambda x: 1 if 'offer viewed' in x else 0)
transcript_clean['transaction'] = transcript_clean['event'].apply(lambda x: 1 if 'transaction' in x else 0)
transcript_clean['offer_completed'] = transcript_clean['event'].apply(lambda x: 1 if 'offer completed' in x else 0)
transcript_clean.head()<jupyter_output><empty_output><jupyter_text>Merge three clean dataset( Portfolio, Profile, Transaction) into one master clean dataset.<jupyter_code># merge 'transcript_clean' dataset with 'portfolio_clean' on 'offer_id'
df_master =transcript_clean.merge(portfolio_clean,how='left',on='offer_id')
# merge'df_master' dataset with 'profile_clean' on 'user_id'
df_master = df_master.merge(profile_clean,how ='left', on = 'user_id')
df_master.head()
df_master.info()
# removing rows that contain NaNs
df_master = df_master.dropna(how='any',axis=0)
df_master.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 148805 entries, 0 to 306527
Data columns (total 28 columns):
 #   Column                    Non-Null Count   Dtype  
---  ------                    --------------   -----  
 0   user_id                   148805 non-null  object 
 1   event                     148805 non-null  object 
 2   time                      148805 non-null  int64  
 3   offer_id                  148805 non-null  object 
 4   offer_received            148805 non-null  int64  
 5   offer_viewed              148805 non-null  int64  
 6   transaction               148805 non-null  int64  
 7   offer_completed           148805 non-null  int64  
 8   reward                    148805 non-null  float64
 9   difficulty                148805 non-null  float64
 10  duration                  148805 non-null  float64
 11  offer_type                148805 non-null  object 
 12  channel_email             148805 non-null  float64
 13  channel_mobile            148805 non-null  f[...]<jupyter_text>Questions and Visualization  Questions1.What is the most common age distribution for Starbucks customers?2.What is the most common income for Starbucks customers?3.Who gets more income, male or female?4.Which type of offers each gender likes?5.From all the offers that customers reviewed, how many offers they completed?6.How many new customers Starbucks gets each year?7.What is the income among different ages, at what ages would have the most disposable income? 1.What is the most common age distribution for Starbucks customers?<jupyter_code>plt.figure(figsize=(16, 4))
sns.boxplot(df_master['age'])
plt.title('Age Distribution')
plt.xlabel('Age')
plt.xticks(rotation = 90)
plt.show();<jupyter_output><empty_output><jupyter_text>From the above age distribution plot, we can see that most of ages in our profile dataframe falls in-between 40 and 80. We already notice one outlier which is 118.  2.What is the most common income for Starbucks customers?<jupyter_code>plt.figure(figsize=(16, 4))
sns.boxplot(df_master['income'])
plt.title('Income Distribution')
plt.xlabel('income')
plt.xticks(rotation = 90)
plt.show();<jupyter_output><empty_output><jupyter_text>From the above income distribution plot, we can see that the most common of income for the customers that falls between 50K and 80K. 3.Who gets more income, male or female?<jupyter_code>plt.figure(figsize=(14, 6))
sns.violinplot(x=df_master[df_master['gender'] != 'NA']['gender'], y=df_master['income'])
plt.title('Income vs Gender')
plt.ylabel('Income')
plt.xlabel('Gender')
plt.xticks(rotation = 0)
plt.show();<jupyter_output><empty_output><jupyter_text>The graph above shows that income median (the white dot) for females (around 70k) is higher than males (around 60k). We can also see that the income range for females is 40k to 100k. The income range for the males is 40k to 70k which close to median. 4.Which type of offers each gender likes?<jupyter_code>plt.figure(figsize=(14, 6))
sns.countplot(x=df_master[df_master["gender"] != 'NA']['gender'], hue="offer_type", data=df_master)
plt.title('Most Popular Offers to Each Gender')
plt.ylabel('Total')
plt.xlabel('Gender')
plt.xticks(rotation = 0)
plt.legend(title='Offer Type')
plt.show();<jupyter_output><empty_output><jupyter_text>From the above plot graph, we can see that both genders like bogo and discount offers and they have the same reaction to informational offers, they both not interested in it. 5.From all the offers that customers reviewed, how many offers they completed?<jupyter_code>total_trans_g_o = df_master[df_master["gender"]!= 'NA'].groupby(['gender','offer_type']).count()
total_trans_g_e = df_master[df_master["gender"]!= 'NA'].groupby(['gender','event']).count()
total_trans_go_o_t = total_trans_g_o.loc[('F')]['event'].sum()
total_trans_go_o_tt = total_trans_g_o.loc[('M')]['event'].sum()
total_trans_go_o_t_offers_f = total_trans_g_o.loc[('F')].loc[['bogo', 'discount', 'informational']]['event'].sum()
total_trans_go_o_t_offers_m = total_trans_g_o.loc[('M')].loc[['bogo', 'discount', 'informational']]['event'].sum()

print('For Females:')
print(f"Number of offer completed: {total_trans_g_e.loc[('F', 'offer completed')].values[0]}, {round((total_trans_g_e.loc[('F', 'offer completed')].values[0]/total_trans_g_e.loc[('F', 'offer received')].values[0])*100,2)}% of total offers received.")
print(f"Number of offer received: {total_trans_g_e.loc[('F', 'offer received')].values[0]}, {round((total_trans_g_e.loc[('F', 'offer received')].values[0]/total_trans_go_o_t_offers_f)*100,2)}% of total.")
print(f"Number of offer viewed: {total_trans_g_e.loc[('F', 'offer viewed')].values[0]}, {round((total_trans_g_e.loc[('F', 'offer viewed')].values[0]/total_trans_go_o_t_offers_f)*100,2)}% of total.")
print(f"Number of transaction: {total_trans_g_e.loc[('F', 'transaction')].values[0]}, {round((total_trans_g_e.loc[('F', 'transaction')].values[0]/total_trans_go_o_t)*100,2)}% of total.")

print('\nFor Males:')
print(f"Number of offer completed: {total_trans_g_e.loc[('M', 'offer completed')].values[0]}, {round((total_trans_g_e.loc[('M', 'offer completed')].values[0]/total_trans_g_e.loc[('M', 'offer received')].values[0])*100,2)}% of total offers received.")
print(f"Number of offer received: {total_trans_g_e.loc[('M', 'offer received')].values[0]}, {round((total_trans_g_e.loc[('M', 'offer received')].values[0]/total_trans_go_o_t_offers_m)*100,2)}% of total.")
print(f"Number of offer viewed: {total_trans_g_e.loc[('M', 'offer viewed')].values[0]}, {round((total_trans_g_e.loc[('M', 'offer viewed')].values[0]/total_trans_go_o_t_offers_m)*100,2)}% of total.")
print(f"Number of transaction: {total_trans_g_e.loc[('M', 'transaction')].values[0]}, {round((total_trans_g_e.loc[('M', 'transaction')].values[0]/total_trans_go_o_t)*100,2)}% of total.")<jupyter_output>For Females:
Number of offer completed: 15477, 56.37% of total offers received.
Number of offer received: 27456, 43.09% of total.
Number of offer viewed: 20786, 32.62% of total.
Number of transaction: 15477, 24.29% of total.

For Males:
Number of offer completed: 16466, 43.18% of total offers received.
Number of offer received: 38129, 46.0% of total.
Number of offer viewed: 28301, 34.14% of total.
Number of transaction: 16466, 25.84% of total.<jupyter_text>Females completed 56% of the offers they received, it is 13% more than males, but males made more transactions than females, 25% to 24%. 6.How many new customers Starbucks gets each year?<jupyter_code>df_master['membership_year'] = df_master['membership_year'].astype(int)
plt.figure(figsize=(16, 4))
sns.countplot(df_master['membership_year'])
plt.title('Number of Profiles In Each Year')
plt.ylabel('Number of Profiles')
plt.xlabel('Year')
plt.xticks()
plt.show();
# getting the number of customers that became members on 2017
members_2017 = (df_master['membership_year']==2017).sum()

# getting the total number of members among all the available years
total = df_master['membership_year'].count()

# getting the percentages of customers that became members on 2017
perc_2017 = round((members_2017/total)*100,2)

print(members_2017)
perc_2017<jupyter_output>56063<jupyter_text>From the above plot graph we can see that, 2017 has the highest number of profiles. About 38% of all the customers are registered as members on 2017. 7.What is the income among different ages, at what ages would have the most disposable income?<jupyter_code>dataAge = profile.groupby('age')['income'].sum().reset_index()
dataAge.head()
dataAge.plot.bar(x='age', y='income', rot=0, figsize=(20,6),color = '#00A0A0')<jupyter_output><empty_output><jupyter_text>The age group with the most disposable income is between age 48 and 70.  Data Modeling  Data Prep<jupyter_code>df_master.info()
# dropping all columns that have been encoded or not needed for training model
df_model = df_master.drop(columns = ['event','user_id','time','offer_id','transaction','offer_type','gender','offer_received','offer_viewed'])
#moving the y denpendent variable to the first column
col_name = "offer_completed"
first_col = df_model.pop(col_name)
df_model.insert(0, col_name, first_col)
df_model.info()
X = df_model.iloc[:, 1:].values
y = df_model.iloc[:, 0].values
# feature scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit_transform(X)
#splitting data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)<jupyter_output><empty_output><jupyter_text>Supervised Model<jupyter_code>#training classifier
from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators = 200, criterion = 'entropy', random_state = 0)
classifier.fit(X_train, y_train)
#evaluating classifier with confusion matrix

from sklearn.metrics import confusion_matrix
y_pred = classifier.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)
#evluating classifier with k-fold cross-validation

from sklearn.model_selection import cross_val_score
accuracy = cross_val_score(estimator = classifier, X= X_train, y = y_train, cv = 10)
print("Accuracy: {:.2f} %".format(accuracy.mean()*100))
print("Standard Deviation: {:.2f} %".format(accuracy.std()*100))
df_model.info()
# Feature importance in RandomForest Classifier
col = [ 'reward','difficulty','duration','channel_email','channel_mobile' 
       'channel_social', 'channel_web', 'offer_type_bogo','offer_type_informational','offer_type_discount','age','income',
       'membership_year','membership_month','membership_date','gender_M','gender_F','gender_O']
#modelname.feature_importance_
imp = classifier.feature_importances_
#plot
fig, ax = plt.subplots() 
width = 0.4 # the width of the bars 
ind = np.arange(len(imp)) # the x locations for the groups
ax.barh(ind, imp, width, color='green')
ax.set_yticks(ind+width/10)
ax.set_yticklabels(col, minor=False)
plt.title('Feature importance in RandomForest Classifier')
plt.xlabel('Relative importance')
plt.ylabel('feature') 
plt.figure(figsize=(5,5))
fig.set_size_inches(6.5, 4.5, forward=True)<jupyter_output><empty_output><jupyter_text>Unsupervised model<jupyter_code>import tensorflow as tf

#Training artificial neural network
ann = tf.keras.models.Sequential()
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))
ann.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

ann.fit(X_train, y_train, batch_size = 32, epochs = 100)
# evaluating ann model

y_pred1 = ann.predict(X_test)
y_pred1 = (y_pred1 > 0.5)
cm = confusion_matrix(y_test, y_pred1)
print(cm)<jupyter_output>[[23281     0]
 [ 6480     0]]
StarcoderdataJupyterStructuredCleanDedup