<jupyter_start><jupyter_text>Capstone Project - The Battle of Neighborhoods Where to live in Paris? ![alt text](paris-cityscape-overview-guide.jpg)  I. Introduction Paris is a vibrant and complex city. For someone who has not been living in Paris for many years, the city may look impenetrable. Which neighborhoods are great for a coffee? Which neighborhoods are famous for its markets? Where are located the best bars? When confronted with these questions, tourists and new residents generally use a guide or use websites such as Yelp or Tripadvisor. My experience with these platforms has often been disappointing because the website (or the guide) is not tailored to my tastes and preferences. For this capstone project, I would like to offer an alternative based on data mining and clustering. Data on Paris neighborhoods amenities (bars, cafés, museums, bakeries, etc.) can easily be collected and treated to generate a map of Paris. Based on the user's preferences, we can direct the user towards a specific neighborhood in Paris. This approach can be seen as a refinement of the traditional tourism websites, with the addition of a data-driven customization layer. Having a data-driven map of Paris is also helpful for people moving to Paris when deciding where to live. The French capital is extremely expensive. Yes, [the Marais](https://en.wikipedia.org/wiki/The_Marais) is great, but maybe you prefer living in the much cheaper 19th or 20th? To answer this question, we need data and a robust clustering methodology.  II. Data To create a data-driven map of Paris' neighborhoods, I will data from [Foursquare API](https://developer.foursquare.com/). Foursquare defines itself as "a location technology platform dedicated to improving how people move through the real world".In practice, people use Foursquare [platform](https://foursquare.com/bestbarsuk) or app to find places. They can then rate the place, give a rating, add photos and/or a description. The Foursquare API allows us the retrieve the data that was created by users. I will also use data from Wikipedia to get information on Paris. This [page](https://en.wikipedia.org/wiki/Arrondissements_of_Paris) contains the name of Paris areas ("arrondissements"), as well as some basic information such as area and population.  --------------  III. Cluster Analysis  Table of ContentsPart A: Download Neighborhoods DataPart B: Geocode NeighborhoodsPart C: Cluster Analysis1. Download and Explore Dataset using the Foursquare API 2. Explore Neighborhoods in Paris 3. Analyze Each Neighborhood4. Cluster Neighborhoods5. Examine Clusters      Part A: Download Neighborhoods Data<jupyter_code>import numpy as np # library to handle data in a vectorized manner

import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

#!conda install -c conda-forge geopy --yes # uncomment this line if you haven't completed the Foursquare API lab
from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

# import k-means from clustering stage
from sklearn.cluster import KMeans

#!conda install -c conda-forge folium=0.5.0 --yes # uncomment this line if you haven't completed the Foursquare API lab
import folium # map rendering library
import time

#to create wordclouds
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator

import matplotlib.pyplot as plt
%matplotlib inline

print('Libraries imported.')
address = 'Paris, France'

geolocator = Nominatim(user_agent="par<PASSWORD>")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geograpical coordinate of Paris are {}, {}.'.format(latitude, longitude))
# create map of New York using latitude and longitude values
map_paris = folium.Map(location=[latitude, longitude], zoom_start=12)

map_paris<jupyter_output><empty_output><jupyter_text>Let's try to add "arrondissements" to the map above.<jupyter_code>import pandas as pd
import numpy as np

link = "https://en.wikipedia.org/wiki/Arrondissements_of_Paris"

tables = pd.read_html(link, header=0)

# read_html creates a list of dataframes.
# Let's just take the first one:
df = tables[2]

# rename the first column:
df.rename(columns = {"Arrondissement (R for Right Bank, L for Left Bank)" : "Arrondissement"}, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Part B: Geocode Neighborhoods Let's geocode the dataframe above:<jupyter_code># List to store values
Postcode = []
Borough = []
Neighbourhood = []

# Options
# Normal sleeping time
sleeping_time = 0.1
# Sleeping time in case of error
sleeping_time_error = 1.0
# Max number of attempts (sometimes, we don't get a value)
max_nb_attempts = 2
geolocator = Nominatim(user_agent="Coursera_Capstone")
#Initialization
df["Latitude"] = 0
df["Longitude"] = 0

for index, row in df.iterrows():
    # Select the first neighborhood
    l =  row["Name"] + ", Paris France"
    print(l)
    # Loop until success
    current_attempt = 0
    while True:
        current_attempt +=1 
        try:
            location = geolocator.geocode(l)
            print((location.latitude, location.longitude))
        except:
            print("Error with geolocator.geocode(l)")
            print("Sleeping for a while and trying again")
            time.sleep(sleeping_time_error)      
        # Sleep for some time to prevent us being blocked:
        if location is not None:
            time.sleep(sleeping_time)
            df.loc[index, "Latitude"] = location.latitude
            df.loc[index, "Longitude"] = location.longitude
            # exit while loop
            break
        # If error, sleep a little bit longer
        else:
            print("Sleeping for a while and trying again")
            time.sleep(sleeping_time_error)
        # Exit if reached the max number of attempts
        if current_attempt == max_nb_attempts:
            print("Max number of attempts reached. Setting lat and lon to 0")
            df.loc[index,"Latitude"] = 0
            df.loc[index,"Longitude"] = 0
            break
df.head()
# create map of New York using latitude and longitude values
map_paris = folium.Map(location=[latitude, longitude], zoom_start=12)

# add markers to map
for (lat, lng, borough, arr) in zip(df['Latitude'], df['Longitude'], df['Name'], df['Arrondissement']):
    label = '{},{}'.format(borough, arr)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_paris)  
    
map_paris<jupyter_output><empty_output><jupyter_text>![alt text](map_Paris_2.png)  Part C: Cluster Analysis  Download and Explore Dataset using the Foursquare API  Let's first read credentials, which are stored in a json file<jupyter_code>from pathlib import Path
import json

path_to_main = Path().absolute()


#insert your own generated keys:
#Read JSON data into the datastore variable
filename = str(path_to_main) + "/ID_Foursquare.json"
if filename:
    with open(filename, 'r') as f:
        ids = json.load(f)
else:
    print("ID_Foursquare.json not found. Cannot connect to Foursquare.")

CLIENT_ID = ids["Client_Id"] # your Foursquare ID
CLIENT_SECRET = ids["Client_Secret"] # your Foursquare Secret
ACCESS_TOKEN = ids["Access_Token"]
VERSION = '20180604'
df.head()<jupyter_output><empty_output><jupyter_text>Now, let's get the top 100 venues that are in the first arrondissement with radius of 500 meters.<jupyter_code>LIMIT = 500 # limit of number of venues returned by Foursquare API
radius = 1000 # define radius (in meters)

# create URL
url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
    CLIENT_ID, 
    CLIENT_SECRET, 
    VERSION, 
    df.loc[0, "Latitude"], 
    df.loc[0, "Longitude"], 
    radius, 
    LIMIT)<jupyter_output><empty_output><jupyter_text>Let's examine the request<jupyter_code>results = requests.get(url).json()<jupyter_output><empty_output><jupyter_text>From the Foursquare lab in the previous module, we know that all the information is in the *items* key. Before we proceed, let's borrow the **get_category_type** function from the Foursquare lab.<jupyter_code># function that extracts the category of the venue
def get_category_type(row):
    try:
        categories_list = row['categories']
    except:
        categories_list = row['venue.categories']
        
    if len(categories_list) == 0:
        return None
    else:
        return categories_list[0]['name']<jupyter_output><empty_output><jupyter_text>Now we are ready to clean the json and structure it into a *pandas* dataframe.<jupyter_code>venues = results['response']['groups'][0]['items']
    
nearby_venues = json_normalize(venues) # flatten JSON

# filter columns
filtered_columns = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
nearby_venues =nearby_venues.loc[:, filtered_columns]

# filter the category for each row
nearby_venues['venue.categories'] = nearby_venues.apply(get_category_type, axis=1)

# clean columns
nearby_venues.columns = [col.split(".")[-1] for col in nearby_venues.columns]

nearby_venues.head()<jupyter_output><empty_output><jupyter_text>And how many venues were returned by Foursquare?<jupyter_code>print('{} venues were returned by Foursquare.'.format(nearby_venues.shape[0]))<jupyter_output>100 venues were returned by Foursquare.<jupyter_text>Let's create a function to repeat the same process to all the neighborhoods in Paris<jupyter_code>def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)<jupyter_output><empty_output><jupyter_text>Let's collect venues for each arrondissement.<jupyter_code>paris_venues = getNearbyVenues(names=df['Name'],
                                   latitudes=df['Latitude'],
                                   longitudes=df['Longitude'],
                                radius=radius)<jupyter_output>Louvre
Bourse
Temple
Hôtel-de-Ville
Panthéon
Luxembourg
Palais-Bourbon
Élysée
Opéra
Entrepôt
Popincourt
Reuilly
Gobelins
Observatoire
Vaugirard
Passy
Batignolles-Monceau
Butte-Montmartre
Buttes-Chaumont
Ménilmontant<jupyter_text>Let's check the size of the resulting dataframe<jupyter_code>print(paris_venues.shape)
toronto_venues.head()<jupyter_output>(1921, 7)<jupyter_text>Let's check how many venues were returned for each neighborhood<jupyter_code>paris_venues.groupby('Neighborhood').count()<jupyter_output><empty_output><jupyter_text>How many different categories do we have?<jupyter_code>print('There are {} uniques categories.'.format(len(paris_venues['Venue Category'].unique())))<jupyter_output>There are 230 uniques categories.<jupyter_text>Let's save the dataframe to disk.<jupyter_code>paris_venues.to_csv("data/paris_venues.csv")
print("paris_venues.csv saved")<jupyter_output>paris_venues.csv saved<jupyter_text>3. Analyze Each Neighborhood<jupyter_code># one hot encoding
paris_onehot = pd.get_dummies(paris_venues[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
paris_onehot['Neighborhood'] = paris_venues['Neighborhood'] 

# move neighborhood column to the first column
fixed_columns = [paris_onehot.columns[-1]] + list(paris_onehot.columns[:-1])
paris_onehot = paris_onehot[fixed_columns]

print(paris_onehot.shape)
paris_onehot.head()<jupyter_output>(1921, 231)<jupyter_text>Next, let's group rows by neighborhood and by taking the mean of the frequency of occurrence of each category<jupyter_code>paris_grouped = paris_onehot.groupby('Neighborhood').mean().reset_index()
paris_grouped.head()<jupyter_output><empty_output><jupyter_text>paris_grouped should have 20 rows. Is it the case?<jupyter_code>paris_grouped.shape<jupyter_output><empty_output><jupyter_text>Let's print each neighborhood along with the top 5 most common venues Not surprisingly, the category **"French Restaurant"** often comes first<jupyter_code>num_top_venues = 5

for hood in paris_grouped['Neighborhood']:
    print("----"+ hood+"----")
    temp = paris_grouped[paris_grouped['Neighborhood'] == hood].T.reset_index()
    temp.columns = ['venue','freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending=False).reset_index(drop=True).head(num_top_venues))
    print('\n')
    print('\n')<jupyter_output>----Batignolles-Monceau----
                venue  freq
0   French Restaurant  0.20
1               Hotel  0.06
2  Italian Restaurant  0.06
3            Wine Bar  0.05
4                 Bar  0.04




----Bourse----
                 venue  freq
0    French Restaurant  0.11
1  Japanese Restaurant  0.07
2             Wine Bar  0.06
3                Hotel  0.05
4               Bakery  0.04




----Butte-Montmartre----
               venue  freq
0  French Restaurant  0.18
1                Bar  0.09
2        Pizza Place  0.05
3             Bistro  0.04
4               Café  0.04




----Buttes-Chaumont----
               venue  freq
0  French Restaurant  0.14
1                Bar  0.09
2               Café  0.07
3         Restaurant  0.05
4       Cocktail Bar  0.04




----Entrepôt----
               venue  freq
0  French Restaurant  0.08
1        Coffee Shop  0.07
2  Indian Restaurant  0.05
3              Hotel  0.04
4        Pizza Place  0.04




----Gobelins----
                   venue  [...]<jupyter_text>Let's put that into a *pandas* dataframe Now let's create the new dataframe and display the top 10 venues for each neighborhood.<jupyter_code>def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]
num_top_venues = 10

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhood']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhood'] = paris_grouped['Neighborhood']

for ind in np.arange(paris_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(paris_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted<jupyter_output><empty_output><jupyter_text>4. Cluster Neighborhoods Run *k*-means to cluster the neighborhood into 5 clusters.<jupyter_code># set number of clusters
kclusters = 5

paris_grouped_clustering = paris_grouped.drop('Neighborhood', 1)

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(paris_grouped_clustering)

# check cluster labels generated for each row in the dataframe
kmeans.labels_[0:10]<jupyter_output><empty_output><jupyter_text>Let's create a new dataframe that includes the cluster as well as the top 10 venues for each neighborhood.<jupyter_code># add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)
paris_merged = df

# rename for consistency:
paris_merged.rename(columns={"Name": "Neighborhood"}, inplace=True)

# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood
paris_merged = paris_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')

# drop a missing labels
paris_merged.dropna(subset = ["Cluster Labels"], inplace=True)

# convert to int
paris_merged["Cluster Labels"] = paris_merged["Cluster Labels"].astype(int)


paris_merged.head()
# create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=12)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(paris_merged['Latitude'], paris_merged['Longitude'], paris_merged['Arrondissement'], paris_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[cluster-1],
        fill=True,
        fill_color=rainbow[cluster-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters<jupyter_output><empty_output><jupyter_text>![alt text](map_paris_clusters.png)  5. Examine Clusters Now, you can examine each cluster and determine the discriminating venue categories that distinguish each cluster.<jupyter_code>for k in range(0,kclusters):
    print("---------------------")
    print("Cluster {}".format(k))
    print("---------------------")
    print(paris_merged.loc[paris_merged['Cluster Labels'] == k, paris_merged.columns[[1] + list(range(5, paris_merged.shape[1]))]].head())
    print("\n")<jupyter_output>---------------------
Cluster 0
---------------------
      Neighborhood Density (2005)(inhabitants per km2) Peak of population  \
4         Panthéon                               23849               1911   
6   Palais-Bourbon                               13552               1926   
7           Élysée                                9972               1891   
13    Observatoire                               23964               1954   
14       Vaugirard                               27335               1962   

                       Mayor   Latitude  Longitude  Cluster Labels  \
4     <NAME> (LR)  48.846191   2.346079               0   
6          <NAME> (LR)  48.861596   2.317909               0   
7   <NAME> (LR)  48.846644   2.369830               0   
13         <NAME> (PS)  48.829567   2.323962               0   
14      <NAME> (LR)  48.841430   2.296165               0   

   1st Most Common Venue 2nd Most Common Venue 3rd Most Common[...]<jupyter_text>Cluster 1 The first cluster (label 0) is composed of the **5th, 7th, 8th, 14th, 15th, 16th and 17th arrondissements**.* The first most common venue: French restaurants* The second most common venue: hotels* The third most common venue: italian restaurants<jupyter_code>df_cluster_1 = paris_merged[paris_merged['Cluster Labels'] == 0]
df_cluster_1<jupyter_output><empty_output><jupyter_text>Visualize common venues<jupyter_code>list_venues = []
for index, row in df_cluster_1.iterrows():
    #loop over top 10 venues:
    for i in range(1,11):
        #let's create a ponderation to take into account the order
        #that is, we add several times the same word based on the order
        #(it would be more accurate to weight by the actual number of occurances)
        for k in range(0,i):
            list_venues.append(df_cluster_1.iloc[0, -i])
list_venues = (" ").join(list_venues)

# instantiate a word cloud object\n",
cluster1_wc = WordCloud(collocations=False, background_color='white', max_words=2000)
cluster1_wc.generate(list_venues)

plt.figure(figsize=(15,8))
plt.imshow(cluster1_wc)
plt.axis("off")
plt.savefig("cluster1.png", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>![alt text](cluster1.png)  Cluster 2 The first cluster (label 1) is composed of the **12th arrondissement**.* The first most common venue: theater* The second most common venue: stadium* The third most common venue: recreation center<jupyter_code>df_cluster_2 = paris_merged[paris_merged['Cluster Labels'] == 1]
df_cluster_2
list_venues = []
for index, row in df_cluster_2.iterrows():
    #loop over top 10 venues:
    for i in range(1,11):
        #let's create a ponderation to take into account the order
        #that is, we add several times the same word based on the order
        #(it would be more accurate to weight by the actual number of occurances)
        for k in range(0,i):
            list_venues.append(df_cluster_2.iloc[0, -i])
list_venues = (" ").join(list_venues)

# instantiate a word cloud object\n",
cluster1_wc = WordCloud(collocations=False, background_color='white', max_words=2000)
cluster1_wc.generate(list_venues)

plt.figure(figsize=(15,8))
plt.imshow(cluster1_wc)
plt.axis("off")
plt.savefig("cluster2.png", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>![alt text](cluster2.png)  Cluster 3 The first cluster (label 2) is composed of the **1st, 2nd, 3rd, 4th, 6th, 9th and 10th arrondissements**.* The first most common venue: French restaurants* The second most common venue and third most common venue are quite diverse (coffee shops, hotels, wine bars)<jupyter_code>df_cluster_3 = paris_merged[paris_merged['Cluster Labels'] == 2]
df_cluster_3
list_venues = []
for index, row in df_cluster_3.iterrows():
    #loop over top 10 venues:
    for i in range(1,11):
        #let's create a ponderation to take into account the order
        #that is, we add several times the same word based on the order
        #(it would be more accurate to weight by the actual number of occurances)
        for k in range(0,i):
            list_venues.append(df_cluster_3.iloc[0, -i])
list_venues = (" ").join(list_venues)

# instantiate a word cloud object\n",
cluster1_wc = WordCloud(collocations=False, background_color='white', max_words=2000)
cluster1_wc.generate(list_venues)

plt.figure(figsize=(15,8))
plt.imshow(cluster1_wc)
plt.axis("off")
plt.savefig("cluster3.png", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>![alt text](cluster3.png)  Cluster 4 The 4th cluster (label 3) is composed of the **11th, 18th, 19th, and 20th arrondissements**.* The first most common venue: French restaurants* The second most common venue: bars* The third most common venue: pizza places<jupyter_code>df_cluster_4 = paris_merged[paris_merged['Cluster Labels'] == 3]
df_cluster_4
list_venues = []
for index, row in df_cluster_4.iterrows():
    #loop over top 10 venues:
    for i in range(1,11):
        #let's create a ponderation to take into account the order
        #that is, we add several times the same word based on the order
        #(it would be more accurate to weight by the actual number of occurances)
        for k in range(0,i):
            list_venues.append(df_cluster_4.iloc[0, -i])
list_venues = (" ").join(list_venues)

# instantiate a word cloud object\n",
cluster1_wc = WordCloud(collocations=False, background_color='white', max_words=2000)
cluster1_wc.generate(list_venues)

plt.figure(figsize=(15,8))
plt.imshow(cluster1_wc)
plt.axis("off")
plt.savefig("cluster4.png", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>![alt text](cluster4.png)  Cluster 5 The 5th cluster (label 4) is composed of the **11th, 18th, 19th, and 20th arrondissements**.* The first most common venue: Vietnamese restaurants* The second most common venue: Thai restaurants* The third most common venue: French restaurants<jupyter_code>df_cluster_5 = paris_merged[paris_merged['Cluster Labels'] == 4]
df_cluster_5
list_venues = []
for index, row in df_cluster_5.iterrows():
    #loop over top 10 venues:
    for i in range(1,11):
        #let's create a ponderation to take into account the order
        #that is, we add several times the same word based on the order
        #(it would be more accurate to weight by the actual number of occurances)
        for k in range(0,i):
            list_venues.append(df_cluster_5.iloc[0, -i])
list_venues = (" ").join(list_venues)

# instantiate a word cloud object\n",
cluster1_wc = WordCloud(collocations=False, background_color='white', max_words=2000)
cluster1_wc.generate(list_venues)

plt.figure(figsize=(15,8))
plt.imshow(cluster1_wc)
plt.axis("off")
plt.savefig("cluster5.png", bbox_inches='tight')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup