<jupyter_start><jupyter_text>__DATA PREPROCESSING__<jupyter_code># Separate hotels by country
china_hotels = hotels[hotels['city_id'] == 31497].reset_index()
greece_hotels = hotels[hotels['city_id'] == 14121].reset_index()
usa_hotels = hotels[hotels['city_id'] == 14257].reset_index()
neth_hotels = hotels[hotels['city_id'] == 27561].reset_index()
# Clear one misleading city name
pois['country'] = pois['country'].replace("Hong Kong", "China")
pois.head()
# Get unique poi types
poi_types = pois['poi_types'].tolist()
poi_str = ", ".join(poi_type for poi_type in poi_types)
poi_str_list = poi_str.split(",")
poi_list = list(set(poi_str_list))

for i in range(len(poi_list)):
    poi_list[i] = poi_list[i].lstrip(' ')

poi_list
# Manipulate pois dataset to get only nightlife features that are involved
elim_index = []
for index, row in pois.iterrows():
    if "Bar / Pub" in row['poi_types'] or "Disco / Nightclub" in row['poi_types'] or "Casino" in row['poi_types']:
        elim_index.append(index)
        
poi_elim = pois[pois.index.isin(elim_index)].reset_index()
poi_elim.head()
# Separate pois by country
china_poi = poi_elim[poi_elim['city_id'] == 31497].reset_index()
greece_poi = poi_elim[poi_elim['city_id'] == 14121].reset_index()
usa_poi = poi_elim[poi_elim['city_id'] == 14257].reset_index()
neth_poi = poi_elim[poi_elim['city_id'] == 27561].reset_index()<jupyter_output><empty_output><jupyter_text>__ALGORITHM EXPLANATION__After exploring the data and obtaining insights, I start to implement my algorithm here.The basic idea of this algorithm is to assign nigthlife scores to given hotels according to their distance to "Disco / Nightclub", "Bar / Pub", and "Casino". I also took "distance_to_center" feature into account for being closely related with the scoring structure. I dug into it in further process.I begin with calculating each hotel's distance to all selected poi locations by using longitude and latitude for each city separately.  Here I start with Hotels in China and their distance to selected poi locations and repeat this process for each city.  While calculating the distance, I take the __mean__ of the list of distances for each hotel to get an average distance that hotel has for nightlife locations.  __Note:__ I placed __"club_club_hotel"__ and "__party_people__" features into the dataset as well by filling unknown values with "__0.0__" values. Nevertheless, I decided not to use  this feature because first, those include quite a lot of unknown information in it, second, "__0.0__" value would mean negative for a nightlife aspect where there is actually no information about  this.<jupyter_code>china_distances = []
for i in range(len(china_hotels)):
    per_bar_distance = []
    hotel_long = china_hotels.iloc[i][6]
    hotel_lat = china_hotels.iloc[i][7]
    hotel_coord = (hotel_lat, hotel_long)
    for j in range(len(china_poi)):
        poi_long = china_poi.iloc[j][5]
        poi_lat = china_poi.iloc[j][6]
        poi_coord = (poi_lat, poi_long)
        
        # Calculate the distance by using given longitude and latitude in "meters"
        hotel_poi_distance = geopy.distance.geodesic(hotel_coord, poi_coord).m
        per_bar_distance.append(hotel_poi_distance)
    china_distances.append(per_bar_distance)

china_hotels['nightlife_distance'] = china_distances
china_hotels['mean_distance'] = [statistics.mean(distance_list) for distance_list in china_distances]
china_hotels['median_distance'] = [statistics.median(distance_list) for distance_list in china_distances]
china_train = china_hotels[["hotel_id", "city_id", "distance_to_center", "club_club_hotel", "party_people", "nightlife_distance", "mean_distance", "median_distance", "overall_rating"]]
china_train = china_train.fillna(0.0)
china_train<jupyter_output><empty_output><jupyter_text>- I used scatter plot to visualize the distribution of the __"mean_distance"__ and __"distance_to_center"__ features. Besides this plot shows that these two features are directly  proportional to each other in a linear trend.  - One other takeaway from this plot is that there is a __density__ where the hotel's distance to city center and the mean distance to nightlife locations are taking the lowest values. On the other hand, as in China data and others, there is an outlier where the value of each feature is the highest. The data point is appearing in a very far position from where other  sample points located on the plot<jupyter_code>plt.figure(figsize=(15, 15))
plt.scatter(china_train.distance_to_center, china_train.mean_distance)
plt.title("China Hotels", fontsize=18)
plt.xlabel('distance_to_center', fontsize=18)
plt.ylabel('mean_distance', fontsize=18)
plt.show()
greece_distances = []
for i in range(len(greece_hotels)):
    per_bar_distance = []
    hotel_long = greece_hotels.iloc[i][6]
    hotel_lat = greece_hotels.iloc[i][7]
    hotel_coord = (hotel_lat, hotel_long)
    for j in range(len(greece_poi)):
        poi_long = greece_poi.iloc[j][5]
        poi_lat = greece_poi.iloc[j][6]
        poi_coord = (poi_lat, poi_long)
        hotel_poi_distance = geopy.distance.geodesic(hotel_coord, poi_coord).m
        per_bar_distance.append(hotel_poi_distance)
    greece_distances.append(per_bar_distance)

greece_hotels['nightlife_distance'] = greece_distances
greece_hotels['mean_distance'] = [statistics.mean(distance_list) for distance_list in greece_distances]
greece_hotels['median_distance'] = [statistics.median(distance_list) for distance_list in greece_distances]
greece_train = greece_hotels[["hotel_id", "city_id", "distance_to_center", "club_club_hotel", "party_people", "nightlife_distance", "mean_distance", "median_distance", "overall_rating"]]
greece_train = greece_train.fillna(0.0)
greece_train
plt.figure(figsize=(15, 15))
plt.scatter(greece_train.distance_to_center, greece_train.mean_distance)
plt.title("Greece Hotels", fontsize=18)
plt.xlabel('distance_to_center', fontsize=18)
plt.ylabel('mean_distance', fontsize=18)
plt.show()
usa_distances = []
for i in range(len(usa_hotels)):
    per_bar_distance = []
    hotel_long = usa_hotels.iloc[i][6]
    hotel_lat = usa_hotels.iloc[i][7]
    hotel_coord = (hotel_lat, hotel_long)
    for j in range(len(usa_poi)):
        poi_long = usa_poi.iloc[j][5]
        poi_lat = usa_poi.iloc[j][6]
        poi_coord = (poi_lat, poi_long)
        hotel_poi_distance = geopy.distance.geodesic(hotel_coord, poi_coord).m
        per_bar_distance.append(hotel_poi_distance)
    usa_distances.append(per_bar_distance)

usa_hotels['nightlife_distance'] = usa_distances
usa_hotels['mean_distance'] = [statistics.mean(distance_list) for distance_list in usa_distances]
usa_hotels['median_distance'] = [statistics.median(distance_list) for distance_list in usa_distances]
usa_train = usa_hotels[["hotel_id", "city_id", "distance_to_center", "club_club_hotel", "party_people", "nightlife_distance", "mean_distance", "median_distance", "overall_rating"]]
usa_train = usa_train.fillna(0.0)
usa_train
plt.figure(figsize=(15, 15))
plt.scatter(usa_train.distance_to_center, usa_train.mean_distance)
plt.title("USA Hotels", fontsize=18)
plt.xlabel('distance_to_center', fontsize=18)
plt.ylabel('mean_distance', fontsize=18)
plt.show()
neth_distances = []
for i in range(len(neth_hotels)):
    per_bar_distance = []
    hotel_long = neth_hotels.iloc[i][6]
    hotel_lat = neth_hotels.iloc[i][7]
    hotel_coord = (hotel_lat, hotel_long)
    for j in range(len(neth_poi)):
        poi_long = neth_poi.iloc[j][5]
        poi_lat = neth_poi.iloc[j][6]
        poi_coord = (poi_lat, poi_long)
        hotel_poi_distance = geopy.distance.geodesic(hotel_coord, poi_coord).m
        per_bar_distance.append(hotel_poi_distance)
    neth_distances.append(per_bar_distance)

neth_hotels['nightlife_distance'] = neth_distances
neth_hotels['mean_distance'] = [statistics.mean(distance_list) for distance_list in neth_distances]
neth_hotels['median_distance'] = [statistics.median(distance_list) for distance_list in neth_distances]
neth_train = neth_hotels[["hotel_id", "city_id", "distance_to_center", "club_club_hotel", "party_people", "nightlife_distance", "mean_distance", "median_distance", "overall_rating"]]
neth_train = neth_train.fillna(0.0)
neth_train
plt.figure(figsize=(15, 15))
plt.scatter(neth_train.distance_to_center, neth_train.mean_distance)
plt.title("Netherlands Hotels", fontsize=18)
plt.xlabel('distance_to_center', fontsize=18)
plt.ylabel('mean_distance', fontsize=18)
plt.show()<jupyter_output><empty_output><jupyter_text>__MERGE 4 CITY HOTELS DATA__  - After calculating the mean distance of each hotel in each city to the nightlife locations inside its own city area, finally I can merge all these data to put it in the process of   assigning nightlife scores. Again, by doing that, I am calculating only the distance value inside their own city, not to  another nightlife location inside another city, so that I can  give scores based on just __"distance"__ metric.<jupyter_code># CONCAT ALL 4 CITY HOTELS DATA
frames = [china_train, greece_train, usa_train, neth_train]
data = pd.concat(frames).reset_index()
del data['index']
data
plt.figure(figsize=(15, 15))
plt.scatter(data.mean_distance, data.distance_to_center)
plt.title("400 Hotels Distribution", fontsize=18)
plt.xlabel('mean_distance', fontsize=18)
plt.ylabel('distance_to_center', fontsize=18)
plt.show()<jupyter_output><empty_output><jupyter_text>I sorted the data by giving the first priority to the __"mean_distance"__ feature and the second to the __"distance_to_center"__ feature. The reason for counting distance to center when  giving score to hotels but not keeping it as the priority feature is that there is a chance if a hotel is closer to the city center it might be closer to the nightlife locations, and  this hypothesis is sort of being supported by the linear trend between those two features, but since this is an assumption I took it as a supportive feature.<jupyter_code>data_sorted = data.sort_values(["mean_distance", "distance_to_center"], ascending=[True, True])
data_sorted<jupyter_output><empty_output><jupyter_text>As I mentioned above, the correlation matrix also shows that they are correlated, but as  description of the feature__(distance_to_center)__, it does not prove that hotel would be closer to  nightlife locations if it is closer to the city center absolutely.<jupyter_code>data_to_corr = data_sorted[["distance_to_center", "mean_distance"]]
f = plt.figure(figsize=(19, 15))
plt.matshow(data_to_corr.corr(), fignum=f.number)
plt.xticks(range(data_to_corr.shape[1]), data_to_corr.columns, fontsize=14, rotation=45)
plt.yticks(range(data_to_corr.shape[1]), data_to_corr.columns, fontsize=14)
cb = plt.colorbar()
cb.ax.tick_params(labelsize=14)
plt.title('Correlation Matrix', fontsize=16);
plt.savefig('correlation_matrix.png')<jupyter_output><empty_output><jupyter_text>__HANDLING OUTLIERS__  __WINSORIZING__  To handle outliers and decrease the effect in the dataset I use __winsorizing__ method. __Winsorizing__ limits the extreme values in the dataset to reduce the effect of outliers, by setting outliers to a specified percentile.<jupyter_code>lower_bound = 0.1
upper_bound = 0.95
res = data_sorted[["distance_to_center", "mean_distance"]].quantile([lower_bound, upper_bound])
res
# WINSORIZING DATA
X = StandardScaler().fit_transform(data_sorted[["mean_distance", "distance_to_center"]])
X = winsorize(X, limits=[0.01, 0.05])
len(X)<jupyter_output><empty_output><jupyter_text>After winsorizing(handling outliers) the data points are more scattered than being extremely densed or extremely outside of the mean.<jupyter_code>plt.figure(figsize=(15, 15))
plt.scatter(X[:, 0], X[:, 1])
plt.title("400 Hotels Distribution", fontsize=18)
plt.xlabel('mean_distance', fontsize=18)
plt.ylabel('distance_to_center', fontsize=18)
plt.show()<jupyter_output><empty_output><jupyter_text>__CLUSTERING__  - Elbow method is being used to determine the ideal number of clusters. This method tries number of cluster amounts to test to see until which point the sum of squared distances are  being decreased significantly. Here, usually, the number that is on the elbow of the curve is being selected. - From this plot I can see that elbow of the curve shows around k=40 (k-> number of k-means clusters)<jupyter_code># ELBOW METHOD FOR #CLUSTERS
sum_of_squared_distances = []
K = range(10,100)
for k in K:
    k_means = KMeans(n_clusters=k)
    model = k_means.fit(X)
    sum_of_squared_distances.append(k_means.inertia_)

plt.figure(figsize=(15,15))
plt.plot(K, sum_of_squared_distances, 'bx-')
plt.xlabel('k')
plt.ylabel('sum_of_squared_distances')
plt.title('elbow method for optimal k')
plt.show()<jupyter_output><empty_output><jupyter_text>From elbow method I can see that elbow of the curve is around 40. Although, I chose 100 clusters to divide hotels for assigning scores from 1 to 100 according to their mean distance to  nightlife locations and distance to city center. In addition to that, I use __sillhouette score__ and __calinski harabazs score__ to validate the number of clusters which gives higher  score for 100 clusters. The reason is that the data is being spreaded/scattered enough to cluster them into 100.<jupyter_code># K-MEANS CLUSTERING
k_means = KMeans(n_clusters=40)
model = k_means.fit(X)

y_hat = k_means.predict(X)
labels = k_means.labels_
sill_score = metrics.silhouette_score(X, labels, metric='euclidean')
cal_score = metrics.calinski_harabasz_score(X, labels)
print("Sillhouette Score: ", sill_score)
print("Calinski Score: ", cal_score)

plt.figure(figsize=(15, 15))
plt.scatter(X[:, 0], X[:, 1], s=1)
plt.scatter(k_means.cluster_centers_[:,0] ,k_means.cluster_centers_[:,1], color='black', s=5)
plt.xlabel("mean_distance", fontsize=18)
plt.ylabel("distance_to_center", fontsize=18)
plt.show()
# K-MEANS CLUSTERING
k_means = KMeans(n_clusters=100)
model = k_means.fit(X)

y_hat = k_means.predict(X)
labels = k_means.labels_
sill_score = metrics.silhouette_score(X, labels, metric='euclidean')
cal_score = metrics.calinski_harabasz_score(X, labels)
print("Sillhouette Score: ", sill_score)
print("Calinski Score: ", cal_score)

plt.figure(figsize=(15, 15))
plt.scatter(X[:, 0], X[:, 1], s=1)
plt.scatter(k_means.cluster_centers_[:,0] ,k_means.cluster_centers_[:,1], color='black', s=5)
plt.xlabel("mean_distance", fontsize=18)
plt.ylabel("distance_to_center", fontsize=18)
plt.show()<jupyter_output>Sillhouette Score:  0.47214150099452956
Calinski Score:  10535.495935993727<jupyter_text>Now that I have 100 clusters for the data points, I will calculate the magnitude of each data point to its own assigned cluster's centroid and sort them in descending order because the  lower the distance a hotel has to each feature(left corner of the plot) the higher score it will take. By this I made a way of distinction of assigining different scores amongst the  hotels.<jupyter_code>def sorted_cluster(x, model=None):
    if model == None:
        model = KMeans()
    model = sorted_cluster_centers_(model, x)
    model = sorted_labels_(model, x)
    return model

def sorted_cluster_centers_(model, x):
    new_centroids = []
    magnitude = []
    for center in model.cluster_centers_:
        magnitude.append(np.sqrt(center.dot(center)))
    idx_argsort = np.argsort(magnitude)
    model.cluster_centers_ = model.cluster_centers_[idx_argsort]
    return model

def sorted_labels_(sorted_model, x):
    sorted_model.labels_ = sorted_model.predict(x)
    return sorted_model
k_means = sorted_cluster(X, k_means)
print(k_means.cluster_centers_)
print([score+1 for score in k_means.labels_])
data_sorted["score"] = [score+1 for score in k_means.labels_]
data_sorted
final_data = data_sorted[["hotel_id", "city_id", "score"]]
final_data
final_data.to_csv('result_data.csv')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup