# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Part 1: Data Wrangling

# ## Introduction
#
# This project is a self-made end to end machine learning project in which I scrape a website called 'Jendela 360'. The scraped dataset is saved in a csv file named 'Apartment Data Raw'. The dataset contains the details of apartment units available to be rented in Jakarta and its surrouding (Jabodetabek region) on October 18, 2020. The data discussed here might not be up-to-date. 
#
# Problem Statement of this project:
# "Based on the scraped data of apartments in Jakarta and its surrounding, the writer aims to construct a machine learning model to predict the annual rent price of apartment units. If possible, the writer aims to find which feature/factors has the most immpact on an apartment unit's annual rent price."
#
# In the first notebook, we are going to load the raw dataset and conduct data wrangling to draw insights and clean the data. Our goal is to have a cleaned dataset at the end of this notebook, so we can use the cleaned data to create and test regression models in the second notebook.
#
# Last but not least, this project is non-profit and made for learning purposes only.

# ## Importing Packages

# +
# Essentials
import numpy as np
import pandas as pd
import datetime
import random

# Plots
import matplotlib.pyplot as plt
import seaborn as sns
# %matplotlib inline

# Models
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, AdaBoostRegressor, BaggingRegressor
from sklearn.kernel_ridge import KernelRidge
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.linear_model import ElasticNet, ElasticNetCV
from sklearn.svm import SVR
from mlxtend.regressor import StackingCVRegressor
import lightgbm as lgb
from lightgbm import LGBMRegressor
from xgboost import XGBRegressor

# Stats
from scipy.stats import skew, norm
from scipy.special import boxcox1p
from scipy.stats import boxcox_normmax

# Misc
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold, cross_val_score
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import scale
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import RobustScaler
from sklearn.decomposition import PCA
# -

# ## Importing Dataset

# +
raw_df = pd.read_csv('Web Scraping/Apartment Dataset Raw.csv')
#We are going to save the unaltered dataset as raw_df, and use the dataframe 'df' to do the next data wrangling operations

df = raw_df
df.head()
# -

df = df.rename({'Unnamed: 0' : 'Index'}, axis = 'columns')

# ## Data Cleaning

# ### Raw Data Shape and Column Description

print(df.columns)

df.shape

# Each row represents a unique unit of apartment which was displayed on Jendela 360 for rent on 18th October 2020. We have 5339 rows and 47 columns. The columns represent various characteristics of each unit, and is described as follows.
#
# The following columns describe the identification data of each unit (location, name, etc).
# * Index: the index of each row (self-ecplanatory) starting at 0.
# * URL: the URL each apartment unit page on Jendela 360 website.
# * Unit_Name: the apartment unit name on its page.
# * Unit_ID: the ID of each page (the last seven characters of the URL). Unique for each apartment unit.
# * Apt_Name: the apartment building name of the unit. 
# * Street: the street address of the unit.
# * Locality: the local district of the unit.
# * Region: the city of the unit.
# * Longitude and Latitude: the geographical longitude and latitude coordinate of the unit
# * Floor: the floor location of the unit.
# * Tower: the name of the tower in which the unit is located in.
#
# The following columns describe the facilities of each apartment unit. The two columns which houses numerical (quantitative) data about each apartment unit's facilities are:
# * No_Rooms: the number of bedrooms in each apartment unit.
# * Area: the area in meter suqared of each apartment unit.
#
# The other columns which describe the facilities of each unit are categorical in nature. The value of each column is '1' if the facility is present, and '0' if the facility is not present. These columns are:
# * Furnished (1 represents that the unit is fully furnished, and vice versa)
# * AC
# * Water_Heater
# * Dining_Set
# * Electricity 
# * Bed
# * Access_Card
# * Kitchen
# * Fridge
# * Washing_Machine
# * TV
# * ATM
# * TV_Cable
# * Grocery 
# * Internet
# * Swim_Pool (swimming pool)
# * Laundry
# * Security
# * Basketball (basketball field)
# * Multipurpose_room
# * Gym
# * Jogging (jogging track)
# * Tennis (tennis field)
# * Restaurant
# * Playground
#
# The following columns describe the fee of each unit. The only fee that each apartment has is the annual rent price. Not all apartment units are available to be rented on a monthly term. There are also cases where the deposit and service charges are not listed. Furthermore, it will be very easy to predict the annual price if we know the monthly price, as we just need to multiply it by 12. That's why we are going to remove every fee column in the dataset and only take the annual rent price (in rupiah) as the dependent variable of our model.
#
# * Currency: the currency unit of the listed price.
# * Monthly_Price: the monthly payment fee if the tenant wishes to rent it on monthly term.
# * Annual_Price: the annual payment fee if the tenant wishes to rent it on yearly term.
# * Deposit_Currency: the currency unit of the listed deposit charge.
# * Deposit_Charge: the initial deposit charge.
# * Service_Currency: the currency unit of the service charge.
# * Service_Charge: the service charge of the unit.

# ### Omiting ERROR Rows
#
# The web scraper uses a ```try:...except:``` block to keep on reading and scraping new pages even if the current iteration raises an error. This is done so the scraping process could be automated, and if a web page raises an error, we don't have to restart the scraping process from the beginning again. If a page raises an error, the whole row (except the URL) will be filled with the string 'ERROR'. The best way to find 'ERROR' rows is to find which rows that have an 'ERROR' Apt_Name column, as that is the features that exists in all apartment unit web pages.
#
# In this step, we are going to remove all 'ERROR' rows.

df.shape

df = df[df.Apt_Name != 'ERROR']

df = df.reset_index(drop = True, inplace=False)
df.shape

# We can see that there are 18 rows which are omitted. These rows are the 'ERROR' rows.

# ### Identifying the Dependent/Outcome Variable

# Referring to the initial problem statement of this project, we hereby decide that the annual rent price of the apartment will be our dependent variable for the regression model. Furthermore, we should not look at the values from monthly price, deposit charge, and service charge, as we would like to predict the annual rent price only using the apartment unit's identification data (location) and facilities. 
#
# After deciding which variable will be our outcome variable, we should make sure that the annual price data is in the same currency unit. If the currency of the annual rent price is in dollars, we have to convert it to Rupiah.
#
# The assumption used is that 1 USD = 14,700 IDR.

df.Currency.value_counts()

# We see that there are 5200 apartment unit rent prices which are listed in Rupiah, 57 prices which are listed in US Dollars. We need to convert the price of these 57 apartment units from USD to IDR. To convert it, we need to multiply the Annual_Price value by 14700 if the value of Currency equals to 'USD'. However, before doing any of that, we need to make sure that the values in Annual_Price columns are read as numbers by pandas. 

df.Annual_Price

# As we can see, the 'Annual_Price' has the data type of object. This means we need to convert it to float first, before multiplying it by 14700 if it is in USD to convert it properly. 

# +
Rupiah_Annual_Price = list()
currency_changed = 0
for i, price in enumerate(df.Annual_Price):
    if df.Currency[i] == 'USD':
        Rupiah_Annual_Price.append(float(price)*14700)
        currency_changed += 1
    else:
        Rupiah_Annual_Price.append(float(price))
        
df['Rupiah_Annual_Price'] = Rupiah_Annual_Price
print(currency_changed)
# -

# The currency_changed counter is used to tell us how many currency conversion has been done, and we are glad to see that there are 57 currency conversions, which is the same number of 'USD' occurences in the 'Currency' column of our dataset.
#
# Next, we are going to remove the columns which are no longer needed ('Currency', 'Annual_Price' 'Monthly_Price', 'Deposit_Currency', 'Deposit_Charge', 'Service_Currency', 'Service_Charge'). 
#
# We are then renaming the 'Rupiah_Annual_Price' to 'AnnualPrice'.

df = df.drop(['Currency', 'Annual_Price', 'Monthly_Price', 'Deposit_Currency', 'Deposit_Charge', 'Service_Currency', 
              'Service_Charge'], axis = 'columns')
df = df.rename({'Rupiah_Annual_Price':'AnnualPrice'}, axis = 'columns')

df.head()

# ## Exploratory Data Analysis

df.columns

# In this step, we are going to do some data exploration to gain more insights on our dataset. First, we'll drop columns which we think might not be insightful for our model. We'll drop the 'Street' and 'Tower' column as it's quite difficult to parse and does not supply us with any insightful information. The 'Street' column is irrelevant as we have 'Locality', 'Region', as well as 'Longitude' & 'Latitude' column to draw geospatial insights form. The 'Tower' column is dropped because it's the name of the tower of each unit, and each apartment complex has different tower names. We suspect that the 'Unit_Name' and 'Apt_Name' might be dropped too, but we'll inspect them in a little bit to see if there are any insights we can draw from those columns. 
#
# Note: We'll keep the 'URL' and 'Unit_ID' until we finish exploring the data in case we want to check on specific apartment units.

df = df.drop(['Street', 'Tower'], axis = 'columns')

# Next, we are going to inspect the 'Unit_Name' and 'Apt_Name' columns.

df[['Apt_Name', 'Unit_Name']].head()

# It seems that the 'Apt_Name' column just indicates the overall name of our Apartment complex, while the 'Unit_Name' mentions the number of bedrooms, and in some case, the furnished status of the apartment. Interestingly, the furnished status in 'Unit_Name' are divided into three levels: 'Non Furnished', 'Semi Furnished', and 'Fully Furnished'. However, in our 'Furnished' column, there are only two levels: 'Non Furnished' and 'Fully Furnished'.
#
# We can add a new level to our 'Furnished' feature by creating a 'Semi Furnished' level if the 'Unit_Name' of a particular row has the word 'semi' in it. We'll create a new column called 'FurnishedNew' for this feature.

FurnishedNew = list()
for i in range(len(df['Index'])):
    if df.Furnished[i] == '1':
        FurnishedNew.append('Full')
    elif df.Furnished[i] == '0':
        if 'semi' in df.Unit_Name[i].lower():
            FurnishedNew.append('Semi')
        else:
            FurnishedNew.append('Non')
df['FurnishedNew'] = FurnishedNew

df.FurnishedNew.value_counts()

# We'll see if this new feature is better than the existing 'Furnished' column. If this feature makes the model worse, then we'll simply use the two level 'Furnished' feature. We'll then drop the 'Apt_Name' and 'Unit_Name' column.

df = df.drop(['Unit_Name', 'Apt_Name'], axis = 'columns')

df.head()

# Next, we are going to analyse each column and see if it is a good feature for our model or not. We will be plotting each feature against the predicted value, the 'AnnualPrice'. While there are other ways to perform feature selection which are relatively more automated, the writer chooses to do this to gain more insights personally on the dataset.

# #### Number of Bedrooms

bedroom_df = df[['URL','No_Rooms', 'AnnualPrice']]
bedroom_df.No_Rooms.value_counts()

# The apartment units in our dataset have 0 till 6 'number of bedrooms'. What does '0' number of bedroom means? During the scraping process, the writer discover that studio apartment units are written as having '0' number of bedrooms in the ```.json``` schema of the web page. We can then use ```df.groupby``` to see the average annual rent price of each category.

avg_no_rooms = bedroom_df.groupby('No_Rooms')['AnnualPrice'].mean().reset_index().rename({'AnnualPrice':'Average Annual Price'}, axis = 'columns')
print(avg_no_rooms)
avg_no_rooms.plot(x = 'No_Rooms', y = 'Average Annual Price', kind = 'bar', figsize = [5,5])

# First column and we're already greeted with a surprise. Why is the studio apartment unit's average price higher than the average price of apartment units with 6 bedrooms? This is why exploring our dataset manually, or the way I prefer to say it - 'personally', is important. This data does not match our common sense, and we need to investigate it. The first thing to do in this situation is try to check for outliers.

studio_check = bedroom_df['No_Rooms'] == '0'
sns.boxplot(x = bedroom_df[studio_check].AnnualPrice)

# First, we filter the 'AnnualPrice' column by 'No_Rooms' category. After selecting the annual rent prices of apartment units which are studio-typed, we can draw the boxplot using seaborn and we see there are two outliers. Let's check these out.

bedroom_df[studio_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

# After sorted by Annual Price, the top two apartment units have prices that are clearly beyond what's 'the norm' for studio apartment units. Using ```pd.set_option('display.max_colwidth', None)```, we can get the URL for these two apartment units, and then see for ourselves in their respective page.

pd.set_option('display.max_colwidth', None)
bedroom_df[studio_check].sort_values(by=['AnnualPrice'], ascending=False).head(2).URL

# ![Website](54_million_studio.PNG "Upon opening the first link, we see a fifty four million dollars studio apartment")

# Upon looking at the first link, we see that this 25 meter squared, studio apartment, is priced at fifty four million dollars. I think we can see the problem here. There are a few pages in which the currency used is wrong. Even apartment with 6 bedrooms are not priced fifty four million dollars a year. This unit's price should be fifty four million rupiah. 
#
# The second unit in question also shares the problem. This time, the studio apartment is priced at thirty million dollars. We first need to clean this mess before we continue exploring the other columns. Let's also check if other number of bedrooms share the same issue.

br2_check = bedroom_df['No_Rooms'] == '2'
sns.boxplot(x = bedroom_df[br2_check].AnnualPrice)

bedroom_df[br2_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

# Turns out the problem isn't unique to studio apartments. We have to solve this issue first then, and unfortunately this can only be done in a relatively manual manner (checking the URL one by one). I'll get back after resolving this issue.

# #### Finding and Fixing Outliers based on Number of Bedrooms

# Create boolean identifiers

studio_check = bedroom_df['No_Rooms'] == '0'
br1_check = bedroom_df['No_Rooms'] == '1'
br2_check = bedroom_df['No_Rooms'] == '2'
br3_check = bedroom_df['No_Rooms'] == '3'
br4_check = bedroom_df['No_Rooms'] == '4'
br5_check = bedroom_df['No_Rooms'] == '5'
br6_check = bedroom_df['No_Rooms'] == '6'

# Fix for No_Rooms = '0' (Studio-Type)

bedroom_df[studio_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

df.loc[df.Unit_ID == 'sgpa014', 'AnnualPrice'] = 54000000

df.loc[df.Unit_ID == 'pgva007', 'AnnualPrice'] = 30000000

# Fix for No_Rooms = '1' (One Bedroom)

bedroom_df[br1_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

sns.boxplot(x = bedroom_df[br1_check].AnnualPrice)

# I think the rent price for 1 bedroom appartment units are skewed to the right. None of the five highest apartment units (of one bedroom) have annual rent prices displayed in dollars. However, we're going to remove one point which is the highest priced apartment unit as it's quite far from the rest of the data points. 

i = df[((df.Unit_ID == 'frrb001'))].index
df = df.drop(i)

# Fix for 'No_Rooms' = '2' (Two Bedrooms)

bedroom_df[br2_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

sns.boxplot(x = bedroom_df[br2_check].AnnualPrice)

df.loc[df.Unit_ID == 'blmc009', 'AnnualPrice'] = 50400000

# Fix for 'No_Rooms' = '3' (Three Bedrooms)

bedroom_df[br3_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

sns.boxplot(x = bedroom_df[br3_check].AnnualPrice)

# It turns out that the highest bedroom price is still in Rupiah. However, the rightmost data point is considerably far away from the rest of the data points, and we'll consider it as an outlier to be removed.

i = df[((df.Unit_ID == 'esdd002'))].index
df = df.drop(i)

# Fix for 'No_Rooms' = '4' (Four Bedrooms)

bedroom_df[br4_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

sns.boxplot(x = bedroom_df[br4_check].AnnualPrice)

# Although there seems to be two outliers, upon further checking they don't seem to be a case of misused currency. However, those two rightmost points are considerably far away from the rest of the other data points, and we'll consider them as outliers to be removed. These two prices are even higher than apartment units with 6 bedrooms, and do not represent the norm. 

i = df[((df.Unit_ID == 'pkrf001'))].index
df = df.drop(i)
i = df[((df.Unit_ID == 'ppre001'))].index
df = df.drop(i)

# Fix for 'No_Rooms' = '5' (Five Bedrooms)

bedroom_df[br5_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

# There are only two apartment units in our dataset which has five bedrooms. We are not going to remove anything for now.

# Fix for 'No_Rooms' = '6' (Six Bedrooms)

bedroom_df[br6_check].sort_values(by=['AnnualPrice'], ascending=False).head(5)

# There is only one aaprtment unit with six bedrooms. We are not going to remove anything for now - however, we might combine the units with 4, 5, and 6 into one category.

br456_check = (bedroom_df.No_Rooms== '4') | (bedroom_df.No_Rooms == '5') | (bedroom_df.No_Rooms == '6')

sns.boxplot(x = bedroom_df[br456_check].AnnualPrice)

# #### Checking on the Updated Dataframe for No_Rooms Feature

# +
New_No_Rooms = list()
for i, br_no in enumerate(df.No_Rooms):
    br_float = int(br_no)
    if br_float >= 4:
        New_No_Rooms.append(4)
    else:
        New_No_Rooms.append(br_float)
        
df.drop(['No_Rooms'], axis = 'columns')
df['No_Rooms'] = New_No_Rooms
# -

bedroom_df_updated = df[['URL','No_Rooms', 'AnnualPrice']]
avg_no_rooms = bedroom_df_updated.groupby('No_Rooms')['AnnualPrice'].mean().reset_index().rename({'AnnualPrice':'Average Annual Price'}, axis = 'columns')
print(avg_no_rooms)
avg_no_rooms.plot(x = 'No_Rooms', y = 'Average Annual Price', kind = 'bar', figsize = [5,5])

sns.boxplot(x = "No_Rooms", y = 'AnnualPrice', data = df)

# There we go. Now it made sense - the more number of bedrooms an apartment unit has, the higher the annual rent price. However, there no apartment units which are priced way above the other units in the same category. Through evaluating outliers and checking on the source data, we have 'cleaned' the 'No_Rooms' feature for now. 
#
# The last step taken for this feature column is grouping the categories of '4', '5', and '6'. There are only 3 units out of our more than 5000 rows which have 5 and 6 bedrooms, and that is not quite representative. 
#
# Now, we might ask, why the new category (of units with 4 and more bedrooms) are given the value '4'? Shouldn't it be '4 and more'? 
#
# Yes. It represents the number of bedrooms of 4 and more. However, this categorical variable will be treated as ordinal variable in the machine learning model. That's why we have to keep the values as integers. We'll just have to keep in our mind later when writing the final report, that the number '4' in No_Rooms feature not only represents units with 4 bedrooms, but also units with more than 4 bedrooms. 

# #### Analyzing Location Feature Columns

# The next part of our features to be discussed are the columns which describe where our unit is on the map. There are four columns being discussed here - two which are categorical ('Locality' and 'Region'), as well as two continuous columns ('Longitude' and 'Latitude'). First let's look at the 'Region' columns.

df.Region.value_counts()

# Whoa. Turns out the scraped pages also includes apartment units from outside Jakarta and its surroundings. To stay true to our problem statement, we'll remove regions outside 'Jabodetabek'. 

df = df[(df.Region == 'Jakarta Selatan') | (df.Region == 'Jakarta Barat') | (df.Region == 'Jakarta Pusat') | (df.Region == 'Jakarta Timur') | (df.Region == 'Jakarta Utara') | (df.Region == 'Tangerang') | (df.Region == 'Bekasi') | (df.Region == 'Depok') | (df.Region == 'Bogor')]

# Let's visualize the data using a boxplot again. Now, we're investigating if differences in regions affect annual rent price.

dims = (12,8)
fig, ax = plt.subplots(figsize=dims)
sns.boxplot(x = "Region", y = 'AnnualPrice', data = df, ax=ax)

JakBar = df['Region'] == 'Jakarta Barat'
df[JakBar][['URL', 'AnnualPrice']].sort_values(by = ['AnnualPrice'], ascending=False).head(1)

# From the visualization, we can see that the region in DKI Jakarta with the highest average annual rent price is 'Jakarta Selatan', followed by 'Jakarta Pusat', 'Jakarta Barat', 'Jakarta Utara', and 'Jakarta Timur' consecutively. Regions outside Jakarta have lower average prices than regions inside Jakarta. This distribution makes sense, as it is quite a common knowledge for Jakartans to know that the region with the highest property price in Jakarta is 'Jakarta Selatan'. 
#
# There seems to be an outlier in 'Jakarta Barat', but upon further checking - it's the only unit with 6 bedrooms, so the price reflects more of its number of rooms than its region. We will not remove this data point for now.

# There are a few options on how we are going to use the locations columns in our model:
#
# Option 1: Uses one hot encoding on Region. This seems to be the go-to-solution if we wishes to make location a categorical variable. We'll divide the area into six major Regions - West, North, South, East, Center Jakarta, and outside Jakarta (we group Bogor, Depok, Tangerang, and Bekasi into one Region).
#
# Option 2: Uses one hot encoding on Locality. There are over 90 different local districts in this data set, and one hot encoding would mean that we'll have 90+ extra feature columns of zeros and ones. Furthermore, a lot of these local districts have only one apartment unit.
#
# Option 3: Uses the 'Longitude' and 'Latitude' column as continuous variables. This could be the case if we notice a pattern on the longitude and latitude data. We could also do clustering algorithm on longitude and latitude data. 
#
# We'll look into the 'Longitude' and 'Latitude' columns first.

print(df.Longitude.dtype)
print(df.Latitude.dtype)

# It seems that these two columns are classified as 'object' and not 'float' by Pandas. We need to transform them first.

# +
df = df.reset_index(drop = True, inplace=False)
Longitude_Float = list()
Latitude_Float = list()

for i in range(len(df.Index)):
    Longitude_Float.append(float(df.Longitude[i]))
    Latitude_Float.append(float(df.Latitude[i]))

df.drop(['Longitude', 'Latitude'], axis = 'columns')
df['Longitude'] = Longitude_Float
df['Latitude'] = Latitude_Float
# -

df.Longitude.plot()

# After converting both columns to float, let's visualize each column to analyze if there are any outliers. As the geographical location chosen for this project is quite close to each other, there shouldn't be any outliers. The 'Longitude' dataset makes sense: all our apartment units have Longitude between 106.6 until 107.2.

df.Latitude.plot()

# The 'Latitude' feature column, however, seems to have yet another issue related to an error in data entering. Most of the apartment units have Latitude around -6, which makes sense, as Jakarta (and its surrounding) are located slightly beneath the Equator. Howver, there are a few data points which have latitude of 6. This is suspicious as it could very well be a case of forgetting to add '-' (the negative sign) during data entry process for these apartment units. For now, let's assume this to be the case, and put a negative value on the latitude feature of these apartment units.

Latitude_fixed = [la if la<0 else -1*la for la in df.Latitude]
df.drop(['Latitude'], axis = 'columns')
df['Latitude'] = Latitude_fixed

df.Latitude.plot()

# This distribution made more sense as the value of 'Latitude' ranges from -6.6 to -6.1, not a big margin, and the three data points with the lowest 'Latitude' seems to be apartment units outside Jakarta (maybe in Bogor/Depok).

# #### Analyzing Furnished Status Feature Column
#
# Now, let's visualize and take a look at the two columns describing the furnished status of each apartment unit - the original 'Furnished', and our newly created 'FurnishedNew'. 

fig, (ax1, ax2) = plt.subplots(1, 2)
sns.scatterplot(x = "Furnished", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax1)
sns.scatterplot(x = "FurnishedNew", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax2)
fig.tight_layout()

# There are two takeaways from this: first, the discrepancy between non-furnished and fully furnished apartment units' prices doesn't seem to be that big. Second, our new column, 'FeatureNew', shows that semi-furnished apartments have lower prices compared to non-furnished ones. 
#
# What should we make of this? Our new feature column doesn't seem to work well - this might be because not all apartment units which are semi-furnished write that they are 'semi-furnished' in their page name. The population of 'semi-furnished' apartments may be much more than what was being labeled as 'Semi'. This explains two things: why adding an extra category doesn't work well, and why the discrepancy between '0' and '1' is not that far away from each other.
#
# This could indicate that 'Furnished' is not a good predictor for AnnualPrice, but we'll decide it later in the next feature engineering section.

# #### Analyzing Floor Position of Apartment Units
#
# The feature column we're looking at this section is the 'Floor' column. We'll see if there are differences in annual rent price between units with different floor positions.

sns.boxplot(x = "Floor", y = 'AnnualPrice', data = df)

# Not only the discrepancy among all floor locations seem to be miniscule, we also have quite a few apartment units with no labels of their floor location. For now, let's not use this categorical variable in our model.

df = df.drop(['Floor'], axis = 'columns')
df = df.reset_index(drop = True, inplace=False)

# #### Analyzing Area of Units to AnnualPrice

# +
Area_Float = list()

for i in range(len(df.Index)):
    Area_Float.append(float(df.Area[i]))

df.drop(['Area'], axis = 'columns')
df['Area'] = Area_Float
# -

dims = (8,5)
fig, ax = plt.subplots(figsize=dims)
sns.scatterplot(x = "Area", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax)

# Based on the above plot, we can see that the general trend is that AnnualPrice increases as Area increases. We also see that as number of bedrooms increases, area also increases. However, there are a few data points which are scattered far from the others that we need to investigate. They could be outliers and we should remove them.

df[['URL', 'Area']].sort_values(by=['Area'], ascending = False).head(12)

# There are six apartment units with areas above 500 meter squared. That's a huge apartment unit - two of them even reaches more than seven thousand meter squared. These units are not what in most people's mind when they're looking to rent an apartment unit - as these units come in the form of condominium or penthouse. We'll be removing these six units from our data set. In the deployment stage of this machine learning model, we'll limit the maximum Area to be 350 meter squared, as that is already a very big apartment unit.

# +
i = df[((df.Unit_ID == 'tacc001'))].index
df = df.drop(i)

i = df[((df.Unit_ID == 'tacc002'))].index
df = df.drop(i)

i = df[((df.Unit_ID == 'kmvd027'))].index
df = df.drop(i)

i = df[((df.Unit_ID == 'mqrd023'))].index
df = df.drop(i)

i = df[((df.Unit_ID == 'csbe001'))].index
df = df.drop(i)

i = df[((df.Unit_ID == 'stme008'))].index
df = df.drop(i)
# -

df = df.reset_index(drop = True, inplace=False)
df.shape

dims = (8,5)
fig, ax = plt.subplots(figsize=dims)
sns.scatterplot(x = "Area", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax)

# This visualization made more sense as there are no far outliers. However, something we notice is that there are some apartment units which are listed as having 0 Area. We'll simply remove these units as it's impossible for apartment units to have 0 meter squared of area. We'll consider than 20 meter squared is the minimum apartment unit area. 

df = df[df['Area']>20]

df.columns

# #### Checking Categorical Facility Features
#
# Our last sets of features are the facilities that each unit has. During the web scraping process, I added a column in which it counts how many of these features that the unit has, and store them in a column called 'Total_Facilities'. Let's first take a look at this column, before diving into other facilities one-by-one.

# +
Facilities_Int = list()
for i, count in enumerate(df.Total_Facilities):
    Facilities_Int.append(int(count))

df.drop(['Total_Facilities'], axis = 'columns')
df['Total_Facilities'] = Facilities_Int

sns.boxplot(x="Total_Facilities", data = df)
# -

sns.scatterplot(x="Total_Facilities", y = "AnnualPrice", data = df)

# It seems that most apartment units have at least 10 facilities. The more facilities a unit has, the higher its rend price is. Let's take a look at the units which has features less than 10, and see if they actually have less than 10 features, or there is some errors here.

df[['URL', 'Total_Facilities', 'AnnualPrice', 'Furnished']].sort_values(by = ['Total_Facilities'], ascending = True).head(10)

# The apartment units with low Total_Facilities tend to be Non-Furnished units. However, there's an oddball here - the unit with 0 'Total_Facilities' is a fully-furnished unit! Upon further investigation, based on the photos of the room, there are indeed facilities and it might be some errors in inputing the data (or the unit owner/seller does not describe the facilities fully). We are going to remove that unit from our dataset. As for the other fully-furnished unit with only 3 total facilities, the page and pictures show that it is indeed quite a blank unit. There are beds and sofas - but there is no fancy facilities like TV or Internet. 

i = df[((df.Unit_ID == 'spsa001'))].index
df = df.drop(i)
df = df.reset_index(drop = True, inplace=False)

# Next, we are going to draw boxplots of each facilities. To recall, if a facility is present in a unit, it will has value '1', if not, it will have the value of '0'. We would like to see if the presence of these facilities impact the annual rent price of apartment units. We'll remove facilities whose existence (or inexistence) does not impact the annual rent price. 

fig, (ax1, ax2) = plt.subplots(1, 2)
sns.scatterplot(x = "Furnished", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax1)
sns.scatterplot(x = "FurnishedNew", y = 'AnnualPrice', data = df, hue = 'No_Rooms', ax=ax2)
fig.tight_layout()

fig, ((ax1, ax2, ax3, ax4), (ax5, ax6, ax7, ax8),
     (ax9, ax10, ax11, ax12), (ax13, ax14, ax15, ax16),
     (ax17, ax18, ax19, ax20), (ax21, ax22, ax23, ax24)) = plt.subplots(6, 4, figsize = (15,25))
fig.suptitle('Facilities and AnnualPrice Visualization')
for i, ax in enumerate(fig.get_axes()):
    column = df.columns[i+10]
    sns.boxplot(x = column, y = 'AnnualPrice', data = df, ax = ax)

# Based on the visualization above, for each facilities, the trend is clear - the presence of facilities affects the unit annual rent price positively. This proves to be quite troublesome when we want to do feature selection - we don't know which facility is less important than the other. We'll keep most facilities for the most part, but we'll reomve two of them right away - 'Electricity' and 'Access Card'. Why? Because most apartment units have them - it's not a 'facility' anymore - it is a necessity. There are 300-400 apartments which are listed as having no 'Electricity', but it doesn't really make sense. We do this because we are thinking about the deployment phase of our model. Our future users won't choose to have an apartment unit without 'Electricity' or 'Access Card'.
#
# This concludes our first part. To recap, we have:
# - removed uninsightful columns
# - checked and removed outliers
# - fixed abnormal data (latitude and misused currency)
# - visualize features
#
# We also now have a rough understanding on the annual rent price of apartment units in Jakarta: the most expensive apartments are usually found at Jakarta Selatan - and the more area a unit occupies, the more bedrooms & facilities it has, the higher its annual rent price is. 
#
# In the next part, we are going to:
# - scale numerical features
# - split the dataset into testing and training set
# - create and evaluate baseline model
# - conduct feature engineering based on the feedback gained on baseline model
# - test new models and decide which model is the best

df = df.drop(['Electricity', 'Access_Card'], axis = 'columns')

df.to_csv('Cleaned Apartment Data.csv')









StarcoderdataJupyterScriptsDedupFiltered