<jupyter_start><jupyter_text>1) Data Clean First of all, we want to drop thoese useless columns and to check whether there are any duplicate values in our dataset. If there is any, we will need to drop them.<jupyter_code>%%time
# "Unname: 0" column looks like useless, drop it.
del df['Unnamed: 0']
#"EASE-MENT" column is empty, we are going to drop it.
del df['EASE-MENT']

del df['SALE DATE']
sum(df.duplicated(df.columns))
df = df.drop_duplicates(df.columns, keep='last')
df.shape<jupyter_output><empty_output><jupyter_text>Now, Let's look into details.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 83190 entries, 0 to 84547
Data columns (total 19 columns):
 #   Column                          Non-Null Count  Dtype 
---  ------                          --------------  ----- 
 0   BOROUGH                         83190 non-null  int64 
 1   NEIGHBORHOOD                    83190 non-null  object
 2   BUILDING CLASS CATEGORY         83190 non-null  object
 3   TAX CLASS AT PRESENT            83190 non-null  object
 4   BLOCK                           83190 non-null  int64 
 5   LOT                             83190 non-null  int64 
 6   BUILDING CLASS AT PRESENT       83190 non-null  object
 7   ADDRESS                         83190 non-null  object
 8   APARTMENT NUMBER                83190 non-null  object
 9   ZIP CODE                        83190 non-null  int64 
 10  RESIDENTIAL UNITS               83190 non-null  int64 
 11  COMMERCIAL UNITS                83190 non-null  int64 
 12  TOTAL UNITS                     83190 non-null[...]<jupyter_text>After browsing the information of dataset, we can see they use "-" to represent the missing values. Before prediction, we should clean these rows in the data table. Some of them should not be counted as the sale of real estate. For example, in column "SALE PRICE", these properties with 0 value might be transferred as a gift. Before cleaning missing values, We need to convert some of the columns(for example, SALE PRICE is object, SALE DATE is object, etc) to appropriate datatype.<jupyter_code>df['SALE PRICE'] = pd.to_numeric(df['SALE PRICE'], errors='coerce')
df['LAND SQUARE FEET'] = pd.to_numeric(df['LAND SQUARE FEET'], errors='coerce')
df['GROSS SQUARE FEET']= pd.to_numeric(df['GROSS SQUARE FEET'], errors='coerce')
df['TAX CLASS AT PRESENT'] = df['TAX CLASS AT PRESENT'].astype('category')
df['TAX CLASS AT PRESENT'] = df['TAX CLASS AT PRESENT'].astype('category')
df['BOROUGH'] = df['BOROUGH'].astype('category')

df.shape

df.info()
variables = df.columns

data = []
for variable in variables:
  l = df[variable].count()
  data.append(l)
available_per = np.round(pd.Series(data)/len(df), 3)
plt.figure(figsize=(8,6))
plt.barh(variables, available_per)
plt.title("Percent of available data", fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>As shown in the figure, SALE PRICE, GROSS SQUARE FEET and LAND SQUARE FEET have the lowest percent of available data. But again, SALE PRICE is the value we wanted to predict, and as the data set description states:*Many sales occur with a nonsensically small dollar amount: $0 most commonly. These sales are actually transfers of deeds between parties: for example, parents transferring ownership to their home to a child after moving out for retirement.* We may want to drop those rows.For SQUARE FEET, in this case, we will use mean values to fill them up.<jupyter_code>df=df[df['SALE PRICE']!=0]
df['LAND SQUARE FEET']=df['LAND SQUARE FEET'].fillna(df['LAND SQUARE FEET'].mean())
df['GROSS SQUARE FEET']=df['GROSS SQUARE FEET'].fillna(df['GROSS SQUARE FEET'].mean())
df.shape
# Splitting dataset 
test=df[df['SALE PRICE'].isna()]
sale_house=df[~df['SALE PRICE'].isna()]
test = test.drop(columns='SALE PRICE')
print(test.shape)
test.head()<jupyter_output>(13909, 18)<jupyter_text>Now Let's recap and revisit the details of the remaining dataset.<jupyter_code>print(sale_house.shape)
sale_house.head(10)
sale_house.describe()<jupyter_output><empty_output><jupyter_text>According to the above chart, we can observe that 1) The min value for ZIP CODE is 0, we don't have ZIP CODE that equals to 02) The Building Year of some properties are 0, this should be incorrect as well.3) Some properties have 0 SQUARE FEET...Let's fix them ZIP CODE<jupyter_code>plt.figure(figsize=(10,8))
plt.title(' ZIP CODE ')
plt.xlim(0,12000)

sale_house['ZIP CODE'].value_counts().sort_index().plot.line()
#sale_house = sale_house[(sale_house['ZIP CODE'] != 0)]
#find the most frequent value for zipcode 
from collections import Counter
zip = Counter(sale_house['ZIP CODE'].tolist()).most_common(1)
zip
# so the most common zipcode is 10314
#zipcode=[]
#for i in range(len(sale_house['ZIP CODE'])):
# if sale_house['ZIP CODE'].tolist()[i] == 0:
#   zipcode.append(10314)
# else:
#   zipcode.append(sale_house['ZIP CODE'].tolist()[i])
#sale_house['ZIP CODE'] = zipcode
import numba as nb
@nb.jit()
def frequent(x,y):
  result=[]
  for i in range(len(x)):
    if x.tolist()[i] == 0:
      result.append(y)
    else:
      result.append(x.tolist()[i])
  return result
sale_house['ZIP CODE'] = frequent(sale_house['ZIP CODE'], 10314)
sale_house['ZIP CODE'].value_counts().sort_index().plot.line()<jupyter_output><empty_output><jupyter_text>BUILDING YEAR<jupyter_code>plt.figure(figsize=(10,8))
plt.title(' Year Built ')
plt.xlim(1500,2020)

sale_house['YEAR BUILT'].value_counts().sort_index().plot.line()
#sale_house = sale_house[(sale_house['YEAR BUILT'] > 1875)]
freyear = Counter(sale_house['YEAR BUILT'].tolist()).most_common(2)
freyear
#yearBuilt=[]
#for i in range(len(sale_house['YEAR BUILT'])):
#    if sale_house['YEAR BUILT'].tolist()[i]==0:
#        yearBuilt.append(1920)
#    else:
#        yearBuilt.append(sale_house['YEAR BUILT'].tolist()[i])
#sale_house['YEAR BUILT']=yearBuilt

sale_house['YEAR BUILT'] = frequent(sale_house['YEAR BUILT'], 1920)
sale_house = sale_house[(sale_house['YEAR BUILT'] > 1875)]
sale_house['YEAR BUILT'].value_counts().sort_index().plot.line()<jupyter_output><empty_output><jupyter_text>Finally, it's time check the outliers in our datasets, let's first look into the column sale price.SALE PRICE<jupyter_code>plt.figure(figsize=(10,5))

sns.boxplot(x='SALE PRICE', data=sale_house)
plt.ticklabel_format(style='plain', axis='x')
plt.title('SALE PRICE in USD')
plt.show()
sale_house = sale_house[(sale_house['SALE PRICE'] > 0) & (sale_house['SALE PRICE'] < 500000000)] 

sns.distplot(sale_house['SALE PRICE'])

sale_house = sale_house[(sale_house['SALE PRICE'] > 0) & (sale_house['SALE PRICE'] < 5000000)]

sns.distplot(sale_house['SALE PRICE'])<jupyter_output><empty_output><jupyter_text>SQUARE FEET<jupyter_code>sale_house = sale_house[sale_house['GROSS SQUARE FEET'] < 10000]
sale_house = sale_house[sale_house['LAND SQUARE FEET'] < 10000]

plt.figure(figsize=(10,6))
sns.regplot(x='GROSS SQUARE FEET', y='SALE PRICE', data=sale_house, fit_reg=False, scatter_kws={'alpha':0.3})

plt.figure(figsize=(10,6))
sns.regplot(x='LAND SQUARE FEET', y='SALE PRICE', data=sale_house, fit_reg=False, scatter_kws={'alpha':0.3})<jupyter_output><empty_output><jupyter_text>UNIT NUMBER<jupyter_code>sale_house[["TOTAL UNITS", "SALE PRICE"]].groupby(['TOTAL UNITS'], as_index=False).count().sort_values(by='SALE PRICE', ascending=False)
sale_house = sale_house[(sale_house['TOTAL UNITS'] > 0) & (sale_house['TOTAL UNITS'] != 2261)] 

plt.figure(figsize=(10,6))
sns.boxplot(x='TOTAL UNITS', y='SALE PRICE', data=sale_house)
plt.title('Total Units vs Sale Price')
plt.show()<jupyter_output><empty_output><jupyter_text>2) Preparation<jupyter_code>#"Apartment Number" only has few values, so we are going to drop it.
print("The percent of rows with null in apartment number:" +str(sum(df['APARTMENT NUMBER']==' ')/len(df)))
del sale_house['APARTMENT NUMBER']
del sale_house['ADDRESS']

del sale_house['NEIGHBORHOOD']
del sale_house['BUILDING CLASS AT PRESENT']
del sale_house['BUILDING CLASS AT TIME OF SALE']
cor = sale_house.corr()
fig, ax = plt.subplots(figsize=(12,12))
sns.heatmap(cor,annot=True, ax = ax)<jupyter_output><empty_output><jupyter_text>Removing highing correlated independent variable from datasets.<jupyter_code># sale_house.drop(['RESIDENTIAL UNITS','GROSS SQUARE FEET',],inplace=True, axis=1)
# sale_house.head()
combine=[]
combine = sale_house[['RESIDENTIAL UNITS','GROSS SQUARE FEET']]
combine
#sqr.append(sale_house['RESIDENTIAL UNITS'])
from sklearn.decomposition import PCA
pca = PCA(n_components = 1)
pca.fit(combine)
sale_house['COMBINE RU GSF'] = pca.transform(combine)
del sale_house['RESIDENTIAL UNITS']
del sale_house['GROSS SQUARE FEET']
sale_house.head()
#one hot encoded
#https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/
one_hot_features = ['BOROUGH', 'BUILDING CLASS CATEGORY','TAX CLASS AT PRESENT','TAX CLASS AT TIME OF SALE']
# Convert categorical variables into dummy/indicator variables (i.e. one-hot encoding).
one_hot_encoded = pd.get_dummies(sale_house[one_hot_features])
one_hot_encoded.info(verbose=True, memory_usage=True, null_counts=True)
numeric_data=sale_house.select_dtypes(include=[np.number])
numeric_data.describe()
df = sale_house
scaler = StandardScaler()
scaler.fit(df[numeric_data.columns])
scaled = scaler.transform(df[numeric_data.columns])

for i, col in enumerate(numeric_data.columns):
       df[col] = scaled[:,i]

df.drop(one_hot_features,axis=1,inplace=True)
df = pd.concat([df, one_hot_encoded] ,axis=1)
df.head()
# classifying data into independent and dependent variable
X = df.drop(['SALE PRICE'],axis = 1).values
y = df['SALE PRICE'].values
# creating test and training set data, 70% train, 30% test
X_train,X_test,y_train,y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)<jupyter_output><empty_output><jupyter_text>3) Predition 3.0)Helper Function<jupyter_code>def rmse(y_test,y_pred):
      return np.sqrt(mean_squared_error(y_test,y_pred))<jupyter_output><empty_output><jupyter_text>3.1) Linear<jupyter_code>linear=LinearRegression()
linear.fit(X_train, y_train)
y_pred = linear.predict(X_test)
result_linear = rmse(y_test, y_pred)
result_linear<jupyter_output><empty_output><jupyter_text>3.2)Ridge, Lasso<jupyter_code># fitting linear regression to training set
regressor = Ridge(alpha=0.01, normalize=True)
regressor.fit(X_train,y_train)
y_pred = regressor.predict(X_test)
result_ridge = rmse(y_test, y_pred)
result_ridge
#3.6)Lasso
Lassoregressor = Lasso(alpha = 0.01, normalize =True)
Lassoregressor.fit(X_train,y_train)

y_predict = Lassoregressor.predict(X_test)
result_Lasso = rmse(y_test, y_predict)
result_Lasso<jupyter_output><empty_output><jupyter_text>3.3)Random Forest<jupyter_code>from sklearn.tree import DecisionTreeRegressor
dtree = DecisionTreeRegressor()
dtree.fit(X_train, y_train)
y_pred = dtree.predict(X_test)
result_dt = rmse(y_test, y_pred)
result_dt
dforest = RandomForestRegressor(n_estimators=100, criterion='mse', bootstrap=True, n_jobs=-1)
dforest.fit(X_train, y_train)
y_pred = dforest.predict(X_test)
result_rf = rmse(y_test, y_pred)
result_rf<jupyter_output><empty_output><jupyter_text>3.4)Gradient boosting<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import GridSearchCV
reg = GradientBoostingRegressor(alpha=0.9, ccp_alpha=0.0, criterion='friedman_mse',
                          init=None, learning_rate=0.05, loss='ls', max_depth=7,
                          max_features=None, max_leaf_nodes=None,
                          min_impurity_decrease=0.0, min_impurity_split=None,
                          min_samples_leaf=1, min_samples_split=2,
                          min_weight_fraction_leaf=0.0, n_estimators=500,
                          n_iter_no_change=None, presort='deprecated',
                          random_state=None, subsample=1.0, tol=0.0001,
                          validation_fraction=0.1, verbose=0, warm_start=False)
reg.fit(X_train, y_train)
pre_GBM = reg.predict(X_test)
result_gl = rmse(y_test, pre_GBM)
result_gl<jupyter_output><empty_output><jupyter_text>3.5)AdaBoost<jupyter_code>from sklearn.ensemble import AdaBoostRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
reg = AdaBoostRegressor(base_estimator=None, learning_rate=0.05, loss='linear',
                  n_estimators=50, random_state=None)
reg.fit(X_train, y_train)
pre = reg.predict(X_test)
result_gl = rmse(y_test, pre)
result_gl<jupyter_output><empty_output><jupyter_text>3.6) Stacked<jupyter_code>from sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC
from sklearn.ensemble import RandomForestRegressor,  GradientBoostingRegressor
from sklearn.kernel_ridge import KernelRidge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.metrics import mean_squared_error
import xgboost as xgb
import lightgbm as lgb<jupyter_output><empty_output><jupyter_text>This time we add a cross validation approach.<jupyter_code>n_folds = 5

def rmsle_cv(model):
    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(X_train)
    rmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring="neg_mean_squared_error", cv = kf))
    return(rmse)
linear1 = LinearRegression()
score = rmsle_cv(linear1)
print("\nLasso score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
lasso = make_pipeline(RobustScaler(), Lasso(alpha =0.0005, random_state=1))
score = rmsle_cv(lasso)
print("\nLasso score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
ridge = make_pipeline(RobustScaler(), Ridge(alpha =0.01, random_state=1))
score = rmsle_cv(ridge)
print("\nridge score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
elastic = make_pipeline(RobustScaler(), ElasticNet(alpha =0.01, random_state=1))
score = rmsle_cv(elastic)
print("\nElastic score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
dforest = RandomForestRegressor(n_estimators=100, criterion='mse', bootstrap=True, n_jobs=-1)
score = rmsle_cv(dforest)
print("\nRandomF score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
model_gb = GradientBoostingRegressor(alpha=0.9, ccp_alpha=0.0, criterion='friedman_mse',
                          init=None, learning_rate=0.05, loss='ls', max_depth=7,
                          max_features=None, max_leaf_nodes=None,
                          min_impurity_decrease=0.0, min_impurity_split=None,
                          min_samples_leaf=1, min_samples_split=2,
                          min_weight_fraction_leaf=0.0, n_estimators=500,
                          n_iter_no_change=None, presort='deprecated',
                          random_state=None, subsample=1.0, tol=0.0001,
                          validation_fraction=0.1, verbose=0, warm_start=False)
score = rmsle_cv(model_gb)
print("GB score: {:.4f} ({:.4f})\n" .format(score.mean(), score.std()))

ada = AdaBoostRegressor(base_estimator=None, learning_rate=0.05, loss='linear',
                  n_estimators=50, random_state=None)
score = rmsle_cv(ada)
print("AdaBoost Ridge score: {:.4f} ({:.4f})\n".format(score.mean(), score.std()))
# https://github.com/akshaykumarvikram/kaggle-advanced-regression-algos
class StackingAveragedModels(BaseEstimator, RegressorMixin, TransformerMixin):
    def __init__(self, base_models, meta_model, n_folds=5):
        self.base_models = base_models
        self.meta_model = meta_model
        self.n_folds = n_folds
   
    # Fit the data on clones of the original models
    def fit(self, X, y):
        self.base_models_ = [list() for x in self.base_models]
        self.meta_model_ = clone(self.meta_model)
        kfold = KFold(n_splits=self.n_folds, shuffle=True, random_state=156)
        
        # Train cloned base models then create out-of-fold predictions
        # that are needed to train the cloned meta-model
        out_of_fold_predictions = np.zeros((X.shape[0], len(self.base_models)))
        for i, model in enumerate(self.base_models):
            for train_index, holdout_index in kfold.split(X, y):
                instance = clone(model)
                self.base_models_[i].append(instance)
                instance.fit(X[train_index], y[train_index])
                y_pred = instance.predict(X[holdout_index])
                out_of_fold_predictions[holdout_index, i] = y_pred
                
        # Now train the cloned  meta-model using the out-of-fold predictions as new feature
        self.meta_model_.fit(out_of_fold_predictions, y)
        return self
   
    #Do the predictions of all base models on the test data and use the averaged predictions as 
    #meta-features for the final prediction which is done by the meta-model
    def predict(self, X):
        meta_features = np.column_stack([
            np.column_stack([model.predict(X) for model in base_models]).mean(axis=1)
            for base_models in self.base_models_ ])
        return self.meta_model_.predict(meta_features)
stacked_averaged_models = StackingAveragedModels(base_models = (lasso, ridge, elastic, ada, dforest),
                                                 meta_model = model_gb)

score = rmsle_cv(stacked_averaged_models)
print("Stacking Averaged models score: {:.4f} ({:.4f})".format(score.mean(), score.std()))
stacked_averaged_models = StackingAveragedModels(base_models = (model_gb, dforest),
                                                 meta_model = lasso)

stacked_averaged_models.fit(X_train, y_train)
stacked_train_pred = stacked_averaged_models.predict(X_train)
stacked_pred = stacked_averaged_models.predict(X_test)
print(rmse(y_test, stacked_pred))<jupyter_output><empty_output><jupyter_text>3.7)Weighted Stacking （Our Contribution）<jupyter_code>from sklearn.model_selection import KFold 
X = X_train
Y = y_train
kf = KFold(n_splits=5)
pre_linear = []
pred_Lasso = []
pred_Ridge = []
pred_RF = []
pred_GBM = []
pred_Ada = []
pred_ela = []
for train_index, test_index in kf.split(X):
  linear=LinearRegression()
  linear.fit(X[train_index], Y[train_index])
  y_pred_linear = linear.predict(X[test_index])
  pre_linear = pre_linear + y_pred_linear.tolist()
  ridge = Ridge(alpha=0.01, normalize=True)
  ridge.fit(X[train_index], Y[train_index])
  y_pred_ridge = ridge.predict(X[test_index])
  pred_Ridge = pred_Ridge + y_pred_ridge.tolist()
  lasso = Lasso(alpha = 0.0005, normalize =True)
  lasso.fit(X[train_index], Y[train_index])
  y_pred_lasso = lasso.predict(X[test_index])
  pred_Lasso = pred_Lasso + y_pred_lasso.tolist()
  rf = RandomForestRegressor(n_estimators=100, criterion='mse', bootstrap=True, n_jobs=-1)
  rf.fit(X[train_index], Y[train_index])
  y_pred_rf = rf.predict(X[test_index])
  pred_RF = pred_RF + y_pred_rf.tolist()
  gbm = GradientBoostingRegressor(alpha=0.9, ccp_alpha=0.0, criterion='friedman_mse',
                          init=None, learning_rate=0.05, loss='ls', max_depth=7,
                          max_features=None, max_leaf_nodes=None,
                          min_impurity_decrease=0.0, min_impurity_split=None,
                          min_samples_leaf=1, min_samples_split=2,
                          min_weight_fraction_leaf=0.0, n_estimators=500,
                          n_iter_no_change=None, presort='deprecated',
                          random_state=None, subsample=1.0, tol=0.0001,
                          validation_fraction=0.1, verbose=0, warm_start=False)
  gbm.fit(X[train_index], Y[train_index])
  y_pred_gbm = gbm.predict(X[test_index])
  pred_GBM = pred_GBM + y_pred_gbm.tolist()
  ada = AdaBoostRegressor(base_estimator=None, learning_rate=0.05, loss='linear',
                  n_estimators=50, random_state=None)
  ada.fit(X[train_index], Y[train_index])
  y_pred_ada = ada.predict(X[test_index])
  pred_Ada = pred_Ada + y_pred_ada.tolist()
  ela = ElasticNet(alpha =0.01, random_state=1)
  ela.fit(X[train_index], Y[train_index])
  y_pred_ela = ela.predict(X[test_index])
  pred_ela = pred_ela + y_pred_ela.tolist()
combine = 0*np.array(pred_linear)+0*np.array(pred_Lasso)+0.04*np.array(pred_Ridge)+0*np.array(pred_RF)+0.96*np.array(pred_GBM)+0*np.array(pred_Ada)
rmse(combine,y_train)
from scipy.optimize import nnls 
fin_mat = np.zeros((26231,6))
fin_mat[:,0]=np.array(pre_linear)
fin_mat[:,1]=np.array(pred_Lasso)
fin_mat[:,2]=np.array(pred_Ridge)
fin_mat[:,3]=np.array(pred_RF)
fin_mat[:,4]=np.array(pred_GBM)
fin_mat[:,5]=np.array(pred_Ada)
fin_mat
b=nnls(fin_mat, y_train)
b<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup