<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 1, Module 3*---  Ridge Regression AssignmentWe're going back to our other **New York City** real estate dataset. Instead of predicting apartment rents, you'll predict property sales prices.But not just for condos in Tribeca...- [x] Use a subset of the data where `BUILDING_CLASS_CATEGORY` == `'01 ONE FAMILY DWELLINGS'` and the sale price was more than 100 thousand and less than 2 million.- [x] Do train/test split. Use data from January — March 2019 to train. Use data from April 2019 to test.- [x] Do one-hot encoding of categorical features.- [ ] Do feature selection with `SelectKBest`.- [ ] Fit a ridge regression model with multiple features. Use the `normalize=True` parameter (or do [feature scaling](https://scikit-learn.org/stable/modules/preprocessing.html) beforehand — use the scaler's `fit_transform` method with the train set, and the scaler's `transform` method with the test set)- [ ] Get mean absolute error for the test set.- [ ] As always, commit your notebook to your fork of the GitHub repo.The [NYC Department of Finance](https://www1.nyc.gov/site/finance/taxes/property-rolling-sales-data.page) has a glossary of property sales terms and NYC Building Class Code Descriptions. The data comes from the [NYC OpenData](https://data.cityofnewyork.us/browse?q=NYC%20calendar%20sales) portal. Stretch GoalsDon't worry, you aren't expected to do all these stretch goals! These are just ideas to consider and choose from.- [ ] Add your own stretch goal(s) !- [ ] Instead of `Ridge`, try `LinearRegression`. Depending on how many features you select, your errors will probably blow up! 💥- [ ] Instead of `Ridge`, try [`RidgeCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html).- [ ] Learn more about feature selection:    - ["Permutation importance"](https://www.kaggle.com/dansbecker/permutation-importance)    - [scikit-learn's User Guide for Feature Selection](https://scikit-learn.org/stable/modules/feature_selection.html)    - [mlxtend](http://rasbt.github.io/mlxtend/) library    - scikit-learn-contrib libraries: [boruta_py](https://github.com/scikit-learn-contrib/boruta_py) & [stability-selection](https://github.com/scikit-learn-contrib/stability-selection)    - [_Feature Engineering and Selection_](http://www.feat.engineering/) by Kuhn & Johnson.- [ ] Try [statsmodels](https://www.statsmodels.org/stable/index.html) if you’re interested in more inferential statistical approach to linear regression and feature selection, looking at p values and 95% confidence intervals for the coefficients.- [ ] Read [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf), Chapters 1-3, for more math & theory, but in an accessible, readable way.- [ ] Try [scikit-learn pipelines](https://scikit-learn.org/stable/modules/compose.html).<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'

# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
import pandas as pd
# breaks local envviroment it's not being used so i went ahead and commented it out
#import pandas_profiling

# Read New York City property sales data
df = pd.read_csv(DATA_PATH+'condos/NYC_Citywide_Rolling_Calendar_Sales.csv')

# Change column names: replace spaces with underscores
df.columns = [col.replace(' ', '_') for col in df]

# SALE_PRICE was read as strings.
# Remove symbols, convert to integer
df['SALE_PRICE'] = (
    df['SALE_PRICE']
    .str.replace('$','')
    .str.replace('-','')
    .str.replace(',','')
    .astype(int)
)
# BOROUGH is a numeric column, but arguably should be a categorical feature,
# so convert it from a number to a string
df['BOROUGH'] = df['BOROUGH'].astype(str)
# Reduce cardinality for NEIGHBORHOOD feature

# Get a list of the top 10 neighborhoods
top10 = df['NEIGHBORHOOD'].value_counts()[:10].index

# At locations where the neighborhood is NOT in the top 10, 
# replace the neighborhood with 'OTHER'
df.loc[~df['NEIGHBORHOOD'].isin(top10), 'NEIGHBORHOOD'] = 'OTHER'
# make a copy of that data so that I don't mess anything up
copy=df.copy()
# make a mask for building class category
condition=copy['BUILDING_CLASS_CATEGORY']=='01 ONE FAMILY DWELLINGS'
copy=copy[condition]
# make a mask for building sales price
condition=(copy['SALE_PRICE']<2000000) | (copy['SALE_PRICE']>100000)
copy=copy[condition]
# make a new column to sort date
copy['dt']=pd.to_datetime(copy['SALE_DATE'],infer_datetime_format=True)
# make train test split
train=copy[copy['dt'].dt.month<=3]
test=copy[copy['dt'].dt.month==4]

# check the train set to see if it includes what i want
train['dt'].value_counts().sort_index()
# check the test set to see if it includes what i need
test['dt'].value_counts()
# take a look at diffrent categories that would be good to onehot encode
train.describe(include='object')
# set features based on the cardinality of the data in them
target='SALE_PRICE'
ignore=['dt','BUILDING_CLASS_CATEGORY','BUILDING_CLASS_AT_PRESENT','ADDRESS','APARTMENT_NUMBER','SALE_DATE','BUILDING_CLASS_AT_TIME_OF_SALE']
features=train.columns.drop([target]+ignore)
# print shape
features
# make cv split
X_train=train[features]
y_train=train[target]
X_test=test[features]
y_test=test[target]

# use one hot encoding to encode some of my catigorical variables
import category_encoders as ce
encode=ce.OneHotEncoder(use_cat_names=True)
X_train = encode.fit_transform(X_train)
X_test = encode.transform(X_test)

# use selectKbest to select my best features, or features with highest explained variance
from sklearn.feature_selection import SelectKBest,f_regression
# note that im using 5 features her to reduce the dimentions of my feature matrix by a factor of 2/3
# in the example for lecture it was about 1/2
selector=SelectKBest(score_func=f_regression,k=5)
X_train_selected=selector.fit_transform(X_train,y_train)
X_test_selected=selector.transform(X_test,y_test)

X_test["y"]=y_test
X_train["y"]=y_train
X_test.dropna(inplace=True)
X_train.dropna(inplace=True)
y_test=X_test['y']
y_train=X_train['y']
X_test.drop('y',axis=1,inplace=True)
X_train.drop('y',axis=1,inplace=True)<jupyter_output><empty_output>