<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 3, Module 1*--- Define ML problemsYou will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your decisions.- [ ] Choose your target. Which column in your tabular dataset will you predict?- [ ] Is your problem regression or classification?- [ ] How is your target distributed?    - Classification: How many classes? Are the classes imbalanced?    - Regression: Is the target right-skewed? If so, you may want to log transform the target.- [ ] Choose your evaluation metric(s).    - Classification: Is your majority class frequency >= 50% and < 70% ? If so, you can just use accuracy if you want. Outside that range, accuracy could be misleading. What evaluation metric will you choose, in addition to or instead of accuracy?    - Regression: Will you use mean absolute error, root mean squared error, R^2, or other regression metrics?- [ ] Choose which observations you will use to train, validate, and test your model.    - Are some observations outliers? Will you exclude them?    - Will you do a random split or a time-based split?- [ ] Begin to clean and explore your data.- [ ] Begin to choose which features, if any, to exclude. Would some features "leak" future information?If you haven't found a dataset yet, do that today. [Review requirements for your portfolio project](https://lambdaschool.github.io/ds/unit2) and choose your dataset.Some students worry, ***what if my model isn't “good”?*** Then, [produce a detailed tribute to your wrongness. That is science!](https://twitter.com/nathanwpyle/status/1176860147223867393)<jupyter_code># Choose your target. Which column in your tabular dataset will you predict?
df1 = pd.read_csv('vehicles_trimmed.csv')

# Choose target
target = 'price'
# Is your problem regression or classification?
# Regression, predicting a continuous variable with many levels
# Target distribution

sns.distplot(df1['price'])
print(np.median(df1['price']))
y.describe()
# Very right skewed but the 0s are holding it back even more. 
# Let's look at logs
y = df1['price']
y_log = np.log1p(y)
sns.distplot(y_log)
# Chop out some outliers
df2 = df1[(df1['price'] >= np.percentile(df1['price'], 10)) &
          (df['price'] <= np.percentile(df1['price'], 99.95))]
df2['price'].describe()
y = np.log1p(df2['price'])
sns.distplot(y)
y = y.replace(0, np.NaN)
sns.distplot(y)
# Now it's left skewed
y.describe()
# Choose your evaluation metric(s).

# R^2 and perhaps MAE
#Choose which observations you will use to train, validate, and test your model.
train_orig, test = train_test_split(df2, test_size = 0.1, train_size = 0.9)
test
train, val = train_test_split(train_orig, train_size = 0.9, test_size = 0.1)
train
#  Begin to clean and explore your data.
# - [x] Begin to choose which features, if any, to exclude. Would some features "leak" future information?
# Question about what to do with price distribution: should I keep closing in the quartiles until the lower hump is left out?
# NaNs? Get rid of them?<jupyter_output><empty_output>