<jupyter_start><jupyter_text>0.	Module imports<jupyter_code>import os

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # visualization library

%matplotlib inline<jupyter_output><empty_output><jupyter_text>1.	Data Loading<jupyter_code>TRAIN_PATH = os.path.join("..", "input", "train.csv")
TEST_PATH = os.path.join("..", "input", "test.csv")

train = pd.read_csv(TRAIN_PATH)
test = pd.read_csv(TEST_PATH)
train.head()<jupyter_output><empty_output><jupyter_text>Show the columns in the dataset  2.	Data exploration<jupyter_code>train.describe()<jupyter_output><empty_output><jupyter_text>It seems that there are outliers in the "trip_duration"<jupyter_code>train.isnull().sum()
plt.subplots(figsize=(11,9))
plt.title("destribution of trip_duration values")
train["trip_duration"].plot.box()
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 1458644 entries, id2875421 to id1209952
Data columns (total 10 columns):
vendor_id             1458644 non-null int64
pickup_datetime       1458644 non-null object
dropoff_datetime      1458644 non-null object
passenger_count       1458644 non-null int64
pickup_longitude      1458644 non-null float64
pickup_latitude       1458644 non-null float64
dropoff_longitude     1458644 non-null float64
dropoff_latitude      1458644 non-null float64
store_and_fwd_flag    1458644 non-null object
trip_duration         1458644 non-null int64
dtypes: float64(4), int64(3), object(3)
memory usage: 122.4+ MB<jupyter_text>3.	Data preprocessing I want to ignore some values that I consider to be outliers and that can bias my prediction.And remove the case we have 0 passenger<jupyter_code>train = train[train['passenger_count']>= 1]<jupyter_output><empty_output><jupyter_text>we don't take 0 passenger<jupyter_code>train = train[train['trip_duration']>= 90 ]
train = train[train['trip_duration']<= 10800 ]<jupyter_output><empty_output><jupyter_text>Let's take only : the duration that are less than 3hours, 3*3600 = 10800sand more than 1min30, 90swe will ignore 2112 lines + 15799 lines = 17911 lineswe will ignore 1.23% of the total of our data set  we remove outliers from altitude and longetude<jupyter_code>plt.subplots(figsize=(11,9))
plt.title("destribution of pickup_longitude values")
train["pickup_longitude"].plot.box()
train = train.loc[train['pickup_longitude']> -80]
plt.subplots(figsize=(11,9))
plt.title("destribution of pickup_latitude values")
train["pickup_latitude"].plot.box()
train = train.loc[train['pickup_latitude']< 44]
plt.subplots(figsize=(11,9))
plt.title("destribution of dropoff_longitude values")
train["dropoff_longitude"].plot.box()
train = train.loc[train['dropoff_longitude']> -90]
plt.subplots(figsize=(11,9))
plt.title("destribution of dropoff_latitude values")
train["dropoff_latitude"].plot.box()
train = train.loc[train['dropoff_latitude']> 34]<jupyter_output><empty_output><jupyter_text>4.	Features engineering  We convert string to date<jupyter_code>train['pickup_datetime'] = pd.to_datetime(train['pickup_datetime'], format='%Y-%m-%d %H:%M:%S')
test['pickup_datetime'] = pd.to_datetime(test['pickup_datetime'], format='%Y-%m-%d %H:%M:%S')<jupyter_output><empty_output><jupyter_text>Creation of a new column hour<jupyter_code>train['hour'] = train.loc[:,'pickup_datetime'].dt.hour;
test['hour'] = test.loc[:,'pickup_datetime'].dt.hour;<jupyter_output><empty_output><jupyter_text>Features engineering<jupyter_code>train['dist'] = (abs(train['pickup_latitude']-train['dropoff_latitude'])
                        + abs(train['pickup_longitude']-train['dropoff_longitude']))
test['dist'] = (abs(test['pickup_latitude']-test['dropoff_latitude'])
                        + abs(test['pickup_longitude']-test['dropoff_longitude']))
y_train = train["trip_duration"]
X_train = train[["vendor_id", "passenger_count", "pickup_longitude", "pickup_latitude", "dropoff_longitude","dropoff_latitude", "dist", "hour" ]]<jupyter_output><empty_output><jupyter_text>5.	Choice of a validation method<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, random_state=42)<jupyter_output><empty_output><jupyter_text>6.	Selection of models<jupyter_code>from sklearn.model_selection import cross_val_score

from sklearn.ensemble import RandomForestRegressor

from sklearn.linear_model import SGDRegressor

from sklearn.linear_model import LinearRegression

from sklearn.model_selection import ShuffleSplit
sgd = SGDRegressor()
sgd.fit(X_train, y_train)
rfr = RandomForestRegressor(n_estimators=10,min_samples_leaf=100, min_samples_split=150)
rfr.fit(X_train, y_train)
cv = ShuffleSplit(n_splits=4, test_size=0.8, random_state=42)
cv_scores = cross_val_score(knn, X_train, y_train, cv=cv, scoring= 'neg_mean_squared_log_error')
cv_scores<jupyter_output><empty_output><jupyter_text>7.	Training of the model<jupyter_code>m = RandomForestRegressor(n_estimators=10,min_samples_leaf=100, min_samples_split=150)
m.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>8.	Prediction<jupyter_code>X_test = test[["vendor_id", "passenger_count","pickup_longitude", "pickup_latitude","dropoff_longitude","dropoff_latitude","dist", "hour"]]
prediction = m.predict(X_test)
prediction
test.head()<jupyter_output><empty_output><jupyter_text>9.	Prediction submissions<jupyter_code>my_submission = pd.DataFrame({'id': test.id, 'trip_duration': prediction})
my_submission.head()
my_submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output>