<jupyter_start><jupyter_text>CS 109A/STAT 121A/AC 209A/CSCI E-109A:  Midterm - 2017**Harvard University****Fall 2017****Instructors**: <NAME>, <NAME>, <NAME>, <NAME>--- INSTRUCTIONS- You must submit the Midterm on your own. ** No group submissions are allowed**. You may use any print or online resources but ** you may not work or consult with others**.- Restart the kernel and run the whole notebook again before you submit. - Please submit both a notebook and a pdf. ---  Flight DelaysThe U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled, and diverted flights are published in DOT's monthly Air Travel Consumer Report and in this dataset of 2015 flight delays and cancellations. DataEach entry of the flights.csv file corresponds to a flight.  More than 5,800,000 flights were recorded in 2015. These flights are described according to 31 variables. Further details of these variables can be found here, if you are interested (not needed to answer these questions).  |          Name          | Type | DESCRIPTION ||--------------------------------|---|----------------------------------------------------------------------|| DATE                     | object | The date in python datetime format || MONTH                     | int64 | The month of the year(1-12) || DAY                       | int64 | The day of the month || DAY_OF_WEEK               | int64 | The day of the week(1-7, MON-SUN) || AIRLINE                  | object | An identifier for the airline || FLIGHT_NUMBER             | int64 | The flight number || TAIL_NUMBER              | object | The tail number (aircraft) corresponding to this flight || ORIGIN_AIRPORT           | object | The code for origin airport || DESTINATION_AIRPORT      | object | The code for destination airport || SCHED_DEP                | object | The departure time in python datetime.time format || SCHED_ARR                | object | The arrival time in python datetime.time format || DEPARTURE_DELAY         |  float64| The delay incurred at the origin  (mins) || ARRIVAL_DELAY           | float64 | The delay when the flight reached the (mins) destination || DISTANCE                  | int64 | Distance in miles between origin and destination || SCHEDULED_TIME          | float64 | Scheduled time of flight (minutes) || ELAPSED_TIME            | float64 | Actual time of flight (minutes) || AIR_SYSTEM_DELAY        | float64 | What part of the delay was NASD?(mins) || SECURITY_DELAY          | float64 | What part of the delay was due to security problems? (mins) || AIRLINE_DELAY           | float64 | What part of the delay is due to the airline? (mins) || LATE_AIRCRAFT_DELAY     | float64 | What part of the delay is due to previous flight(s) being late(mins) || WEATHER_DELAY           | float64 | Delay due to extreme weather events(min) |You can read more about the various weather delays [here](https://www.rita.dot.gov/bts/help/aviation/html/understanding.html) if you are so inclined. Data/CaveatsThe data file, flights.csv, is found here (note, it is about 70MB). This data is already preprocessed, reduced, partially cleaned and therefore not identical to the original dataset.<jupyter_code>import numpy as np
import pandas as pd
from datetime import datetime
import time
import matplotlib
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import LogisticRegressionCV
import sklearn.metrics as metrics
from sklearn.preprocessing import PolynomialFeatures
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.tree import DecisionTreeClassifier as DecisionTree
from sklearn.model_selection import cross_val_score
from sklearn.metrics import accuracy_score
from sklearn import preprocessing
from sklearn.tree import export_graphviz
from IPython.display import Image
from IPython.display import display
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LassoCV
from sklearn.linear_model import RidgeCV
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Problem Description We will build two separate models: one model that classifies whether a flight will be delayed and a second model that predicts the length of delay given that a flight is truly delayed. Only consider models taught in class so far. ** Consider the following: **This is a large dataset; think of strategies on how to solve this problem. Create a manageable subsample of the data that you can use to train and test/validate, but eventually you should predict on all the data (excluding the training set). Questions1. (5pts) Create a new variable, `DELAY_OR_NOT`: a boolean/indicator variable which indicates any arrival delay under 15 mins as a 0, and any delay at or above 15 mins as a 1 (`ARRIVAL_DELAY >= 15`).2. (5pts) Make sure you understand the data variable descriptions before you start the analysis. Consider all the columns and determine and list which of these predictors should not be used. 3. (15pts) Perform EDA to gain intuition of the factors that affect delay and provide visuals: do delays vary across airlines, or time of departure, or airport (do, at the very least, Chicago (ORD), Boston (BOS), and your favorite another airport), or airport traffic?4. (20pts) Build a classification model that classifies delays according to `DELAY_OR_NOT`. This is an unbalanced dataset, thus consider the appropriate performance metric when reporting your results. 5. (5pts) Given your model, comment on the importance of factors as related to whether a flight is delayed.6. (5pts) Evaluate your model(s) on your test set, and finally provide a visual to show which airlines are predicted to have the most delays using all the data excluding the training and test set. 7. (15pts) Build a regression model that predicts the length of delay (on the log scale) given that a flight is truly delayed.8. (20pts) Write a report (in the last markdown cell in your notebook with your findings (without code)). Describe the main design decisions you have made with justifications. Clearly explain your methodology and results. This should not be more than 300 words. You may use up to 5 diagrams. **Question 1**(5pts) Create a new variable, DELAY_OR_NOT: a boolean/indicator variable which indicates any arrival delay under 15 mins as a 0, and any delay at or above 15 mins as a 1 (ARRIVAL_DELAY >= 15).<jupyter_code>#Load the data
df = pd.read_csv("cs109a_midterm.csv")
#Display the first couple columns
df.head()
#Let's first clean our dataframe
#See where the missing values are
cols_missing_values = df.columns[df.isnull().any()]
cols_missing_values
#Let's see if we have any missing values in the 'ARRIVAL_DELAY' column
np.sum([df.ARRIVAL_DELAY.isnull()])
#Create the variable 'DELAY_OR_NOT' in our DataFrame based on if 'ARRIVAL_DELAY_>=15 (min)
#Since we saw we do not have any missing values in ARRIVAL_DELAY column, we can proceed.
DELAY_OR_NOT = (df.ARRIVAL_DELAY >= 15).astype(int)
df['DELAY_OR_NOT'] = DELAY_OR_NOT<jupyter_output><empty_output><jupyter_text>**Question 2**(5pts) Make sure you understand the data variable descriptions before you start the analysis. Consider all the columns and determine and list which of these predictors should not be used. **I believe there is several columns which do not help in our analysis** DATE since we already have variables MONTH and DAY and we know all the data points are from 2015 SCHEDULED_TIMEis duplicate information since SCHEDULED_TIME = SCHED_ARRIVAL -  SCHED_DEPARTURE ELAPSED_TIME is duplicate information since ELAPSED_TIME = SCHED_ARRIVAL + ARRIVAL_DELAY - (SCHED_DEPARTURE + DEPARTURE DELAY) FLIGHT NUMBERIt is redundant since we have info about origin/destination and scheduled dep/arr time TAIL NUMBERThis is indicative of each unique plane. When we have a dataset with 800,000 observations and 4819 planes, it is very unlikely our regression model would be able to recognize if there is a particular plane which causes delay. All the delay featuresGiven the fact we already have a "delay_or_not" response we are seeking, our "types" of delays are not predictors of whether there will be a delay or not. In other words, a type of "delay" will predict the length of the delay but not whether there will be an event of a delay.  **So, we will get rid of those in our analysis in this subproblem. I already looked for missing values in the previous subquestion and the missing values happen to be only in the columns I am dropping.**<jupyter_code>#We copy the dataframe, since it's 2017 and we all have enough memory,right? 

new_df = df.copy(deep=True)

#drop all the columns based on the analysis in the cell above
del new_df['DATE']
del new_df['SCHEDULED_TIME']
del new_df['ELAPSED_TIME']
del new_df['FLIGHT_NUMBER']
del new_df['TAIL_NUMBER']

del new_df['ARRIVAL_DELAY']
del new_df['DEPARTURE_DELAY']
del new_df['AIR_SYSTEM_DELAY']
del new_df['AIRLINE_DELAY']
del new_df['SECURITY_DELAY']
del new_df['LATE_AIRCRAFT_DELAY']
del new_df['WEATHER_DELAY']<jupyter_output><empty_output><jupyter_text>**Question 3** (15pts) Perform EDA to gain intuition of the factors that affect delay and provide visuals: do delays vary across airlines, or time of departure, or airport (do, at the very least, Chicago (ORD), Boston (BOS), and your favorite another airport), or airport traffic?<jupyter_code>#We will ONLY look at the visualisation in the training data we will later use to build our model
#(I am splitting train/test here just for purposes for visualisation, I am later using the same random seed 
#to do it again after preproccessing)

#Subsample the data to reduce the # of observations.
np.random.seed(9001)
new_df_for_viz = new_df.sample(n=100000,random_state=6)
#Split the data in test and training (see Lecture 5)
train_df_viz, data_test_NOT_USE = train_test_split(new_df_for_viz,test_size=0.33, random_state=0)
#Let's see what airlines had the highest percentage of delays in 2015
list_of_airlines = [i for i in train_df_viz.AIRLINE.value_counts().index]
airlines_delays = []

for i in list_of_airlines:
    delayed = train_df_viz[train_df_viz.AIRLINE==i][train_df_viz.DELAY_OR_NOT==1].shape[0]
    not_delayed = train_df_viz[train_df_viz.AIRLINE==i][train_df_viz.DELAY_OR_NOT==0].shape[0]
    airlines_delays.append(delayed/(delayed+not_delayed))

plt.figure(figsize=(20,10))
plt.bar([i for i in range(len(airlines_delays))],airlines_delays,color='red',label = "Percentage of delayed flights by Airline")
plt.xticks([i for i in range(len(airlines_delays))],list_of_airlines);
mean_delays=np.mean(airlines_delays)
plt.axhline(mean_delays,label='Mean percentage of delay accross airlines')
plt.legend();
plt.xlabel("Airline")
plt.ylabel("Ratio of delayed flights");<jupyter_output><empty_output><jupyter_text>**We see significant differences in the ratio of delayed flights among airlines. For example, "HA" seems to have relatively less delayed flights than "NK"**<jupyter_code>#Let's see which airline had the most delays in 2015 as well as how many flights were not delayed for that airlines (total number)
import seaborn as sns
plt.figure(figsize=(20,10))
sns.set(style="darkgrid")
sns.countplot(x='AIRLINE',hue='DELAY_OR_NOT',data=train_df_viz,palette="Set2");<jupyter_output><empty_output><jupyter_text>**This graph is related to the one above. However, we are just plotting the absolute number of flights here.**<jupyter_code>#Let's see what days of the week had the most delays in 2015
weekday_delays = []
for i in range(1,8):
    delays = train_df_viz[train_df_viz.DAY_OF_WEEK==i][train_df_viz.DELAY_OR_NOT==1].shape[0]
    total = train_df_viz[train_df_viz.DAY_OF_WEEK==i].shape[0]
    weekday_delays.append(delays/total)
    
plt.figure(figsize=(20,10))
plt.bar([i for i in range(7)],weekday_delays,color='green',label = "Delay in a weekday")
plt.xticks([i for i in range(7)],['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']);
mean_delays=np.mean(weekday_delays)
plt.axhline(mean_delays,label='Mean delay across weekdays',color='gold')
plt.legend();
plt.xlabel("Day of the week")
plt.ylabel("Percentage of delays");<jupyter_output><empty_output><jupyter_text>**This visualisation indicates that certain days have a higher ratio of delayed flights than others. For example, Thursdays have the highest percentage of delayed flights while Saturdays the smallest.**<jupyter_code>#Let's visualize the total delays per day of the week
sns.set(style="darkgrid")
plt.figure(figsize=(20,10))
sns.countplot(x='DAY_OF_WEEK',hue='DELAY_OR_NOT',data=train_df_viz,palette="Set1")
plt.xticks([i for i in range(7)],['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']);<jupyter_output><empty_output><jupyter_text>**This graph is related to the one above. However, we are just plotting the absolute number of flights per day of the week here.**<jupyter_code>#Let's take a look at how many flights in total were delayed in each month 
sns.set(style="darkgrid")
plt.figure(figsize=(20,10))
sns.countplot(x='MONTH',data=train_df_viz[train_df_viz.DELAY_OR_NOT==1],palette="Set1");<jupyter_output><empty_output><jupyter_text>**In this graph, we plot the absolute number of delayed flights in any given month. We see that the lowest number of delayed flights is in September/October, while the most delayed flights are in June/July timeframe.**<jupyter_code>#Let's take a look at the percentage of flights delayed in each month 
months = train_df_viz.groupby('MONTH').mean()
plt.figure(figsize=(20,10))
plt.bar(months.index,months.DELAY_OR_NOT,color='gray',label = "Percentage delay in a month")
plt.axhline(mean_delays,label='Mean percentage delay across months',color='gold')
plt.legend(bbox_to_anchor=(1, 1));
plt.xlabel("Month")
plt.ylabel("Percentage of delays");<jupyter_output><empty_output><jupyter_text>**Here, we plot the ratio of flights delayed each month. The trends reflect what the previous graph with absolute numbers displayed.**<jupyter_code>#Number of delayed/not delayed flights departing every hour, also showing the overall traffic at the airports at a given hour

from datetime import datetime
import time

# string to time tuple
date_str = list(train_df_viz.SCHED_DEP)
time_tuples = []
for i in date_str:
    time_tuple = time.strptime(i, "%H:%M:%S")
    time_tuples.append(time_tuple.tm_hour)
train_df_viz['hour_of_dep'] = np.array(time_tuples)

plt.figure(figsize=(20,10))

sns.countplot(x='hour_of_dep',hue='DELAY_OR_NOT',data=train_df_viz,palette="Set1");
plt.xlabel("hour of the day");<jupyter_output><empty_output><jupyter_text>**Here we can see the highest number of flights departs in the morning while being the lest delayed, while evening flights are the most delayed and there is less of them than in the morning hours**<jupyter_code>#Now, let's plot the percentage of delayed flights dependent on the time departure

hour_delays=[]
for i in range(24):
    delays = train_df_viz[train_df_viz.hour_of_dep==i][train_df_viz.DELAY_OR_NOT==1].shape[0]
    total = train_df_viz[train_df_viz.hour_of_dep==i].shape[0]
    hour_delays.append(delays/total)

plt.figure(figsize=(20,10))
plt.bar([i for i in range(24)],hour_delays,color='gray',label = "Percentage delay in a month")
plt.xticks([i for i in range(24)],[i for i in range(24)]);
mean_delays=np.mean(weekday_delays)


plt.axhline(mean_delays,label='Mean percentage delay across day hours',color='gold')
plt.legend();
plt.xlabel("Hour of the day")
plt.ylabel("Percentage of delays")
del train_df_viz['hour_of_dep'];<jupyter_output><empty_output><jupyter_text>**Seems like most flights are delayed in the evening hours with the peak around 8pm, while morning flights have the least delays.**<jupyter_code>#Let's look at delays at 15 most busy airports (from left as the busiest)

frequent_airports = train_df_viz.ORIGIN_AIRPORT.value_counts().index[0:15]
airport_delays = []
for i in frequent_airports:
    delays = train_df_viz[train_df_viz.ORIGIN_AIRPORT==i][train_df_viz.DELAY_OR_NOT==1].shape[0]
    total = train_df_viz[train_df_viz.ORIGIN_AIRPORT==i].shape[0]
    airport_delays.append(delays/total)

plt.figure(figsize=(20,10))
plt.bar([i for i in range(len(frequent_airports))],airport_delays,color='orange',label = "Percentage delay origin aiport")
plt.xticks([i for i in range(len(frequent_airports))],[i for i in frequent_airports]);
mean_delays=np.mean(weekday_delays)
plt.axhline(mean_delays,label='Mean percentage delay across {} most frequent airports'.format(len(frequent_airports)),color='black')
plt.legend();
plt.xlabel("Origin airport")
plt.ylabel("Percentage of delays");<jupyter_output><empty_output><jupyter_text>**Here, we look at the 15 busiest airports (defined as how many flights depart from it in total in 2015). We can see that ORD or MCO have a high percentage of delayed flights while ATL or SEA have a low percentage of delayed flights.** **Question 4**(20pts) Build a classification model that classifies delays according to DELAY_OR_NOT. This is an unbalanced dataset, thus consider the appropriate performance metric when reporting your results.<jupyter_code>#Now, we need to convert our categorical variables
cols_to_change =['MONTH','DAY','DAY_OF_WEEK','AIRLINE','ORIGIN_AIRPORT','DESTINATION_AIRPORT']

#hot-encode
new_df_dummy = pd.get_dummies(new_df,columns=cols_to_change)

#Subsample the data to reduce the # of observations. 
new_df_sample = new_df_dummy.sample(n=100000,random_state=6)

#Split the data in test and training (see Lecture 5)
np.random.seed(9001)
data_train, data_test = train_test_split(new_df_sample,test_size=0.33, random_state=0)

#Check we actually got the right shapes
print(new_df_sample.shape,df.shape,data_train.shape,data_test.shape)

#now create a remaining df to test in Question 6
remaining_df = new_df_dummy.drop(new_df_sample.index)<jupyter_output><empty_output><jupyter_text>**Change the scheduled time of dep and arr to decimals**<jupyter_code># string to time tuple TRAIN
date_str_dep = list(data_train.SCHED_DEP)
date_str_arr = list(data_train.SCHED_ARR)
time_tuples_dep = []
time_tuples_arr = []
for i,j in zip(date_str_dep,date_str_arr):
    time_tuple_dep = time.strptime(i, "%H:%M:%S")
    time_tuple_arr = time.strptime(j, "%H:%M:%S")
    time_tuples_dep.append(time_tuple_dep.tm_hour+time_tuple_dep.tm_min/60)
    time_tuples_arr.append(time_tuple_arr.tm_hour+time_tuple_arr.tm_min/60)
data_train['SCHED_DEP'] = np.array(time_tuples_dep)
data_train['SCHED_ARR'] = np.array(time_tuples_arr)
# string to time tuple TEST
date_str_dep = list(data_test.SCHED_DEP)
date_str_arr = list(data_test.SCHED_ARR)
time_tuples_dep = []
time_tuples_arr = []
for i,j in zip(date_str_dep,date_str_arr):
    time_tuple_dep = time.strptime(i, "%H:%M:%S")
    time_tuple_arr = time.strptime(j, "%H:%M:%S")
    time_tuples_dep.append(time_tuple_dep.tm_hour+time_tuple_dep.tm_min/60)
    time_tuples_arr.append(time_tuple_arr.tm_hour+time_tuple_arr.tm_min/60)
data_test['SCHED_DEP'] = np.array(time_tuples_dep)
data_test['SCHED_ARR'] = np.array(time_tuples_arr)
# string to time REMAINING df
date_str_dep = list(remaining_df.SCHED_DEP)
date_str_arr = list(remaining_df.SCHED_ARR)
time_tuples_dep = []
time_tuples_arr = []
for i,j in zip(date_str_dep,date_str_arr):
    time_tuple_dep = time.strptime(i, "%H:%M:%S")
    time_tuple_arr = time.strptime(j, "%H:%M:%S")
    time_tuples_dep.append(time_tuple_dep.tm_hour+time_tuple_dep.tm_min/60)
    time_tuples_arr.append(time_tuple_arr.tm_hour+time_tuple_arr.tm_min/60)
remaining_df['SCHED_DEP'] = np.array(time_tuples_dep)
remaining_df['SCHED_ARR'] = np.array(time_tuples_arr)
#check we converted time to decimals correctly in SCHED_DEP and SCHED_ARR features
data_train.head()<jupyter_output><empty_output><jupyter_text>**Let's take a look at how unbalanced our data (delayed/not delayed) is - I see that only ~10% of observations are delayed flights.**<jupyter_code>print(data_train[data_train.DELAY_OR_NOT==1].shape)
print(data_train[data_train.DELAY_OR_NOT==0].shape)<jupyter_output>(6929, 1323)
(60071, 1323)<jupyter_text>**Finally, before I start building out models, I need to standardize all of our continuous variables**<jupyter_code>p_data_train=data_train.copy(deep=True)
p_data_train.loc[:,"SCHED_DEP":"DISTANCE"]=(data_train.loc[:,"SCHED_DEP":"DISTANCE"]-data_train.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train.loc[:,"SCHED_DEP":"DISTANCE"].std()

#test set - use mean and std of the training set
p_data_test=data_test.copy(deep=True)
p_data_test.loc[:,"SCHED_DEP":"DISTANCE"]=(data_test.loc[:,"SCHED_DEP":"DISTANCE"]-data_train.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train.loc[:,"SCHED_DEP":"DISTANCE"].std()

p_remaining_df = remaining_df.copy(deep=True)
p_remaining_df.loc[:,"SCHED_DEP":"DISTANCE"]=(remaining_df.loc[:,"SCHED_DEP":"DISTANCE"]-data_train.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train.loc[:,"SCHED_DEP":"DISTANCE"].std()
p_data_train.shape<jupyter_output><empty_output><jupyter_text>**Now, I am ready to try different classification models**<jupyter_code>x_train = p_data_train.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_train = p_data_train.DELAY_OR_NOT

x_test = p_data_test.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_test = p_data_test.DELAY_OR_NOT<jupyter_output><empty_output><jupyter_text>**I will use the classification models comparison code from Tuesday section 8 which reviewed last years midterm. However, I will throw out the kNN classification since calculating Euclidean distances between over 1000 predictors would be too computationally expensive**<jupyter_code>#following lamda function scores predictions for two-way classifier
score = lambda model, x_train, y_train: pd.Series([model.score(x_train, y_train), 
                                                 model.score(x_train[y_train==0], y_train[y_train==0]),
                                                 model.score(x_train[y_train==1], y_train[y_train==1])], 
                                                index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])
def out_of_box_class(x,y):
    

    #Unweighted logistic regression
    unweighted_logistic = LogisticRegression(C=1000)
    unweighted_logistic.fit(x, y)

    unweighted_log_scores = score(unweighted_logistic, x, y)
    print('unweighted log')


    #Weighted logistic regression
    weighted_logistic = LogisticRegression(C=1000, class_weight='balanced')
    weighted_logistic.fit(x, y)

    weighted_log_scores = score(weighted_logistic, x, y)
    print('weighted log')


    #LDA
    lda = LDA()
    lda.fit(x, y)

    lda_scores = score(lda, x, y)
    print('lda')

    #QDA
    qda = QDA()
    qda.fit(x, y)

    qda_scores = score(qda, x, y)
    print('qda')

    #Decision Tree
    tree = DecisionTree(max_depth=50, class_weight='balanced', criterion='entropy')
    tree.fit(x, y)

    tree_scores = score(tree, x, y)
    print('tree')

    score_df = pd.DataFrame({'unweighted logistic': unweighted_log_scores,
                             'weighted logistic': weighted_log_scores,
                             'lda': lda_scores,
                             'qda': qda_scores,
                             'tree': tree_scores})
    return score_df
out_of_box_class(x_train,y_train)<jupyter_output>unweighted log
weighted log
lda
qda
tree<jupyter_text>**From running these accuracy tests, I can see that the data is likely not multivariate normally (MVN) distributed (see LDA/QDA accuracy score on the class 0/1). Thus, LDA/QDA is not a good fit given the fact they require MVN property. Additionally, the decision tree is clearly overfitting. Thus we will choose weighted logistic regression as our classification model. Further justification is that only ~10% of my training data has DELAY==1 response, thus there needs to be weight assigned to the "positive DELAY" samples to ensure better accuracy of our model. Finally, the overall score of the weighted logit regression (still with non-tuned C parameter) is fairly high.**<jupyter_code># Fit cross-validated weighted L2 logistic regression 
clf = LogisticRegressionCV(fit_intercept=True, penalty='l2',class_weight='balanced')
clf.fit(x_train, y_train)

y_train_hat = clf.predict(x_train)

print("Training set prediction accuracy of logistic regression =", accuracy_score(y_train, y_train_hat))
clf.Cs<jupyter_output><empty_output><jupyter_text>**After tuning the hyperparameter C on my weighted logistic regression, we got C = 10. In logistic regression, as seen in the class, C is the inverse of the regularization parameter $\lambda$.**<jupyter_code>print(x_test.shape,x_train.shape)
clf_test_score = score(clf,x_test,y_test)
print("Accuracy on the test set achieved:")
print(clf_test_score)
auc_logreg_test = metrics.roc_auc_score(y_test, clf.predict_proba(x_test)[:,1])
print("The AUC score on the test set auc_logreg",auc_logreg_test)<jupyter_output>The AUC score on the test set auc_logreg 0.820862632096<jupyter_text>**I was able to beat the 60% benchmark AUC accuracy on the test set with the chosen weighted logistic model (C=10).** **Question 5**(5pts) Given your model, comment on the importance of factors as related to whether a flight is delayed.<jupyter_code>#Let's display some coefficients of our weighted logistic regression
clf.coef_[0][0:7]
#the order of coefficients is the same as the order of columns in my x_train data set
x_train.head()<jupyter_output><empty_output><jupyter_text>**The importance of each feature on flight delays can be shown from the coefficient values. We have 1322 coefficients for each of 1322 features. Each coefficient should represent how important each feature is in predicting whether the flight will be delayed or not. For example, looking at the first 6 coefficients, we have **| SCHED_DEP |SCHED_ARR|DISTANCE|MONTH_1|MONTH_2|MONTH_3|MONTH_4|---|---|---|| 0.94856692|0.03805954| 0.02356447|0.42227514|0.60139052|0.11210393|-0.21800069**And we can clearly see that the SCHED_DEP time is the most important predictor for the events of delayed flights among the displayed features while SCHED_ARR and DISTANCE are not that important and MONTH_4 is the least important. This makes sense with my visualisation that the percentage of delayed flights is much higher in the evening than in the morning. If the coefficient is negative, that means that the particular predictor is the least important (the least contribution of the beta coefficient in the exponential form) ****We could do a bootstrapping of our train data to get e.g. 100 values for each coefficient to get confidence intervals for our coefficients, but this would be too computationally expensive (we have >1,300 features and doing a LogisticRegressionCV one time takes ~10 minutes and I have only 36 hours (960 minutes) to complete the midterm!!)**<jupyter_code>#Now let's take a look at which predictors are most important in predicting a delay
#and which ones are the least importnat
X = x_train.columns
Y= clf.coef_[0]

Z = [x for _,x in sorted(zip(Y,X))]

print("Several features the least important for prediction of delay \n(chose most negative coefficients in the logit regression)")
print(Z[0:10])
print("\n")
print("Most important features predicting delay events (chose most positive coefficients in the logit regression)")
print(Z[-10:])<jupyter_output>Several features the least important for prediction of delay 
(chose most negative coefficients in the logit regression)
['DESTINATION_AIRPORT_LFT', 'DESTINATION_AIRPORT_12191', 'ORIGIN_AIRPORT_11697', 'MONTH_9', 'ORIGIN_AIRPORT_TLH', 'DESTINATION_AIRPORT_DLH', 'ORIGIN_AIRPORT_MGM', 'ORIGIN_AIRPORT_15016', 'ORIGIN_AIRPORT_ABI', 'ORIGIN_AIRPORT_11618']


Most important features predicting delay events (chose most positive coefficients in the logit regression)
['DESTINATION_AIRPORT_14842', 'ORIGIN_AIRPORT_10529', 'ORIGIN_AIRPORT_13244', 'ORIGIN_AIRPORT_12441', 'ORIGIN_AIRPORT_12954', 'DESTINATION_AIRPORT_11057', 'DESTINATION_AIRPORT_12173', 'AIRLINE_NK', 'DESTINATION_AIRPORT_LGA', 'DESTINATION_AIRPORT_12478']<jupyter_text>**As we can see from above, we again confirm our visualisation that month 9 has little importance in predicting delay events (on par with month 10) in accordance with the outcome of the weighted logistic model.** **QUESTION 6**(5pts) Evaluate your model(s) on your test set, and finally provide a visual to show which airlines are predicted to have the most delays using all the data excluding the training and test set. **The model was already evaluated on the test set in the end of questions 4 but I will list it here again.**<jupyter_code>clf_test_score = score(clf,x_test,y_test)
print("Accuracy on the test set achieved:")
print("Accuracy", clf_test_score)
print("The AUC score on the test set auc_logreg",auc_logreg_test)
#Visual showing which airlines are predicted to have the most delays using all data except training and test set

p_remaining_df.head()
y_test_remain = p_remaining_df.DELAY_OR_NOT
x_test_remain = p_remaining_df.drop('DELAY_OR_NOT',axis=1)

#now predict on the full remaining data_set
y_pred_remain = clf.predict(x_test_remain)
score(clf,x_test_remain,y_test_remain)
auc_logreg_rem = metrics.roc_auc_score(y_test_remain, clf.predict_proba(x_test_remain)[:,1])
print("The AUC score for the remaining whole data set is",auc_logreg_rem)<jupyter_output>The AUC score for the remaining whole data set is 0.823005059613<jupyter_text>**The accuracy of prediction achieved on the remaining full dataset is 0.754205 and AUC is 0.823**<jupyter_code>print("Predicted delayed",sum(y_pred_remain==1),"; true delayed",sum(y_test_remain))
original_dataframe = new_df.drop(new_df_sample.index) #getting the rest of indices not used in train/test
predicted_dataframe = new_df.drop(new_df_sample.index) #getting the rest of indices not used in train/test

sns.countplot(x='AIRLINE',hue='DELAY_OR_NOT',data=original_dataframe, palette="Set1");
plt.xlabel("Airline")
plt.ylabel("Truth value count of delayed flights")
plt.title("Truth values");
#Let's look at the mean delay % for each airline (truth)
original_dataframe.groupby("AIRLINE").mean()
predicted_dataframe['DELAY_OR_NOT'] = y_pred_remain
sns.countplot(x='AIRLINE',hue='DELAY_OR_NOT',data=predicted_dataframe,palette="Set1");
plt.xlabel("Airline")
plt.ylabel("Predicted count of delayed flights")
plt.title("Prediction"); 
#Let's look at the mean delay % for each airline (PREDICTION)
predicted_dataframe.groupby("AIRLINE").mean()
#Now take a look at how many flights we classify FALSE POSITIVE and FALSE NEGATIVE
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_curve
print(confusion_matrix(y_test_remain,y_pred_remain))

tn_logreg1, fp_logreg1, fn_logreg1, tp_logreg1 = metrics.confusion_matrix(y_test_remain,y_pred_remain).astype('float').ravel()

print("\nThe true positive rate for the weighted logistic regression model:",tp_logreg1/(tp_logreg1 + fn_logreg1))<jupyter_output>[[478467 154626]
 [ 18645  53203]]

The true positive rate for the weighted logistic regression model: 0.740493820287<jupyter_text>**Comparing the two plots above (truth and prediction, respectively) we see that our model predicts more delayed flights than there actually are: Predicted delayed 207829 (TP:53203,FP:154626), True delayed total 71,848 out of 700,000 flights in the remaining dataset (see the confusion matrix above). In other words, we have more FPRs than FNRs which makes sense as our data is highly unbalanced towards not delayed flights and our model is balanced in a way to put extra weight on the positive classification.** **E.g. for airlines "NK" and "F9", more flights are predicted to be delayed than on time, than in the truth baseline. However, this is to be expected given our AUC of the classification model is ~80%.****Airlines with most delays: 'NK' has 76%/37% delayed flights (predicted/ground truth) followed by 'F9': 60%/22% and 'B6': 47%/16% . We can see that the percentage of delayed flights predicted is higher than ground truth in all cases, which is consistent with the high false positive ratio I have with my model.****Speculation: if the user of my model preferred to get more false positives/false negatives (depending whether the user is more concerned about learning that a flight is predicted to be delayed when it's on time (FPR) or a flight is predicted to be on time when it is delayed - FNR. I would imagine for e.g., a businessman, being on the side of caution and having more FPRs than FNRs and taking precations to not e.g., miss a business meeting, might be a more useful use-case.** **Question 7**(15pts) Build a regression model that predicts the length of delay (on the log scale) given that a flight is truly delayed.<jupyter_code>#First, we want to look at lenght of flight GIVEN the flight is delayed - only select observations when a flight is truly delayed
new_df_regre = df.copy(deep=True)[df.DELAY_OR_NOT==1]
#drop everything as it was, but keep ARRIVAL_TIME as that will be our response variable

del new_df_regre['DATE']
del new_df_regre['SCHEDULED_TIME']
del new_df_regre['ELAPSED_TIME']
del new_df_regre['FLIGHT_NUMBER']
del new_df_regre['TAIL_NUMBER']

del new_df_regre['DEPARTURE_DELAY']
del new_df_regre['AIR_SYSTEM_DELAY']
del new_df_regre['AIRLINE_DELAY']
del new_df_regre['SECURITY_DELAY']
del new_df_regre['LATE_AIRCRAFT_DELAY']
del new_df_regre['WEATHER_DELAY']
new_df_regre.head()
new_df_regre.shape
#Change time to decimal in scheduled dep and arr
date_str_dep = list(new_df_regre.SCHED_DEP)
date_str_arr = list(new_df_regre.SCHED_ARR)
time_tuples_dep = []
time_tuples_arr = []
for i,j in zip(date_str_dep,date_str_arr):
    time_tuple_dep = time.strptime(i, "%H:%M:%S")
    time_tuple_arr = time.strptime(j, "%H:%M:%S")
    time_tuples_dep.append(time_tuple_dep.tm_hour+time_tuple_dep.tm_min/60)
    time_tuples_arr.append(time_tuple_arr.tm_hour+time_tuple_arr.tm_min/60)
new_df_regre['SCHED_DEP'] = np.array(time_tuples_dep)
new_df_regre['SCHED_ARR'] = np.array(time_tuples_arr)
#Check we have continuos time in decimals (hours) and that we only have DELAYED == True observations
new_df_regre.head()
#create a log response variable ARRIVAL TIME
new_df_regre['ARRIVAL_DELAY']=np.log(new_df_regre['ARRIVAL_DELAY'])
#Now, we need to convert our categorical variables
cols_to_change =['MONTH','DAY','DAY_OF_WEEK','AIRLINE','ORIGIN_AIRPORT','DESTINATION_AIRPORT']

new_df_dummy_regre = pd.get_dummies(new_df_regre,columns=cols_to_change)

#Subsample the data to reduce the # of observations. 
new_df_sample_regre = new_df_dummy_regre.sample(n=16000,random_state=6)

#Split the data in test and training (see Lecture 5)
np.random.seed(9001)
data_train_regre, data_test_regre = train_test_split(new_df_sample_regre,test_size=0.33, random_state=0)

#Check we actually got the right shapes
print(new_df_sample_regre.shape,df.shape,data_train_regre.shape,data_test_regre.shape)
data_train_regre.head()
#now we standardize our continuous variables
p_data_train_regre = data_train_regre.copy(deep=True)

p_data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"] = (data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"] - data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"].std()

#test set - use mean and std of the training set
p_data_test_regre=data_test_regre.copy(deep=True)
p_data_test_regre.loc[:,"SCHED_DEP":"DISTANCE"]=(data_test_regre.loc[:,"SCHED_DEP":"DISTANCE"]-data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train_regre.loc[:,"SCHED_DEP":"DISTANCE"].std()
#create x-train, y-train
x_train_regre = p_data_train_regre.copy(deep=True).drop('ARRIVAL_DELAY',axis=1)
y_train_regre = p_data_train_regre.ARRIVAL_DELAY

x_test_regre = p_data_test_regre.copy(deep=True).drop('ARRIVAL_DELAY',axis=1)
y_test_regre = p_data_test_regre.ARRIVAL_DELAY
#We create our regression model by performing regularized linear regression - Lasso.

Lasso_pipe = Pipeline([ ('mlr', LassoCV(cv=3)
                           )])
Lasso_pipe.fit(x_train_regre, y_train_regre)
from sklearn.metrics import r2_score

y_pred_train = Lasso_pipe.predict(x_train_regre)
y_pred = Lasso_pipe.predict(x_test_regre)

print("R^2 accuracy prediction on the training set")
print(r2_score(y_train_regre,y_pred_train))
print("R^2 accuracy prediction on the test set")
print(r2_score(y_test_regre,y_pred))<jupyter_output>R^2 accuracy prediction on the training set
0.0538784085414
R^2 accuracy prediction on the test set
0.0327098717967<jupyter_text>**I achieved R^2 over 3% on the test set so I am in the clear! I used Lasso regression with cross validation as it allows to regularize features coefficients and also removes not-significant coefficients.**  209 Additional questions(10pts) Engineer two additional features that will help improve the classification model's performance.  First engineered feature<jupyter_code>#Let's take a look to find how some of my features are correlated.
x_train.columns[53:67]
add = [i for i in range(53,100)]

features_to_look_at = [i for i in range(15)]
for i in add:
    features_to_look_at.append(i)
a=np.corrcoef(x_train.iloc[:,features_to_look_at].T)
plt.figure(figsize=(20,15))
sns.heatmap(a,xticklabels=list(x_train.iloc[:,features_to_look_at].columns), yticklabels=list(x_train.iloc[:,features_to_look_at].columns));<jupyter_output><empty_output><jupyter_text>**It is hard to tell which features are correlated from the correlation matrix above (we can only display a small subset of the predictors given our predictor space is too large. We get some blank values since certain airports do not have enough flights to calculate the correlation matrix**<jupyter_code>#We will add the interaction feature between day of the week - Thursday and scheduled departure time. 
#We saw in the visualisation that Thursday has the largest % of delayed flights
#and also that evening-hours flights have a higher percentage of delayed than morning ones
#It is computationally not viable to go through all the different interaction terms (even of degree 2), given our large feature space
#So we have to think about which combinations might make logical sense.

x_train_eng1 = p_data_train.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_train_eng1 = p_data_train.DELAY_OR_NOT

x_test_eng1 = p_data_test.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_test_eng1 = p_data_test.DELAY_OR_NOT
#Creating the interaction featured between Thursday and time of departure
x_train_eng1['inter_time_day']= x_train_eng1.DAY_OF_WEEK_4*x_train_eng1.SCHED_DEP
x_test_eng1['inter_time_day']= x_test_eng1.DAY_OF_WEEK_4*x_test_eng1.SCHED_DEP
# Fit cross-validated weighted 'L2'-penalized logistic regression 
clf_eng = LogisticRegressionCV(fit_intercept=True, penalty='l2',class_weight='balanced')
clf_eng.fit(x_train_eng1, y_train_eng1)

y_train_hat = clf_eng.predict(x_train_eng1)

print("Training set prediction accuracy of logistic regression =", accuracy_score(y_train_eng1, y_train_hat))
auc_logreg_eng = metrics.roc_auc_score(y_test_eng1, clf_eng.predict_proba(x_test_eng1)[:,1])
print("The AUC score without engineered features on the test set ",auc_logreg_test)
print("The AUC score with a engineered feature on the test set",auc_logreg_eng)<jupyter_output>The AUC score without engineered features on the test set  0.820862632096
The AUC score with a engineered feature on the test set 0.821622046718<jupyter_text>**We achieved slight improvemenent from AUC of 0.821 to 0.822 on the test set, which may be small, but given the fact we have over 1300 features, this interaction feature still adds a bit of accuracy. I observed most delayed flights happen on Thursdays and also in the evening from the visualisation, thus combining these two features added a bit of classification accuracy makes a logical sense.**<jupyter_code>#Now test the model with the engineered feature on the entire dataset
x_test_remain['inter_time_day'] = x_test_remain.DAY_OF_WEEK_4*x_test_remain.SCHED_DEP
#Calculate the AUC score
auc_logreg_eng_rem = metrics.roc_auc_score(y_test_remain, clf_eng.predict_proba(x_test_remain)[:,1])
print("AUC on the remaining data without engineered features",auc_logreg_rem)
print("AUC on the remaining data WITH a engineered features",auc_logreg_eng_rem)<jupyter_output>AUC on the remaining data without engineered features 0.823005059613
AUC on the remaining data WITH a engineered features 0.822875047292<jupyter_text>**Despite achieving a minimal improvement with the first engineered feature on the test set, we actually did slightly worse on the remaining dataset.**  Second engineered feature<jupyter_code>#Let's get rid of the airports which have less than 100 flights in a year
# - to reduce the complexity of our model by engineering the ORIGIN_AIRPORT feature

new_df_eng = df.copy(deep=True)
del new_df_eng['DATE']
del new_df_eng['SCHEDULED_TIME']
del new_df_eng['ELAPSED_TIME']
del new_df_eng['FLIGHT_NUMBER']
del new_df_eng['TAIL_NUMBER']
del new_df_eng['ARRIVAL_DELAY']
del new_df_eng['DEPARTURE_DELAY']
del new_df_eng['AIR_SYSTEM_DELAY']
del new_df_eng['AIRLINE_DELAY']
del new_df_eng['SECURITY_DELAY']
del new_df_eng['LATE_AIRCRAFT_DELAY']
del new_df_eng['WEATHER_DELAY']
#Change time to decimal in scheduled dep and arr
date_str_dep = list(new_df_eng.SCHED_DEP)
date_str_arr = list(new_df_eng.SCHED_ARR)
time_tuples_dep = []
time_tuples_arr = []
for i,j in zip(date_str_dep,date_str_arr):
    time_tuple_dep = time.strptime(i, "%H:%M:%S")
    time_tuple_arr = time.strptime(j, "%H:%M:%S")
    time_tuples_dep.append(time_tuple_dep.tm_hour+time_tuple_dep.tm_min/60)
    time_tuples_arr.append(time_tuple_arr.tm_hour+time_tuple_arr.tm_min/60)
new_df_eng['SCHED_DEP'] = np.array(time_tuples_dep)
new_df_eng['SCHED_ARR'] = np.array(time_tuples_arr)
#Now, we need to convert our categorical variables
cols_to_change =['MONTH','DAY','DAY_OF_WEEK','AIRLINE','ORIGIN_AIRPORT','DESTINATION_AIRPORT']

new_df_dummy = pd.get_dummies(new_df_eng,columns=cols_to_change)

#get the airports with only more than 100 flights
indices_to_keep = new_df.ORIGIN_AIRPORT.value_counts()[list(new_df.ORIGIN_AIRPORT.value_counts()>100)].index
collection = []
for c in new_df_dummy.columns:
    if "ORIGIN_AIRPORT" in c or "DESTINATION_AIRPORT" in c:
        tag = c.replace("ORIGIN_AIRPORT_", "").replace("DESTINATION_AIRPORT_","")
        if not tag in indices_to_keep:
            collection.append(c)
new_df_dummy=new_df_dummy.drop(collection, axis=1)
#Subsample the data to reduce the # of observations. 
new_df_sample = new_df_dummy.sample(n=100000,random_state=6)

#Split the data in test and training (see Lecture 5)
np.random.seed(9001)
data_train_eng, data_test_eng = train_test_split(new_df_sample,test_size=0.33, random_state=0)

#Check we actually got the right shapes
print(new_df_sample.shape,df.shape,data_train_eng.shape,data_test_eng.shape)
data_train_eng.head()
#now we standardize our continuous variables
p_data_train_eng = data_train_eng.copy(deep=True)

p_data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"] = (data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"] - data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"].std()

#test set - use mean and std of the training set
p_data_test_eng=data_test_eng.copy(deep=True)
p_data_test_eng.loc[:,"SCHED_DEP":"DISTANCE"]=(data_test_eng.loc[:,"SCHED_DEP":"DISTANCE"]-data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"].mean())/data_train_eng.loc[:,"SCHED_DEP":"DISTANCE"].std()
p_data_train_eng.head()
x_train_eng2 = p_data_train_eng.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_train_eng2 = p_data_train_eng.DELAY_OR_NOT

x_test_eng2 = p_data_test_eng.copy(deep=True).drop('DELAY_OR_NOT',axis=1)
y_test_eng2 = p_data_test_eng.DELAY_OR_NOT
# Fit cross-validated weighted L2 logistic regression 
clf_eng2 = LogisticRegressionCV(fit_intercept=True, penalty='l2',class_weight='balanced')
clf_eng2.fit(x_train_eng2, y_train_eng2)

y_train_hat = clf_eng2.predict(x_train_eng2)

print("Training set prediction accuracy of logistic regression =", accuracy_score(y_train_eng2, y_train_hat))
auc_logreg_eng2 = metrics.roc_auc_score(y_test_eng2, clf_eng2.predict_proba(x_test_eng2)[:,1])
print("The AUC score without engineered features on the test set ",auc_logreg_test)
print("The AUC score with a engineered feature (number of airports) on the test set",auc_logreg_eng2)<jupyter_output>The AUC score without engineered features on the test set  0.820862632096
The AUC score with a engineered feature (number of airports) on the test set 0.821096182128<jupyter_text>**We can see we improved our accuracy by engineering the "ORIGIN_AIRPORTS" feature by a bit, but most importantly we reduced the feature space from 1,300 to about 700, which improves the computational speed significantly.** (5pts) Add one additional feature from a data source not given to you. Do this only after you complete the rest of the exam. **Given lack of time, I will just add the "diverted" occurences feature and only look at January as I won't have enough time to clean and concatenate all of the months. I will add the feature called "DIVERTED" to see whether DIVERTED affected delay flights**<jupyter_code>#read the dataset I downloaded from https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time
jan_df = pd.read_csv("Jan_15_flights.csv")
jan_df.head()
#fill in for 0s, we will only be using ARR_DEL15 as our response which has NaNs, other variables with NaN;s will be dropped.
my_df = jan_df.fillna(0)
my_df.columns
del my_df['ACTUAL_ELAPSED_TIME']
del my_df['DIV_AIRPORT_LANDINGS'] #duplicate to DIVERTED 
del my_df['Unnamed: 13']
my_df.head()

cols_to_change =['MONTH','DAY_OF_MONTH','DAY_OF_WEEK','AIRLINE_ID','ORIGIN','DEST']

#hot-encode
new_df_dummy = pd.get_dummies(my_df,columns=cols_to_change)

#Subsample the data to reduce the # of observations. 
new_df_sample = new_df_dummy.sample(n=50000,random_state=6)

#Split the data in test and training (see Lecture 5)
np.random.seed(9001)
data_train, data_test = train_test_split(new_df_sample,test_size=0.33, random_state=0)

#Check we actually got the right shapes
print(new_df_sample.shape,df.shape,data_train.shape,data_test.shape)
data_train.head()
#standardize the data
p_data_train=data_train.copy(deep=True)
p_data_train.iloc[:,[0,1,4]]=(data_train.iloc[:,[0,1,4]]-data_train.iloc[:,[0,1,4]].mean())/data_train.iloc[:,[0,1,4]].std()

#test set - use mean and std of the training set
p_data_test=data_test.copy(deep=True)
p_data_test.iloc[:,[0,1,4]]=(data_test.iloc[:,[0,1,4]]-data_train.iloc[:,[0,1,4]].mean())/data_train.iloc[:,[0,1,4]].std()
p_data_train.head()
x_train = p_data_train.copy(deep=True).drop('ARR_DEL15',axis=1)
y_train = p_data_train.ARR_DEL15

x_test = p_data_test.copy(deep=True).drop('ARR_DEL15',axis=1)
y_test = p_data_test.ARR_DEL15
# Fit cross-validated weighted L2 logistic regression 
clf = LogisticRegressionCV(fit_intercept=True, penalty='l2',class_weight='balanced')
clf.fit(x_train, y_train)

y_train_hat = clf.predict(x_train)

print("Training set prediction accuracy of logistic regression =", accuracy_score(y_train, y_train_hat))
clf_test_score = score(clf,x_test,y_test)
print("Accuracy on the test set achieved:")
print(clf_test_score)
auc_logreg_test = metrics.roc_auc_score(y_test, clf.predict_proba(x_test)[:,1])
print("The AUC score on the test set auc_logreg",auc_logreg_test)<jupyter_output>The AUC score on the test set auc_logreg 0.711382941648
StarcoderdataJupyterStructuredCleanDedup