<jupyter_start><jupyter_text>My first model is overfitting.  New Model<jupyter_code>Y = titanic_df['Survived']

X = titanic_df[['age*fare', 'Parch', 'Embarked', 'Fare']]

lrm = linear_model.LinearRegression()

lrm.fit(X, Y)

print('\nCoefficients: \n', lrm.coef_)
print('\nIntercept: \n', lrm.intercept_)
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.2, random_state = 465)

print("The number of observations in training set is {}".format(X_train.shape[0]))
print("The number of observations in test set is {}".format(X_test.shape[0]))
# We add constant to the model as it's a best practice
# to do so everytime!
X_train = sm.add_constant(X_train)

# We fit an OLS model using statsmodels
results = sm.OLS(y_train, X_train).fit()

# We print the summary results
display(results.summary())
X_test = sm.add_constant(X_test)

# We are making predictions here
y_preds = results.predict(X_test)

plt.scatter(y_test, y_preds)
plt.plot(y_test, y_test, color="red")
plt.xlabel("true values")
plt.ylabel("predicted values")
plt.title("Charges: true and predicted values")
plt.show()

print("Mean absolute error of the prediction is: {}".format(mean_absolute_error(y_test, y_preds)))
print("Mean squared error of the prediction is: {}".format(mse(y_test, y_preds)))
print("Root mean squared error of the prediction is: {}".format(rmse(y_test, y_preds)))
print("Mean absolute percentage error of the prediction is: {}".format(np.mean(np.abs((y_test - y_preds) / y_test)) * 100))
from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression(solver='lbfgs', multi_class="ovr")
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test =  train_test_split(X, Y, test_size=0.20, random_state=111)
log_reg.fit(X_train, y_train)
train_accuracy = log_reg.score(X_train, y_train)
test_accuracy = log_reg.score(X_test, y_test)

print('One-vs.-Rest', '-'*30, 
      'Accuracy on Train Data : {:.2f}'.format(train_accuracy), 
      'Accuracy on Test Data  : {:.2f}'.format(test_accuracy), sep='\n')
log_reg_mnm = LogisticRegression(multi_class='multinomial', solver='lbfgs')
log_reg_mnm.fit(X_train, y_train)

train_accuracy = log_reg_mnm.score(X_train, y_train)
test_accuracy = log_reg_mnm.score(X_test, y_test)

print('Multinomial (Softmax)', '-'*20, 
      'Accuracy on Train Data : {:.2f}'.format(train_accuracy), 
      'Accuracy on Test Data  : {:.2f}'.format(test_accuracy), sep='\n')<jupyter_output>Multinomial (Softmax)
--------------------
Accuracy on Train Data : 0.66
Accuracy on Test Data  : 0.65<jupyter_text>Now my train and test scores came out as I wanted<jupyter_code>C_values = [0.001,0.01, 0.1,1,10,100, 1000]

accuracy_values = pd.DataFrame(columns=['C_values', 'Train Accuracy', 'Test Accuracy'])

for c in C_values:
    # Apply logistic regression model to training data
    lr = LogisticRegression(penalty = 'l2', C = c, random_state = 0, solver='lbfgs', multi_class='multinomial')
    lr.fit(X_train, y_train)
    accuracy_values = accuracy_values.append({'C_values': c,
                                              'Train Accuracy': lr.score(X_train, y_train),
                                              'Test Accuracy': lr.score(X_test, y_test)
                                             }, ignore_index=True)
display(accuracy_values)<jupyter_output><empty_output><jupyter_text>It looks like C=0.001 is the best option for this model.  Performance Metrics<jupyter_code>from sklearn import datasets

titanic = titanic_df
X_train, X_test, y_train, y_test =  train_test_split(X, Y, test_size=0.20, random_state=111)

titanic_prediction_model = LogisticRegression()
titanic_prediction_model.fit(X_train, y_train)

train_prediction = titanic_prediction_model.predict(X_train)
test_prediction = titanic_prediction_model.predict(X_test)

test_prediction_probability = titanic_prediction_model.predict_proba(X_test)[:,1]
from sklearn.metrics import confusion_matrix

confusion_matrix_train = confusion_matrix(y_train, train_prediction)
confusion_matrix_test = confusion_matrix(y_test, test_prediction)

print("Confusion Matrix (Train Data)", "-"*30, confusion_matrix_train, sep="\n")
print("\n\nConfusion Matrix (Test Data)", "-"*30, confusion_matrix_test, sep="\n")
TN = confusion_matrix_test[0][0]
TP = confusion_matrix_test[1][1]
FN = confusion_matrix_test[1][0]
FP = confusion_matrix_test[0][1]

print("True negative amount  :", TN)
print("True positive amount  :", TP)
print("False positive amount :", FP)
print("False negative amount :", FN)<jupyter_output>True negative amount  : 100
True positive amount  : 16
False positive amount : 13
False negative amount : 50<jupyter_text>Accuracy<jupyter_code>from sklearn.metrics import accuracy_score

print("Accuracy value by the model   : ",  titanic_prediction_model.score(X_test, y_test))
print("Calculated accuracy value     : ",  (TN + TP)/(FN + FP + TN + TP))
print("accuracy_score() value        : ",  accuracy_score(y_test, test_prediction))<jupyter_output>Accuracy value by the model   :  0.6480446927374302
Calculated accuracy value     :  0.6480446927374302
accuracy_score() value        :  0.6480446927374302<jupyter_text>Error Rate<jupyter_code>print("Error rate value by the model   : ", 1 - titanic_prediction_model.score(X_test, y_test))
print("Calculated error rate     : ", 1 - (TN + TP)/(FN + FP + TN + TP))
print("error_rate_score() value        : ", 1 - accuracy_score(y_test, test_prediction))<jupyter_output>Error rate value by the model   :  0.35195530726256985
Calculated error rate     :  0.35195530726256985
error_rate_score() value        :  0.35195530726256985<jupyter_text>Precision<jupyter_code>from sklearn.metrics import precision_score

print("Calculated precision value   : ",  (TP)/(FP + TP))
print("precision_score() value      : ",  precision_score(y_test, test_prediction))<jupyter_output>Calculated precision value   :  0.5517241379310345
precision_score() value      :  0.5517241379310345<jupyter_text>Recall<jupyter_code>from sklearn.metrics import recall_score

print("Calculated recall value   : ",  (TP)/(TP + FN))
print("recall_score() value      : ",  recall_score(y_test, test_prediction))<jupyter_output>Calculated recall value   :  0.24242424242424243
recall_score() value      :  0.24242424242424243<jupyter_text>Specificity<jupyter_code>print("Calculated specificity value   : ",  (TN)/(TN + FP))
from sklearn.metrics import f1_score

precision = precision_score(y_test, test_prediction)
recall = recall_score(y_test, test_prediction)


print("Calculated F1 score  : ",  2*((recall*precision)/(recall + precision)))
print("f1_score() value     : ",  f1_score(y_test, test_prediction))<jupyter_output>Calculated F1 score  :  0.33684210526315794
f1_score() value     :  0.33684210526315794<jupyter_text>General View<jupyter_code>from sklearn.metrics import classification_report, precision_recall_fscore_support

print(classification_report(y_test, test_prediction))

print("f1_score() value        : {:.2f}".format(f1_score(y_test, test_prediction)))
print("recall_score() value    : {:.2f}".format(recall_score(y_test, test_prediction)))
print("precision_score() value : {:.2f}".format(precision_score(y_test, test_prediction)))
print('\n')

metrics = precision_recall_fscore_support(y_test, test_prediction)
print("Precision  :" , metrics[0]) 
print("Recall     :" , metrics[1]) 
print("F1 Score   :" , metrics[2])<jupyter_output>precision    recall  f1-score   support

           0       0.67      0.88      0.76       113
           1       0.55      0.24      0.34        66

    accuracy                           0.65       179
   macro avg       0.61      0.56      0.55       179
weighted avg       0.62      0.65      0.60       179

f1_score() value        : 0.34
recall_score() value    : 0.24
precision_score() value : 0.55


Precision  : [0.66666667 0.55172414]
Recall     : [0.88495575 0.24242424]
F1 Score   : [0.76045627 0.33684211]<jupyter_text>I messed up while predicting the survivors. Fortunately, I can guess better whom died :)  ROC / AUC<jupyter_code>test_prediction_proba = titanic_prediction_model.predict_proba(X_test)[:,1]
from sklearn.metrics import roc_curve, roc_auc_score

fpr, tpr, thresholds  = roc_curve(y_test, test_prediction_proba)

import matplotlib.pyplot as plt
# Plot ROC curve
plt.plot([0, 1], [0, 1], 'k--')
plt.plot(fpr, tpr)

plt.xlabel('False Positive Ratio')
plt.ylabel('True Positive Ratio')
plt.title('ROC Curve')

plt.show()

print('AUC value : ', roc_auc_score(y_test, test_prediction_proba))<jupyter_output>AUC value :  0.6924108340037544<jupyter_text>This AUC score is far better than I expected.  Precision/Recall Curves<jupyter_code>from sklearn.metrics import precision_recall_curve

precision, recall, _ = precision_recall_curve(y_test, test_prediction_proba)

plt.plot(recall, precision)

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision / Recall Curve')

plt.show()<jupyter_output><empty_output><jupyter_text>I made my first mistake at the very beginning, that's not good.  Logarithmic Loss<jupyter_code>from sklearn.metrics import log_loss

print("Logarithmic Loss (log-loss) : " , log_loss(y_test, test_prediction_proba))
print("Error Rate                  : " , 1- accuracy_score(y_test, test_prediction))<jupyter_output>Logarithmic Loss (log-loss) :  0.6230488109026635
Error Rate                  :  0.35195530726256985<jupyter_text>My model is pretty good at making a mistake :) Anyway, I like my model. The results are better than I've expected.  Different C values<jupyter_code>lr = LogisticRegression(penalty = 'l2', C = 0.001, random_state = 0, solver='lbfgs', multi_class='multinomial')
lr.fit(X_train, y_train)
Y = titanic_df['Survived']

X = titanic_df[['age*fare', 'Parch', 'Embarked', 'Fare']]

lrm = linear_model.LinearRegression()

lrm.fit(X, Y)

print('\nCoefficients: \n', lrm.coef_)
print('\nIntercept: \n', lrm.intercept_)<jupyter_output>Coefficients: 
 [-1.88622862e-05  1.53275069e-02  6.18025050e-02  3.13192177e-03]

Intercept: 
 0.29094274854881813