# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: venv-datascience
#     language: python
#     name: venv-datascience
# ---

# # Keras TF 2.0 - Classification Project
#
# There are 3 models that we will be demostrating.
#
# + [Example One: Choosing too many epochs and overfitting!](#overfit)
# + [Example Two: Early Stopping](#earlystop)
# + [Example Three: Adding in DropOut Layers](#dropout)
#
# ----
#
# ## The Data
#
# ### Breast cancer wisconsin (diagnostic) dataset
# --------------------------------------------
#
# **Data Set Characteristics:**
#
#     :Number of Instances: 569
#
#     :Number of Attributes: 30 numeric, predictive attributes and the class
#
#     :Attribute Information:
#         - radius (mean of distances from center to points on the perimeter)
#         - texture (standard deviation of gray-scale values)
#         - perimeter
#         - area
#         - smoothness (local variation in radius lengths)
#         - compactness (perimeter^2 / area - 1.0)
#         - concavity (severity of concave portions of the contour)
#         - concave points (number of concave portions of the contour)
#         - symmetry 
#         - fractal dimension ("coastline approximation" - 1)
#
#         The mean, standard error, and "worst" or largest (mean of the three
#         largest values) of these features were computed for each image,
#         resulting in 30 features.  For instance, field 3 is Mean Radius, field
#         13 is Radius SE, field 23 is Worst Radius.
#
#         - class:
#                 - WDBC-Malignant
#                 - WDBC-Benign
#
#     :Summary Statistics:
#
#     ===================================== ====== ======
#                                            Min    Max
#     ===================================== ====== ======
#     radius (mean):                        6.981  28.11
#     texture (mean):                       9.71   39.28
#     perimeter (mean):                     43.79  188.5
#     area (mean):                          143.5  2501.0
#     smoothness (mean):                    0.053  0.163
#     compactness (mean):                   0.019  0.345
#     concavity (mean):                     0.0    0.427
#     concave points (mean):                0.0    0.201
#     symmetry (mean):                      0.106  0.304
#     fractal dimension (mean):             0.05   0.097
#     radius (standard error):              0.112  2.873
#     texture (standard error):             0.36   4.885
#     perimeter (standard error):           0.757  21.98
#     area (standard error):                6.802  542.2
#     smoothness (standard error):          0.002  0.031
#     compactness (standard error):         0.002  0.135
#     concavity (standard error):           0.0    0.396
#     concave points (standard error):      0.0    0.053
#     symmetry (standard error):            0.008  0.079
#     fractal dimension (standard error):   0.001  0.03
#     radius (worst):                       7.93   36.04
#     texture (worst):                      12.02  49.54
#     perimeter (worst):                    50.41  251.2
#     area (worst):                         185.2  4254.0
#     smoothness (worst):                   0.071  0.223
#     compactness (worst):                  0.027  1.058
#     concavity (worst):                    0.0    1.252
#     concave points (worst):               0.0    0.291
#     symmetry (worst):                     0.156  0.664
#     fractal dimension (worst):            0.055  0.208
#     ===================================== ====== ======
#
#     :Missing Attribute Values: None
#
#     :Class Distribution: 212 - Malignant, 357 - Benign
#
#     :Creator:  Dr. <NAME>, <NAME>, <NAME>
#
#     :Donor: <NAME>
#
#     :Date: November, 1995
#
# This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
# https://goo.gl/U2Uwz2
#
# Features are computed from a digitized image of a fine needle
# aspirate (FNA) of a breast mass.  They describe
# characteristics of the cell nuclei present in the image.
#
# Separating plane described above was obtained using
# Multisurface Method-Tree (MSM-T) [<NAME>, "Decision Tree
# Construction Via Linear Programming." Proceedings of the 4th
# Midwest Artificial Intelligence and Cognitive Science Society,
# pp. 97-101, 1992], a classification method which uses linear
# programming to construct a decision tree.  Relevant features
# were selected using an exhaustive search in the space of 1-4
# features and 1-3 separating planes.
#
# The actual linear program used to obtain the separating plane
# in the 3-dimensional space is that described in:
# [<NAME> and <NAME>: "Robust Linear
# Programming Discrimination of Two Linearly Inseparable Sets",
# Optimization Methods and Software 1, 1992, 23-34].
#
# This database is also available through the UW CS ftp server:
#
# ftp ftp.cs.wisc.edu
# # cd math-prog/cpo-dataset/machine-learn/WDBC/
#
# .. topic:: References
#
#    - <NAME>, <NAME> and <NAME>. Nuclear feature extraction 
#      for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on 
#      Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
#      San Jose, CA, 1993.
#    - <NAME>, W.<NAME> and <NAME>. Breast cancer diagnosis and 
#      prognosis via linear programming. Operations Research, 43(4), pages 570-577, 
#      July-August 1995.
#    - <NAME>, W.N. Street, and <NAME>. Machine learning techniques
#      to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) 
#      163-171.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv('data/cancer_classification.csv')

df.head(2)

# # Exploratory Data Analysis

df.info()

df.describe().T

sns.countplot(data=df,x='benign_0__mal_1');

# ### Check the correlation

df.corr()['benign_0__mal_1'].sort_values(ascending=False)

plt.figure(figsize=(10, 5))
df.corr()['benign_0__mal_1'].sort_values()[:-1].plot(kind='bar');

plt.figure(figsize=(12, 8))
sns.heatmap(df.corr());

# ----

# # Train Test Split

X = df.drop('benign_0__mal_1', axis=1)
y = df['benign_0__mal_1']

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=101)

# # Scaling Data

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# # Modelling

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

X_train.shape # we have 30 features

model = Sequential()

# +
model.add(Dense(30, activation='relu'))
model.add(Dense(15, activation='relu'))

# Binary Classification => Sigmoid
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')
# -

# # Training the Model 
#
# # <a name='overfit'>Example One: Choosing too many epochs and overfitting!</a>

model.fit(x=X_train, y=y_train, 
             epochs=600,
             validation_data=(X_test, y_test),
             verbose=1)

losses = pd.DataFrame(model.history.history)

losses.plot();

# This is the perfect example of overfitting. We can see that initally, loss is starting to decrease on validation.
#
# However it get spiked up after a certain point in time of few epoches.

# -----

# # <a name='earlystop'>Example Two: Early Stopping</a>
#
# We obviously trained too much! Let's use early stopping to track the val_loss and stop training once it begins increasing too much!
#
# + **`monitor`**: Quantity to be monitored. Which we are monitoring such as `val_loss` or `accuracy`.
# + **`mode`**: if we are monitoring loss, we want to minimize it. So we use `min`. For the case of accuracy, we want to maximize it. So we use `max`
# + **` patience`**: Number of epochs with no improvement after which training will be stopped. It is a hard stopping.

# +
model = Sequential()

model.add(Dense(30, activation='relu'))
model.add(Dense(15, activation='relu'))

# Binary Classification => Sigmoid
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')
# -

from tensorflow.keras.callbacks import EarlyStopping

# +
#  help(EarlyStopping)
# -

early_stopping = EarlyStopping(monitor='val_loss', 
                                                  mode='min',
                                                  patience=25,
                                                  verbose=1)

model.fit(x=X_train, y=y_train,
             validation_data=(X_test, y_test),
             epochs=600,
             callbacks=[early_stopping],
             verbose=1)

losses = pd.DataFrame(model.history.history)

losses.plot();

# This is the plot that we want to see. Both loss and val_loss are beginning to decrease.

# ------

# # <a name='dropout'>Example Three: Adding in DropOut Layers</a>
#
# **Early Stopping**
#
# - Keras can automatically stop training based on a loss condition on the validation data passed during the model.fit call.
#
# **Dropout Layers**
#
# - can be added to layers to "turn off" neurons during training to prevent overfitting.
#
# - each dropout layer will "drop" a user-defined percentage of neuron units in the previous layer every batch.
#
# **We can define by `model.add(Dropout(0 ~ 1))`: values can between 0 and 1.**
#
# 0 percent , 1 mean 100% of neurons.

from tensorflow.keras.layers import Dropout

# +
model = Sequential()

model.add(Dense(30, activation='relu'))
model.add(Dropout(0.5)) # 50% of randomly selected neurons for each batch (1 entire epoch triaining) will be truned off

model.add(Dense(15, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')
# -

# in combination with early stopping callbacks
early_stopping = EarlyStopping(monitor='val_loss',
                                                  mode='min',
                                                  patience=25,
                                                  verbose=1)

model.fit(x=X_train, y=y_train,
             validation_data=(X_test, y_test),
             callbacks=[early_stopping],
             epochs=600)

# +
losses = pd.DataFrame(model.history.history)

losses.plot();
# -

# This is great. Both loss and val_loss are flattening out at the same time.

# ------
# -----

# # Model Evaluation

# +
predictions = model.predict_classes(X_test)

predictions[:5]
# -

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test, predictions))

confusion_matrix(y_test, predictions)

sns.heatmap(confusion_matrix(y_test, predictions), annot=True, cbar=False);

# -----

StarcoderdataJupyterScriptsDedupFiltered