<jupyter_start><jupyter_text>Utilised code from https://www.kaggle.com/harshsoni/ranzcr-simple-densenet121-approach to build this model *REMEMBER TO CHANGE RUNTIME TO GPU* ImportsImports and Installations<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
# pip install tensorflow_addons -q
# pip install tensorflow-gpu -q
# pip install pathlib
# inbuilt imports
import os
import glob
import pathlib
import tempfile
import functools

# numeric imports
import numpy as np
import pandas as pd

# visual imports
import seaborn as sns
import matplotlib.pyplot as plt
from IPython.display import Image

# modeling imports
import tensorflow as tf
import tensorflow_addons as tfa

import pathlib
%cd C:\Users\Johnn\Desktop\ADS2002\ranzcr-clip-catheter-line-classification
%matplotlib inline
sns.set()<jupyter_output><empty_output><jupyter_text>Define paths for image training later<jupyter_code>train_path = pathlib.Path('C:\\Users\\Johnn\\Desktop\\ADS2002\\ranzcr-clip-catheter-line-classification\\train')
test_path = pathlib.Path('C:\\Users\\Johnn\\Desktop\\ADS2002\\ranzcr-clip-catheter-line-classification\\test')<jupyter_output><empty_output><jupyter_text>Define constants<jupyter_code>IMG_SIZE = 224
BATCH_SIZE = 16

all_labels = ['ETT - Abnormal', 'ETT - Borderline', 'ETT - Normal', 
              'NGT - Abnormal', 'NGT - Borderline','NGT - Incompletely Imaged', 
              'NGT - Normal', 'CVC - Abnormal', 'CVC - Borderline', 'CVC - Normal', 
              'Swan Ganz Catheter Present']<jupyter_output><empty_output><jupyter_text>Define dataframes`train` provides one hot encoding for labelling. `train_coord` gives class counts and coordinates for catheters<jupyter_code>df = pd.read_csv("train.csv")
df_coord = pd.read_csv("train_annotations.csv")

df.head()
df_coord.head()<jupyter_output><empty_output><jupyter_text>Data PathsAdd the paths of images to the dataframe for easy access<jupyter_code>df['path'] = df['StudyInstanceUID'].map(lambda x: str(train_path/(x+'.jpg')))
df.iloc[0][-1]
df.head()<jupyter_output><empty_output><jupyter_text>EDA  ClassesData suggests an imbalance for each class.<jupyter_code>data = df.loc[:, all_labels].sum().sort_values()
dist = pd.DataFrame({'Class': data.index, 'Pos': data.values}, columns=['Class', 'Pos'])
dist
fig = plt.figure(figsize=(6, 5))

sns.barplot(data=dist, y='Class', x='Pos')
plt.xticks(rotation=45)
plt.show()
dist['Neg'] = len(df) - dist.Pos
dist.sort_values(by='Pos')<jupyter_output><empty_output><jupyter_text>`PatientID`Given that `PatientID` is a possible variable, it suggests a one to many relationship between one patient and many detections/data points.<jupyter_code>print(len(df))
print(len(df.PatientID.unique()))
grouped_df = df.set_index('PatientID')
grouped_df.head()
grouped_df.loc['ec89415d1'].sort_values(by=['StudyInstanceUID'])<jupyter_output><empty_output><jupyter_text>As the dataframe above suggests, there exists multiple records for a single patient.Hence, when selecting the images for training and validation, patients of the same ID need to be kept within their distinct splits. Otherwise data leakage occurs, and the model provide outputs without learning, due to already prelearning the output. Thus, for training/validation split, must group on `PatientID` and split accordingly. Splitting Data<jupyter_code>grouped_df = df.groupby('PatientID')
train_list = [group for _, group in grouped_df]<jupyter_output><empty_output><jupyter_text>`train_list` is a list of dataframes for each patient.`train_list[0]` gives dataframe for `PatientID` - 003e65ddb<jupyter_code>def train_valid_splitter(d, train_size=0.8):
    n = len(d)
    trains = d[:int(train_size*n)]
    valids = d[int(train_size*n):]
    return trains, valids

train_split, valid_split = train_valid_splitter(train_list)
train_df = pd.concat(train_split, axis=0)
valid_df = pd.concat(valid_split, axis=0)

print(f'Train Size: {len(train_df)}, Valid Size: {len(valid_df)}')
train_df<jupyter_output><empty_output><jupyter_text>Create Data GeneratorsUsing `ImagDataGenerator`, we can build the inital pipeline of the model.Image transforms include horizontally flipping and rotating (by 0.4 radians, just a randomly chosen number).<jupyter_code>train_generator = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255,
    horizontal_flip=True,
    rotation_range=0.40
)

train_datagen = train_generator.flow_from_dataframe(
    dataframe=train_df,
    x_col='path',
    y_col=all_labels,
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    seed=42,
    class_mode='raw'
)
valid_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255)

valid_datagen = valid_generator.flow_from_dataframe(
    dataframe=valid_df,
    x_col='path',
    y_col=all_labels,
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    seed=42,
    class_mode='raw'
)
def display_batch(batch, n_imgs=9):
    r = int(n_imgs**0.5)
    fig, axs = plt.subplots(r, r, figsize=(12, 15))
    imgs, labels = batch[0], batch[1]
    for i, ax in zip(range(n_imgs), axs.flatten()):
        title = '\n'.join(list(np.array(all_labels)[labels[i].flatten()==1]))
        ax.imshow(imgs[i], cmap='bone')
        ax.set_title(title)
        ax.grid(False)
        ax.set_xticks([])
        ax.set_yticks([])
    plt.show()
display_batch(next(train_datagen))<jupyter_output><empty_output><jupyter_text>Model Building<jupyter_code>pip install image-classifiers -q
from classification_models.tfkeras import Classifiers

ResNet18, proc_func = Classifiers.get('resnet18')
densenet = tf.keras.applications.DenseNet121(weights='imagenet', include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3))
densenet.trainable = False

inputs = densenet.inputs
x = densenet(inputs)

x = tf.keras.layers.GlobalAveragePooling2D()(x)
outputs = tf.keras.layers.Dense(len(all_labels), activation='sigmoid')(x)
model = tf.keras.Model(inputs, outputs)

model.summary()<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/densenet/densenet121_weights_tf_dim_ordering_tf_kernels_notop.h5
29089792/29084464 [==============================] - 3s 0us/step
29097984/29084464 [==============================] - 3s 0us/step
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
densenet121 (Functional)     (None, 7, 7, 1024)        7037504   
_________________________________________________________________
global_average_pooling2d (Gl (None, 1024)              0         
_________________________________________________________________
dense (Dense)                (None, 11)                11275     
===============================================[...]<jupyter_text>Learning RateBuild a learning rate finder to find an optimal lr for finetuning the model.<jupyter_code>class LRFinder:
    def __init__(self, model, lr_range=[1e-10, 1e1], beta=0.98, stop_factor=4):
        self.model = model
        self.lr_range = lr_range
        self.beta = beta
        self.stop_factor = stop_factor
        self.stop_training = False
        self.iterations = 0
        self.mvg_avg_loss = 0
        self.min_loss = 1e9
        self.lrs = []
        self.losses = []

    def _reset(self):
        self.stop_training = False
        self.iterations = 0
        self.mvg_avg_loss = 0
        self.min_loss = 1e9
        self.lrs = []
        self.losses = []

    def _scheduler(self, start_lr, end_lr, iterations):
        self.lr_factor = (end_lr / start_lr)**(1./iterations)

    def on_train_begin(self, logs=None):
        self._reset()

    def on_batch_end(self, batch, logs=None):
        self.iterations += 1

        lr = tf.keras.backend.get_value(self.model.optimizer.lr)
        self.lrs.append(lr)
        tf.keras.backend.set_value(self.model.optimizer.lr, lr*self.lr_factor)

        loss = logs['loss']
        self.mvg_avg_loss = (self.beta*self.mvg_avg_loss) + ((1-self.beta)*loss)
        smooth_loss = self.mvg_avg_loss / (1-(self.beta**self.iterations))
        self.losses.append(smooth_loss)

        stop_loss = self.stop_factor * self.min_loss
        if self.iterations > 1 and smooth_loss > self.stop_factor:
            self.stop_training = True

        if self.iterations == 0 or smooth_loss < self.min_loss:
            self.min_loss = smooth_loss
#         print(f'\nIterations: {self.iterations}, lr: {lr}, loss: {smooth_loss}/{loss}, lrf: {self.lr_factor}')

    def on_epoch_end(self, epoch, logs=None):
        if self.stop_training:
            self.model.stop_training = True
            return

    def find(self, train_ds, epochs=None, steps_per_epoch=None, batch_size=32):
        if epochs is None:
            raise ValueError(f'Invalid value {epochs} for epochs')

        if steps_per_epoch is None:
            steps_per_epoch = len(train_ds)
            
        self._scheduler(self.lr_range[0], self.lr_range[1], steps_per_epoch*epochs)

        with tempfile.NamedTemporaryFile(prefix='init', suffix='.h5') as init_config:
            # save model config
            self.model.save_weights(init_config.name)
            init_lr = tf.keras.backend.get_value(self.model.optimizer.lr)
            
            tf.keras.backend.set_value(self.model.optimizer.lr, self.lr_range[0])

            lr_finder_cb = tf.keras.callbacks.LambdaCallback(
                on_train_begin= lambda logs: self.on_train_begin(logs),
                on_batch_end= lambda batch, logs: self.on_batch_end(batch, logs),
                on_epoch_end= lambda epoch, logs: self.on_epoch_end(epoch, logs)
            )

            self.model.fit(train_ds, epochs=epochs, steps_per_epoch=steps_per_epoch,
                           callbacks=[lr_finder_cb])

            # restore model config
            tf.keras.backend.set_value(self.model.optimizer.lr, init_lr)
            self.model.load_weights(init_config.name)

    def plot_loss(self, skip_begin=10, skip_end=1, title=""):
        lrs = self.lrs[skip_begin:-skip_end]
        losses = self.losses[skip_begin:-skip_end]
        plt.plot(lrs, losses)
        plt.xscale("log")
        plt.xlabel("Learning Rate (Log Scale)")
        plt.ylabel("Loss")<jupyter_output><empty_output><jupyter_text>Compute lr<jupyter_code># model.compile(optimizer='adam',
#               loss='binary_crossentropy',
#               metrics=['binary_accuracy', tf.keras.metrics.AUC()])

# lr_finder = LRFinder(model)
# lr_finder.find(train_datagen, epochs=10)<jupyter_output><empty_output><jupyter_text>For our CLR we can use a minimum learning rate of 1e-2 and a maximum learning rate of 1e-5.`CyclicalLearningRate` provided by tensorflow_addons library, keep the initial learning rate to 1e-5 and the maximum learning rate to be 1e-2 and use a 'traingular' approach of CLR. We are keeping the step size = 2 epochs. Hence, to perform 1 cycle we will need to perform 4 epochs.<jupyter_code>def scale_fn(x):  return 1.
clr = tfa.optimizers.CyclicalLearningRate(
    initial_learning_rate=1e-5,
    maximal_learning_rate=1e-2,
    scale_fn=scale_fn,
    step_size=2*len(train_datagen)*BATCH_SIZE,
    scale_mode='cyclic'
)<jupyter_output><empty_output><jupyter_text>To evaluate our model we will use 'binary_accuracy' and 'AUROC Score', as our model itself will be evaluated on the AUROC score.<jupyter_code>model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=clr),
              loss='binary_crossentropy',
              metrics=['binary_accuracy', tf.keras.metrics.AUC()])
model.fit(train_datagen, epochs=4, batch_size=BATCH_SIZE, 
          steps_per_epoch=len(train_datagen), 
          validation_data=valid_datagen,
          validation_steps=len(valid_datagen))
model.save("model_1")<jupyter_output>INFO:tensorflow:Assets written to: model_1\assets<jupyter_text>Now, we unfreze the model and finetune the entire model.<jupyter_code>densenet.trainable = True
model.summary()
clr = tfa.optimizers.CyclicalLearningRate(
    initial_learning_rate=1e-5, #before: 1e-9
    maximal_learning_rate=1e-2, #before: 1e-4
    scale_fn=scale_fn,
    step_size=2*len(train_datagen)*BATCH_SIZE,
    scale_mode='cyclic'
)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=clr),
              loss='binary_crossentropy',
              metrics=['binary_accuracy', tf.keras.metrics.AUC()])
model.fit(train_datagen, epochs=4, batch_size=BATCH_SIZE, 
          steps_per_epoch=len(train_datagen), 
          validation_data=valid_datagen,
          validation_steps=len(valid_datagen))<jupyter_output>Epoch 1/4
 379/1507 [======>.......................] - ETA: 38:27:31 - loss: 0.2642 - binary_accuracy: 0.8905 - auc_1: 0.9044
StarcoderdataJupyterStructuredCleanDedup