<jupyter_start><jupyter_text>![title](https://i.ibb.co/f2W87Fg/logo2020.png)---           Task 17 - You Only Look Once Object DetectionYou only look once (YOLO) is a state-of-the-art, real-time object detection system. The latest improvement of YOLO achieve mAP of 57.9% on COCO test-dev with an ability of processing images at 30 FPS.Currently there are several version of improvements, from YOLO, YOLO9000, YOLOv2, and the latest is YOLOv3In this exercise, we will build a simplified version of YOLO detection system and train it using **makeshift** detection dataset Write down your Name and Student ID<jupyter_code>## --- start your code here ----

NIM  = 1301170073
Nama = '<NAME>'

## --- end your code here ----<jupyter_output><empty_output><jupyter_text>------[Part 0] Import Libraries  --- 1 - Import Moduleimport the necessary modules<jupyter_code>import numpy as np
import tensorflow as tf
from PIL import Image, ImageDraw, ImageFont
from tensorflow.keras.layers import *
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras import backend as K

import matplotlib.pyplot as plt
%matplotlib inline

np.set_printoptions(precision=2,suppress=True)<jupyter_output><empty_output><jupyter_text>download font type needed to write to the imageyou can change this to another font type of your liking<jupyter_code>!wget -O 'font.ttf' 'http://webpagepublicity.com/free-fonts/x/Xerox%20Sans%20Serif%20Narrow.ttf' -q<jupyter_output><empty_output><jupyter_text>--- 2 - Turn off Eager ExecutionSince this simplified version is using old Keras backend, when using TensorFlow 2.0 we need to turn off the Eager executionto do that run the following cell<jupyter_code>tf.compat.v1.disable_eager_execution()<jupyter_output><empty_output><jupyter_text>------[Part 1] Detection DatasetNow the Detection Dataset is usually quite expensive to build, and the already existing datasets are usually quite large in scale. Which is too much for our simplified exerciseSo let's generate a simplified dataset which is only images of string labels simulating the obects much like the image belowThis way we can generate a detection dataset with much cheaper process --- 1 - Define Class and ColorNow let's define the class and colorFirst, define each object label in `cl_label` list. You can change the label or even add more class if you want.**But the last labes should always be `'background'`**Then define what color you want the label to be drawn on the image. You can very much use black and white images, but the network will train easier if the color vary for each classAlso define the background image color outside the `cl_color` list --- **EXERCISE:** change or add the labels and their corresponding color<jupyter_code>cl_label = ['car',    'person',    'background']
cl_color = ['darkred', 'orangered', 'cyanred']
cl_hot   = to_categorical(np.arange(len(cl_label)))

bg_color = 'LightGray'

classes  = list(zip(cl_label, cl_hot, cl_color))
nb_class = len(classes)<jupyter_output><empty_output><jupyter_text>let's see the class list<jupyter_code>print('number of class:',nb_class)

for c in classes:
    print(c)<jupyter_output>number of class: 3
('car', array([1., 0., 0.], dtype=float32), 'darkred')
('person', array([0., 1., 0.], dtype=float32), 'orangered')
('background', array([0., 0., 1.], dtype=float32), 'cyanred')<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classesnumber of class: 3('bike', array([1., 0., 0.], dtype=float32), 'darkred')('person', array([0., 1., 0.], dtype=float32), 'orangered')('background', array([0., 0., 1.], dtype=float32), '') --- 2 - Define Image and Grid SizeIn the actual YOLO Detection system, it divides the input image into an $S\times S$ grid.Each grid cell predicts only one object and it incorporates a fixed number of boundary boxes called **anchors** or **priors**. The number anchors are defined by the shape of the bounding boxes. This tries to incorporate that in the real-life domain, the boundary boxes are not arbitrary. For example, cars and pedestrians have different box orientation (aspect ratio). Instead of predicting 5 arbitrary boundary boxes, we predict offsets to each of the constrained anchor boxes ---The more cells the image is divided to increase the detection accuracy as it will be more detailed. But too much cells and the detection will slow down.For this exercise, we will use only one anchor.The implementation in this exercise actually support multiple anchors, but we're not going to use it.You can, however, change the image size and grid number. For example changing it to $96\times 96$ with $3\times 3$ grid --- **EXERCISE:** Define the image size and grid size<jupyter_code>img_size  = (64, 64)
grid_size = (2, 2)<jupyter_output><empty_output><jupyter_text>for this exercise, we only use 1 anchor<jupyter_code>nb_anchor = 1<jupyter_output><empty_output><jupyter_text>Now calculate the cell size in each grid<jupyter_code>img_w,  img_h  = img_size
grid_w, grid_h = grid_size

cell_w = img_w//grid_w
cell_h = img_h//grid_h
print('image w/h :', img_w, img_h)
print('grid w/h  :', grid_w, grid_h)
print('cell w/h  :', cell_w, cell_h)<jupyter_output>image w/h : 64 64
grid w/h  : 2 2
cell w/h  : 32 32<jupyter_text>**EXPECTED OUTPUT**:if you're using default $64\times 64$ image with $2\times 2$ gridimage w/h : 64 64grid w/h  : 2 2cell w/h  : 32 32 --- 3 - Detection TargetThis is how we will generate our dataset.**So read this carefully*** First we start off with a **blank image*** For each grid cell, we randomly add either **ONE** object or **BACKGROUND**   * Adding background means that there is no object to draw in that grid  * If it is an object, randomly draw the object in that grid range  * Generate the target list* Each grid cell returns a list as the **cell target*** All lists of cell targets of an image are then combined into the **image target** listNow for the explanation of each list --- a. Cell TargetEach cell has a list of number as target consisting three parts: **class score**, **bounding box**, and **object confidence**grid target list* The **class score** is one-hot label score very much like classification target, indicates what object classified in that cell location. * The **bounding box** consists of $4$ values indicates the starting position ($x, y$ coordinates) followed by its box **width** and **height** ($w, h$)  * The $x$ and $y$ is a scalar ranged from $0$ to number of grid $S$  * The $w$ and $h$ is a scalar ranged from $0$ to $1$ scaling the **image width** and **image height*** The **object confidence** score is a value range from $0$ to $1$ indicates the network confidance of the appearance of an object in that locationThe length of cell target is&nbsp; $C+4+1$, where&nbsp; $C$ &nbsp;is the number of object to be detected. Usually background is not included in label, but for our exercise we did. Thus for this exercise, with $3$ classes, the length of target should be&nbsp; $3+4+1=8$ --- b. Multiple AnchorsAs mentioned before, we can use multiple anchor box detection. This is useful if we want to detect multiple objects in one cell, or if we want to specialize different box shape for different class.anchor boxFor multiple Anchors, the target list will be multiplied as many achors usedSo the length will be&nbsp; $B*(C+4+1)$, where&nbsp; $C$ &nbsp;is the number of class and&nbsp; $B$ &nbsp;is the number of anchorgrid target list (version A)Another variation in multiple anchor box (**like the one implemented in this exercise**) is that although it uses multiple anchor, each grid only perform **single detection**.Thus the &nbsp;$C$&nbsp; number of class only defined once at the beginning. So the length will only be&nbsp; $C + B*(4+1)$grid target list (version B) --- c. Image TargetEach grid cell has a list $B*(C+4+1)$. Now for an input image of $S\times S$ grid cell, we stack all those lists into a giant image target listSo the target length of an image is $S*S*\big(B*(C+4+1)\big)$Or if we use the other alternative, like the image below, we will get list with the length of$S*S*\big(C+B*(4+1)\big)$ --- 4 - Add Background FunctionIf in a grid, indicates by `row` and `col` position, we want to add a background target (no object), then all we have to do is generate a target in the center of that grid celladd background at $(1,0)$For example, if we have a background in grid $( 1, 0)$ from a $2\times 2$ &nbsp;grid as depicted above, then* the target $x$ will be at $1.5$($\text{col}+\frac{1}{2}$  )* the target $y$ will be at $0.5$ ($\text{row}+\frac{1}{2}$  )* the width $w$ and height $h$ will be $0.5$ and $0.5$ (half the image, since it's $2\times 2$ grid)* lastly, the object confidence is $0$ --- **EXERCISE:** Implement the function as defined above<jupyter_code>def add_background(y_hot, col, row):

    # convert y_hot vector 
    # from array float into a list of integers
    y_hot = list(y_hot.astype('int'))

    # calculate bounding box
    x = col + 0.5    # col number + 1/2
    y = row + 0.5    # row number + 1/2
    w = 1 / grid_w   # 1 / grid width
    h = 1 / grid_h    # 1 / grid height
    p = 0    # probability is 0

    # combine all into a single list
    target = []
    target += y_hot
    target += [x, y, w, h]
    target += [0]

    return target<jupyter_output><empty_output><jupyter_text>Check your implementation<jupyter_code>(l, y, c) = classes[-1]

print('background at (0,0):')
t = add_background(y, 0, 0)
print('tarhet :', t)
print('len(t) :', len(t), '\n')


print('background at (1,0):')
t = add_background(y, 1, 0)
print('tarhet :', t)
print('len(t) :', len(t), '\n')<jupyter_output>background at (0,0):
tarhet : [0, 0, 1, 0.5, 0.5, 0.5, 0.5, 0]
len(t) : 8 

background at (1,0):
tarhet : [0, 0, 1, 1.5, 0.5, 0.5, 0.5, 0]
len(t) : 8<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes  background at (0,0):  tarhet : [0, 0, 1, 0.5, 0.5, 0.5, 0.5, 0]  len(t) : 8  background at (1,0):  tarhet : [0, 0, 1, 1.5, 0.5, 0.5, 0.5, 0]  len(t) : 8 --- 5 - Add Object FunctionThen if we want to add an object sized &nbsp; $[w_\text{obj}\times h_\text{obj}]$  &nbsp;to the image at **CENTER** coordinate of $(x, y)$, we need to draw the bounding box from half $w_\text{obj}$ up and half $h_\text{obj}$ left to the coordinateFor example as shown in the picture, if we were to add  * an object sized $[24\times 10]$ * to an image sized $[64\times 64]$ with $[2\times 2]$ grid frame* at a **CENTER** coordinate of $x=20$ and $y=40$ Then the object **STARTING** coordinate to draw is* $x_\text{obj} = 20-\frac{24}{2} = 8$* $y_\text{obj} = 40-\frac{10}{2} = 35$Now to get the target, we need to scale it accordingly, thus we get* $x = 20/32 = 0.625$* $y = 40/32 = 1.25$* $w = 24/64 = 0.375$* $h = 10/64 = 0.156$Lastly, the object confidence is $1$ --- **EXERCISE:** Implement the function as defined above<jupyter_code>def add_object(img_draw, obj_idx, x, y):

    # retrieve the object information
    (label, y_hot, color) = classes[obj_idx]

    # convert y_hot vector 
    # from array float into a list of integers
    # see implementation above
    y_hot = list(y_hot.astype('int'))
    
    # set object width and height
    obj_w = len(label)*6
    obj_h = 10

    # calculate object coordinates
    obj_x = x - (obj_w / 2)   # x - object width / 2
    obj_y = y - (obj_h / 2)   # y - object height / 2
    
    # draw the object (label) to image
    img_draw.text((obj_x, obj_y), label, fill=color)

    # calculate bounding box
    x = x / cell_w    # x / cell width
    y = y / cell_h    # y / cell height
    w = obj_w / img_w    # object width / image width
    h = obj_h / img_h    # object height / image height
    p = 1    # probability is 1

    # combine all into a single list
    target = []
    target += y_hot
    target += [x, y, w, h]
    target += [1]

    return target<jupyter_output><empty_output><jupyter_text>Check your implementation<jupyter_code>test_img = Image.new('RGB', img_size, color=bg_color)
draw = ImageDraw.Draw(test_img)
x = 20
y = 40

target = add_object(draw, 0, x, y)

plt.imshow(test_img)
plt.axis('off')
plt.show()

print('target:',target)<jupyter_output><empty_output><jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes with $64\times 64$ image with $2\times 2$ gridtarget: [1, 0, 0, 0.625, 1.25, 0.375, 0.15625, 1] --- 6 - Generate Data FunctionNow we define our generate data function which repeatedly randomize an object and put it into each grid of an image.If the randomized object is `background`, call `add_background()` function, andif it is an object, randomize the center location, and call `add_object()` function --- **EXERCISE:** Implement the function as defined above<jupyter_code>def generate_data(nb_data):

    # create empty list for data and label
    x_train = []
    y_train = []

    for j in range(0, nb_data):

        # generate empty image base
        base_img = Image.new('RGB', img_size, color=bg_color)
        img_draw = ImageDraw.Draw(base_img)
        
        # create empty list for targets
        img_target = []

        # loop over all grid (row and col)
        for row in range(grid_w):

            for col in range(grid_h):

                # randomize object id
                # call .randint() function from numpy 
                # with random range (0 , nb_class)
                idx = np.random.randint(0, nb_class)

                # get class information
                (label, y_hot, color) = classes[idx]
                
                # check if it's a background
                if (label=='background'):

                    # call add_background() funtion with input y_hot, col, and row
                    new_target = add_background(y_hot, col, row)

                else:
                    # randomize coordinate center location
                    # call .randint() function from numpy 
                    # with random range (col*cell_w , (col+1)*cell_w)
                    x = np.random.randint(col*cell_w, (col+1)*cell_w)

                    # call .randint() function from numpy 
                    # with random range (row*cell_h , (row+1)*cell_h)
                    y = np.random.randint(row*cell_h , (row+1)*cell_h)
                    
                    # call add_object() function with input img_draw, idx, x, and y
                    new_target = add_object(img_draw, idx, x, y)

                # append new_target into img_target list
                img_target.append(new_target)

        # convert base_img into a numpy array
        np_img = np.asarray(base_img)

        # append np_img into x_train list
        x_train.append(np_img)   

        # append img_target into y_train list
        y_train.append(img_target)

    # convert into numpy array
    x_train = np.array(x_train).astype('float32')/255
    y_train = np.array(y_train).astype('float32')

    return x_train, y_train<jupyter_output><empty_output><jupyter_text>--- 7 - Generate DataAnd that's itNow let's generate $5000$ training data and another $1000$ as validation data<jupyter_code>x_train, y_train = generate_data(5000)
x_val, y_val     = generate_data(1000)

print('x_train shape:', x_train.shape)
print('y_train shape:', y_train.shape)<jupyter_output>x_train shape: (5000, 64, 64, 3)
y_train shape: (5000, 4, 8)<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes $64\times 64$ image with $2\times 2$ gridx_train shape: (5000, 64, 64, 3)y_train shape: (5000, 4, 8) Let's view the first image from training set and its target<jupyter_code>plt.imshow(x_train[0])
plt.axis('off')
plt.show()

for tgt in y_train[0]:
    print(tgt)<jupyter_output><empty_output><jupyter_text>Now view 10 images from training setYou should see various images with random number of objects at random locations<jupyter_code>plt.figure(figsize=(10,6))
for i in range(10):
    plt.subplot(2,5,i+1)
    plt.imshow(x_train[i])
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>------[Part 2] Visualizing Detection BoxNow let's create a function to draw the bounding box to an image.**REMEMBER** that we're using target list design where each grid consist of a list with length $C + B*(4+1)$ --- 1 - Box Grid FunctionGiven an image, the target `y_grid`, along with `col` and `row` number for current grid location, we draw the bounding box as follow:* First, extract the class from `y_grid` and get the class index* Then reshape the rest of `y_grid` into a matrix of `(nb_anchor, 5)`* Then loop over each anchor box * Extract the $x, y, w, h, $ and $p$ from the anchor * The $x$ and $y$ from prediction is in range $0$ to $1$ as the result from the grid input, and not the image. So to draw the prediction box, we need to add $x$ and $y$ with `row` and `col` number * Now to perform a simplified **Non-max Suppression**, check if confidence $p$ is higher than `max_p` from other anchor * If the current confidence $p$ is higher, calculate and overwrite the bounding box coordinates * Now, it's a little bit different when we draw the rectangle as it uses $[x1,y1,x2,y2]$ rather than using **width** and **height** * Set color to draw the box. For example, target box is **gray**, high confidence prediction is  **green**, and low confidence prediction is  **blue** * If the class is not background, draw the box --- **EXERCISE:** Implement the function as defined above<jupyter_code>def draw_box_grid(draw, y_grid, row=0, col=0, pad=0, is_prediction=False, threshold  = 0.1):
    
    # initialize max probability = 0
    max_p   = 0

    # get class from y_grid[0:nb_class]
    y_class = y_grid[0:nb_class]

    # get class id, use np.argmax() with input y_class
    y_pred  = np.argmax(y_class)

    # reshape the rest of anchor list into (nb_anchor, 5)
    anchor_box = y_grid[nb_class:].reshape(nb_anchor, 5)

    # loop over anchor box
    for b in anchor_box:

        # retrieve x, y, w, h and p
        # from b[0] to b[4]
        x = b[0]
        y = b[1]
        w = b[2]
        h = b[3]
        p = b[4]         

        # add row number to x 
        x += row
        # add col number to y 
        y += col

        # if current probability is higher than max
        if p > max_p:
            max_p = p
            rx1   = (x * cell_w) - (w*img_w / 2) + pad  # x*cell width - w*image width/2 + pad
            ry1   = (y * cell_h) - (h*img_h / 2) + pad  # y*cell height - h*image height/2 + pad
            rx2   = w*img_w + rx1  # w*image width + rx1
            ry2   = h*img_h + ry1  # h*image height + ry1

    # is object if probability is higher than threshold
    is_object = p > threshold

    # set color box
    if is_prediction:
        if is_object:
            box_color = 'LimeGreen'
        else:
            box_color = 'RoyalBlue'
    else:
        box_color = 'DarkGray'

    # if class is not background, draw the box
    if y_pred!=nb_class-1:
        draw.rectangle([rx1, ry1, rx2, ry2], outline=box_color)

    return y_pred, is_object<jupyter_output><empty_output><jupyter_text>Check your implementation<jupyter_code>test_img = Image.new('RGB', img_size, color=bg_color)
draw = ImageDraw.Draw(test_img)
x = 20
y = 40

target = add_object(draw, 0, x, y)
plt.subplot(121)
plt.imshow(test_img)
plt.axis('off')

print('original target',target)
target = np.array(target)

y_pred, is_object = draw_box_grid(draw, target, row=0, col=0, is_prediction=False)
plt.subplot(122)
plt.imshow(test_img)
plt.axis('off')
plt.show()
print('predicted:',y_pred, ', is object?',is_object)<jupyter_output>original target [1, 0, 0, 0.625, 1.25, 0.28125, 0.15625, 1]<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes $64\times 64$ image with $2\times 2$ gridoriginal target [1, 0, 0, 0.625, 1.25, 0.375, 0.15625, 1]predicted: 0 , is object? True --- 2 - Draw Bounding Box FunctionNow we define function to iterate over grid and draw the bounding box in it.Here's step by step of this function* First we pad the image with zero pixels as a place to write the prediction class* Loop over the grid * Draw the target box * If prediction is provided, draw the prediction box * write the predicted class on top and bottom of the image note that the code only designed for $2\times 2$ grid * set the text color. For example **green** if it's the correct class, **pink** if it's wrong or having low confidence, and **black** if prediction is not provided --- **EXERCISE:** Implement the function as defined above<jupyter_code>def draw_bbox(np_img, target, prediction=None, pad=10, threshold=0.1):

    # rescale np_img by multiplying it with 255
    img = np_img * 255

    # add white padding as a place to write class prediction
    img = np.pad(img, ((pad,pad),(pad,pad), (0,0)), 'constant', constant_values=255)

    # generate image
    img      = Image.fromarray(img.astype('uint8'), 'RGB')
    img_draw = ImageDraw.Draw(img)
    font     = ImageFont.truetype(r'font.ttf', 10) 

    for row in range(grid_w):

        for col in range(grid_h):

            # get current target for this grid
            target_grid = target[col*grid_h+row]

            # call draw_box_grid() function with input img_draw, target_grid, and pad=pad
            y_act,_ = draw_box_grid(img_draw, target_grid, pad=pad)

            if prediction is not None:

                # get current prediction for this grid
                prediction_grid = prediction[col*grid_h+row]

                # call draw_box_grid() function with input
                # img_draw, prediction_grid, row, col, 
                # pad=pad, is_prediction=True, and threshold=threshold
                y_pred, conf = draw_box_grid(img_draw, 
                                            prediction_grid, 
                                            row, col, 
                                            pad=pad, 
                                            is_prediction=True, 
                                            threshold=threshold)
                
                # set color for predicted class writing
                if y_pred == y_act and conf:
                    # correct detection
                    fill = 'DarkGreen'
                    s = str(y_act)+'-'+str(y_pred)
                else:
                    # wrong detection or low confidence
                    fill = 'DeepPink'
                    s = str(y_act)+'-'+str(y_pred)+' (?)'
                    
            else:
                # drawing target
                s = str(y_act)
                fill = 'black'
                
            # draw text class target/prediction
            img_draw.text((row+row*cell_w+pad, col+img_h*col+1+col*8), s, fill=fill, font=font)

    # convert img into a numpy array
    img = np.array(img)

    return img<jupyter_output><empty_output><jupyter_text>Check your implementationYou should see that the "objects" are correctly localized with its bounding boxes.<jupyter_code>plt.figure(figsize=(15,8))
for i in range(10):
    img = draw_bbox(x_train[i], y_train[i])
    plt.subplot(2,5,i+1)
    plt.imshow(img)
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>------[Part 3] YOLO DetectionAnd we're arrive at the main function, the YOLO Detection itselfFirst we define our classification model. The original YOLO network consists of $24$ convolutional layers alternating between $3\times 3$ convolution and $1\times 1$ bottleneck layers to reduce the features space from preceding layers, followed by $2$ FC layers from a $448\times 448$ image input. But for our simple exercise, we won't need that big network.So let's define our own --- 1 - Define Input ShapeUse Functional API to build the model. --- **EXERCISE:** Define input and output shape<jupyter_code># input shape is a tuple of image height, image width, and 3 channel
input_shape   = (img_h, img_w, 3)

# total grid is grid width * grid height
total_grid    = grid_w * grid_h

# anchor length is number class + number anchor * 5
anchor_length = nb_class + nb_anchor*5<jupyter_output><empty_output><jupyter_text>Check your implementation<jupyter_code>print('input shape   :', input_shape)
print('total grid    :', total_grid)
print('anchor length :', anchor_length)<jupyter_output>input shape   : (64, 64, 3)
total grid    : 4
anchor length : 8<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes $64\times 64$ image with $2\times 2$ gridinput shape   : (64, 64, 3)total grid    : 4anchor length : 8 --- 2 - Define Classification ModelUse Functional API to build the model. --- **EXERCISE:** Build model implementing the architecture as follow    * Input layer of  (img_h,img_w,3)      * conv layer with 16 filters of 1x1 without activation    * conv layer with 32 filters of 3x3 without activation    * LeakyReLU activation with alpha 0.01     * MaxPool2D layer          * conv layer with 16 filters of 3x3 without activation    * conv layer with 32 filters of 3x3 without activation    * LeakyReLU activation with alpha 0.01     * MaxPool2D layer          * Flatten layer     * Fully connected layer with 256 neurons and activation sigmoid    * Fully connected layer with output_shape neurons and activation sigmoid    * Reshape layer to reshape output activation to (grid, anchors)  shape<jupyter_code># define Input() layer with shape=input_shape
input_tensor = Input(shape=input_shape)

# add conv2d to input_tensor with 16 filters, kernel size 1
x = Conv2D(16, kernel_size=1)(input_tensor)

# add conv2d to x with 32 filters, kernel size 3
x = Conv2D(32, kernel_size=3)(x)

# add leakyrelu to x alpha=0.3
x = LeakyReLU()(x)

# add maxpool2d layer to x
x = MaxPool2D()(x)

# add conv2d to x with 16 filters, kernel size 3
x = Conv2D(16, kernel_size=3)(x)

# add conv2d to x with 32 filters, kernel size 3
x = Conv2D(32, kernel_size=3)(x)

# add leakyrelu to x alpha=0.3
x = LeakyReLU()(x)

# add maxpool2d layer to x
x = MaxPool2D()(x)

# add flatten layer to x
x = Flatten()(x)

# add dense layer to x with 256 neurons and sigmoid activation
x = Dense(256, activation='sigmoid')(x)

# add dense layer to x with total_grid * anchor_length neurons and sigmoid activation
x = Dense(total_grid*anchor_length, activation='sigmoid')(x)

# add reshape layer to x with input (total_grid, anchor_length)
output_tensor = Reshape((total_grid, anchor_length))(x)

# initialize model by calling Model() function with input input_tensor and output_tensor
model = Model(input_tensor, output_tensor)<jupyter_output><empty_output><jupyter_text>Now to check your implementation<jupyter_code>print(model.summary())<jupyter_output>Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 64, 64, 3)]       0         
_________________________________________________________________
conv2d (Conv2D)              (None, 64, 64, 16)        64        
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 62, 62, 32)        4640      
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 62, 62, 32)        0         
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 31, 31, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 29, 29, 16)        4624      
______________________________________________________[...]<jupyter_text>**EXPECTED OUTPUT**:if you're using default $3$ classes $64\times 64$ image with $2\times 2$ gridinput_? (InputLayer)         [(None, 64, 64, 3)]       0         conv2d (Conv2D)              (None, 64, 64, 16)        64        ...dense_? (Dense)              (None, 32)                8224      reshape (Reshape)            (None, 4, 8)              0         Total params: 1,406,896Trainable params: 1,406,896Non-trainable params: 0_________________________________________________________________ Visualize the Network architecture<jupyter_code>plot_model(model, show_shapes=True,
    show_layer_names=False,
    rankdir='LR',
    expand_nested=False,
    dpi=60
)<jupyter_output><empty_output><jupyter_text>--- 3 - Define LossYOLO uses sum-squared error between the predictions and the ground truth to calculate loss. The loss function composes of:* the **classification loss.*** the **localization loss** (errors between the predicted boundary box and the ground truth).* the **confidence loss** (the objectness of the box). --- a. Classification LossIf an object is detected, the classification loss at each cell is the squared error of the class conditional probabilities for each class:$$ \sum_{i=0}^{S^2}{1}_i^\text{obj} \sum_{c \in \text{classes}}\big(p_i(c)-\hat{p_i}(c)\big)^2$$ --- b. Localization LossThe localization loss measures the errors in the predicted boundary box locations and sizes. We only count the box responsible for detecting the object.$$ x\lambda_\text{coord}\sum_{i=0}^{S^2}\sum_{j=0}^{B}{1}_{ij}^\text{obj} \big[(x_i-\hat{x}_i)^2+(y_i-\hat{y}_i)^2\big]\\+\lambda_\text{coord}\sum_{i=0}^{S^2}\sum_{j=0}^{B}\rm{1}_{ij}^\text{obj} \big[(\sqrt{w_i}-\sqrt{\hat{w}_i})^2+(\sqrt{h_i}-\sqrt{\hat{h}_i})^2\big]$$ --- c. Confidence LossIf an object is detected in the box, the confidence loss (measuring the objectness of the box) is:$$\sum_{i=0}^{S^2}\sum_{j=0}^{B}{1}_{ij}^\text{obj}(C_i-\hat{C}_i)^2$$If an object is **NOT** detected in the box, the confidence loss is:$$\lambda_\text{noobj}\sum_{i=0}^{S^2}\sum_{j=0}^{B}{1}_{ij}^\text{noobj}(C_i-\hat{C}_i)^2$$ --- d. Complete ImplementationNow don't be disheartened, we already prepared the loss implementation for you. So read the code carefully.<jupyter_code>def custom_loss(y_true, y_pred):
  
    grid = np.array([ [[float(x),float(y)]]*nb_anchor   for y in range(grid_h) for x in range(grid_w)])

    # get the actual and predicted class
    y_true_class   = y_true[...,0:nb_class-1]
    y_pred_class   = y_pred[...,0:nb_class-1]

    # get the actual and predicted box
    pred_boxes     = K.reshape(y_pred[...,nb_class:], (-1,grid_w*grid_h,nb_anchor,5))
    true_boxes     = K.reshape(y_true[...,nb_class:], (-1,grid_w*grid_h,nb_anchor,5))
    
    # get predicted coordinates and confidence
    y_pred_xy      = pred_boxes[...,0:2] + K.variable(grid)
    y_pred_wh      = pred_boxes[...,2:4]
    y_pred_conf    = pred_boxes[...,4]
   
    # get actual coordinates and confidence
    y_true_xy      = true_boxes[...,0:2]
    y_true_wh      = true_boxes[...,2:4]
    y_true_conf    = true_boxes[...,4]

    # calculate classification loss
    clss_loss      = K.sum(K.square(y_true_class - y_pred_class), axis=-1)
    
    # calculate localization loss
    xy_loss        = K.sum(K.sum(K.square(y_true_xy - y_pred_xy),axis=-1)*y_true_conf, axis=-1)
    wh_loss        = K.sum(K.sum(K.square(K.sqrt(y_true_wh) - K.sqrt(y_pred_wh)), axis=-1)*y_true_conf, axis=-1)

    # non-max suppression
    intersect_wh   = K.maximum(K.zeros_like(y_pred_wh), (y_pred_wh + y_true_wh)/2 - K.abs(y_pred_xy - y_true_xy) )
    intersect_area = intersect_wh[...,0] * intersect_wh[...,1]

    true_area      = y_true_wh[...,0] * y_true_wh[...,1]
    pred_area      = y_pred_wh[...,0] * y_pred_wh[...,1]

    # calculate intersection over union
    union_area     = pred_area + true_area - intersect_area
    iou            = intersect_area / union_area

    # calculate confidence loss
    conf_loss      = K.sum(K.square(y_true_conf*iou - y_pred_conf)*y_true_conf, axis=-1)

    # sum all losses
    d = clss_loss + xy_loss + wh_loss + conf_loss    

    return d<jupyter_output><empty_output><jupyter_text>--- 4 - Compile ModelNow compile model by adding the custom loss and Adam optimizer --- **EXERCISE:** compile model using custom_loss and Adam optimizer with lr=0.001<jupyter_code>from tensorflow.keras.optimizers import Adam

# initialize Adam optimizer with lr=0.001
myAdam = Adam(learning_rate=0.001)

# compile model with loss=custom_loss and optimizer=myAdam
model.compile(optimizer=myAdam, loss=custom_loss)<jupyter_output><empty_output><jupyter_text>--- 5 - Train ModelTrain the model for 80 epochs<jupyter_code>hist = model.fit(x_train, y_train, validation_data=(x_val, y_val), batch_size=512, epochs=80)<jupyter_output>Train on 5000 samples, validate on 1000 samples
Epoch 1/80
5000/5000 [==============================] - ETA: 0s - loss: 0.6297WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_v1.py:2048: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.
5000/5000 [==============================] - 1s 280us/sample - loss: 0.6297 - val_loss: 0.5482
Epoch 2/80
5000/5000 [==============================] - 1s 164us/sample - loss: 0.4931 - val_loss: 0.4304
Epoch 3/80
5000/5000 [==============================] - 1s 162us/sample - loss: 0.3954 - val_loss: 0.3611
Epoch 4/80
5000/5000 [==============================] - 1s 163us/sample - loss: 0.3432 - val_loss: 0.3217
Epoch 5/80
5000/5000 [==============================] - 1s 164us/sample - loss: 0.2999 - val_loss: 0.2810
E[...]<jupyter_text>**EXPECTED OUTPUT**:your loss should start around 0.6 and end around 0.01 after 80 epochs Visualize training history<jupyter_code>plt.rcParams['figure.figsize'] = [8, 4]
plt.subplots_adjust(wspace=0.2)

plt.plot(hist.history['loss'])
plt.plot(hist.history['val_loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Val'])
plt.show()<jupyter_output><empty_output><jupyter_text>--- 6 - Test DetectionLet's show the detection results --- a. Using Data TrainFirst, let's see the performance on training data<jupyter_code>y_pred = model.predict(x_train)
plt.figure(figsize=(15,8))
for i in range(10):
    img = draw_bbox(x_train[i], y_train[i], y_pred[i], threshold=0.1)
    plt.subplot(2,5,i+1)
    plt.imshow(img)
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>You should see that most of the objects are detected with quite precisionthe **pink** label and **blue** detection box means low confidence or wrong classificationyou can train it longer to get more precision on the detection boxyou can also lower the threshold when drawing the detection box --- b. Using Data ValidationNow view the detection results on validation data<jupyter_code>y_pred_val = model.predict(x_val)
plt.figure(figsize=(15,8))
for i in range(10):
    img = draw_bbox(x_val[i], y_val[i], y_pred_val[i], threshold=0.1)
    plt.subplot(2,5,i+1)
    plt.imshow(img)
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>--- c. Using Data TestLastly, let's generate another 20 new data, and test it<jupyter_code># generate 20 new data
x_test, y_test = generate_data(20)

# predict new data
y_pred_test = model.predict(x_test)
plt.figure(figsize=(15,12))
for i in range(20):
    img = draw_bbox(x_test[i], y_test[i], y_pred_test[i], threshold=0.1)
    plt.subplot(4,5,i+1)
    plt.imshow(img)
    plt.axis('off')
plt.tight_layout()
plt.show()<jupyter_output><empty_output>