# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/tandem-team/Machine-Learning-Collection/blob/master/MaskRCNN_Expln.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="DXItCGVYhLmw"
# ##Mask RCNN Architecture Explanation

# + [markdown] id="xrmrZKPthQzI"
# ###1. Object detection , Semantic Segmentation and Instance segmentation

# + [markdown] id="8wPuBrJkzJYu"
#
# Architectures like YOLO and SSD are efficient at the task of `object localization`, which is detecting objects in the image and drawing a bounding box around them. But the other major task in computer vision is `semantic segmentation` which is creating a pixel level boundary instead of a generic bounding box around the object. The pixel level classification allows for a much clearer distinction between different objects in the image.
# \
# A specific case of semantic segmentation is `Instance segmentation` where in the pixels in the image, in addition to the different classes, are also classified based on the number of objects in each class.
# \
# In this session, we'll discuss `Instance segmentaion`. Specifically, we will discuss Mask RCNN architecture which is uniquely designed for this task.
# \
# \
# <img src="https://files.ai-pool.com/d/DV8TLgkWsAEGsEs.jpg" alt="MarineGEO circle logo" style="height: 100px; width:100px;"/>

# + [markdown] id="vz3JF1p2zJHX"
# ##1.1 Mask RCNN architecture
# In 2017 <NAME> et al. proposed Mask R-CNN. It was developed on top of Faster RCNN which is a Region-based Convolution Neural Net that performs Object Detection and returns bounding boxes and the corresponding class labels. To this architecture(i.e., Faster RCNN) a layer is added for predicting the segmentation values.
#
# ###Architecture Explanation
# Mask RCNN, as the author explains, is a natural extension of the Faster RCNN detector.
#
#
# So similar to Faster RCNN, Mask RCNN has a backbone network and 2 stages there after.
# \
# Backbone network : A pretrained CNN network usually Resnet50 or Resnet101. This is used as a feature extractor. The ouput feature map of this stage is fed to both the Stage1 and 2.
# \
# Note: While Resnet works fine in this network, the authors of Mask RCNN have proposed an improved Feature Pyramid Network backbone for the architecture. For simplicity, this will be discussed later.
# \
# Stage 1 : A lightweight neural network that takes the feature map and outputs a set of rectangular regions in the feature map which have the potential for an object of one of the desired classes. In the output, each rectangular region is associated with an objectness score and the coordinates of the rectangle.
#
# Stage 2 : 
#
#
# \
# Now the backbone network is a simple vanilla pre-trained Resnet network. So we will discuss the next part, i.e. the Stage 1 Region proposal network.

# + [markdown] id="jRlfpnLtzJ0R"
# ##1.2 Region Proposal Network & Anchor Boxes
# The Region proposal network produces regions of interest from the feature map that it recieves as input. It does this by moving a sliding window of size `n*n` along the feature map. Each window thus obtained is then fed to the fully connnected layers - box regression layers and box-classification layer. In addition, it is also fed to the network that predicts the masks for the object.
#
# \
# At each sliding-window location, we simultaneously
# predict multiple region proposals, where the number
# of maximum possible proposals for each location is
# denoted as k. So the regression layer has 4k outputs encoding
# the coordinates of k boxes, and the classification layer outputs
# 2k scores that estimate probability of object or not
# object for each proposal. The k proposals are parameterized relative to k reference boxes, which we call `anchors`. An anchor is centered at the sliding window
# in question, and is associated with a scale and aspect
# ratio (Second figure below). By default we use 3 scales and
# 3 aspect ratios, yielding k = 9 anchors at each sliding
# position. For a convolutional feature map of a size
# W × H (typically ∼2,400), there are W Hk anchors in
# total.
#
# Below is an illustration of anchor boxes distributed over the input image(In practise the anchor boxes are decided over the feature map from the backbone network. Also the number of anchor boxes will be considerably more than shown here.)
#
# <img src="https://miro.medium.com/max/2160/1*ryi4m758IblLvDLhl_l9aA.png" alt="MarineGEO circle logo" style="height: 100px; width:100px;"/>
#
# \
# Below is an illustration of the different scales and aspect ratios of the anchor boxes that are used in the architecture.
# <img src="https://miro.medium.com/max/1400/0*n6pZEyvW47nlcdQz." alt="MarineGEO circle logo" style="height: 100px; width:100px;"/>
#
#
# \
# Now the output of the Region proposal network are `anchor class` and the refined `coordinates` for the bounding boxes.
#
#
# *   The `anchor class` is either `background` or `foreground`.
# *   The bounding boxes that are predicted often don't center over the object perfectly. Hence a delta for the coordinates is estimated by the RPN network which are the refined `coordinates`.
# \
# After this, the predictions from the RPN are filtered based on the anchor class. Also the bounding boxes that are closely located to each other are eliminated by measuring their overlap by a technique called Non-maxima suppression. Thus, a final set of proposals are selected for the next stage.
#
#
#
#

# + [markdown] id="vpQGcCK4zFIv"
# ##ROI Pool vs ROI Align
# Region of Interest Pooling operation is a simple neural net layer that performs maxpooling on inputs of non-uniform sizes to gain a fixed size feature map

# + [markdown] id="QVXFHMZ1YSFJ"
# ##Mask Representations

# + [markdown] id="TarfYbMQzKKL"
# ## Loss Function

# + [markdown] id="TmJdVKs-zKiD"
# ##Training the network

StarcoderdataJupyterScriptsDedupFiltered