<jupyter_start><jupyter_text>Convolutional Neural NetworksSean WadeThe main difference between CNNs and standard DNNs is that neurons are arranged in 3 dimensions, **width, height, and depth**.  Each layer in a CNN takes in an input 3d tensor and output a 3d tensor.There are 3 main layers in typical CNN:- Convolutional Layer- Pooling Layer- Fully-Connected Layer  Convolutional LayerConvolutional kernels are nothing new to image processing.  This kernels are small matricies that we use for blurring, sharpening, edge detection, and more.$\text{Identity: } \quad \begin{bmatrix}0 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 0\end{bmatrix}$$\text{Blur: } \qquad  \begin{bmatrix}1/9 & 1/9 & 1/9 \\1/9 & 1/9 & 1/9 \\1/9 & 1/9 & 1/9\end{bmatrix}$$\text{Sobel (edge): }  \begin{bmatrix}-1 & 0 & 1 \\-2 & 0 & 2 \\-1 & 0 & 1\end{bmatrix}$By convolving these kernels across the image we can isolate certain characteristics, or **features**.  A large number at a location after the convolution indicates a large **activation** or response to the kernel.  In the case of the sobel filter a high activation indicates there is an edge present.  This site is an awesome visualization of kernels: http://setosa.io/ev/image-kernels/ CNNs use filters like above for identifying features in 2D space.  However, the magic with CNNs is that these filters are not predetermined, but rather they are learned.  Also these leaned filters are stacked to look for complex, high level features.  For example, instead of just detecting edges we can now detect cars and tires.  These filters are small spacially, meaning along the width and height, but always extend to the full depth of the input volume. For example 5x5x3 for a 224x224x3 image. Hyperparameters - W - Input volume size - F - Receptive field size - S - Stride - P - Amount of 0 padding used  Output size = $\frac{(W-F+2 P)}{S}+1$  Pooling Layer​The function of this layer is to reduce the spacial size of the representation. The benefit of this is three fold: reduce the amount of parameters, lower computation time, and control overfitting.  There are variants like max, average, or overlapping.​​It is worth noting that while this has been standard, a few recent publications have abandoned it.  It could be falling out of favor.  Fully Connected LayerA fully connected layer takes the activations and flattens them to behave like a normal layer in a DNN.  Now with all these pieces we can create a whole CNN architecture.  Learning MNIST<jupyter_code>from __future__ import absolute_import, division, print_function
import numpy as np
import tensorflow as tf
import math
import random
from matplotlib import pyplot as plt
#from utils import load_CIFAR10
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
# Load in data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('data/MNIST_data', one_hot=True)
plt.imshow(np.reshape(mnist.train.images[42],[28,28]))
print(mnist.train.labels[42])
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)<jupyter_output><empty_output><jupyter_text>Standard DNN<jupyter_code>%%time

# Placeholders
y_true = tf.placeholder(tf.float32, [None, 10])
x = tf.placeholder(tf.float32, [None, 784])

# Layer 1
W1 = weight_variable([784, 100])
b1 = bias_variable([100])
h1 = tf.nn.relu(tf.matmul(x, W1) + b1)

# Layer 2
W2 = weight_variable([100, 10])
b2 = bias_variable([10])
y = tf.matmul(h1, W2) + b2

# Define loss and optimizer
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_true))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_true, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Start session
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# Train
for i in range(5000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    if i % 100 == 0:
        _, acc = sess.run([train_step, accuracy], feed_dict={x: batch_xs, y_true: batch_ys})
        print("step [%d]: training accuracy %.4f" % (i, acc))
    else:
        sess.run(train_step, feed_dict={x: batch_xs, y_true: batch_ys})
        

# Test trained model
print("--- RESULTS ---")
print("Test Accuracy: %.4f" % sess.run(accuracy, feed_dict={x: mnist.test.images[:1000],
                              y_true: mnist.test.labels[:1000]}))
sess.close()
tf.reset_default_graph()<jupyter_output>step [0]: training accuracy 0.1400
step [100]: training accuracy 0.8500
step [200]: training accuracy 0.9200
step [300]: training accuracy 0.9300
step [400]: training accuracy 0.9500
step [500]: training accuracy 0.9400
step [600]: training accuracy 0.9200
step [700]: training accuracy 0.9600
step [800]: training accuracy 0.9200
step [900]: training accuracy 0.9300
step [1000]: training accuracy 0.9600
step [1100]: training accuracy 0.9900
step [1200]: training accuracy 0.9600
step [1300]: training accuracy 0.9900
step [1400]: training accuracy 0.9700
step [1500]: training accuracy 0.9800
step [1600]: training accuracy 0.9600
step [1700]: training accuracy 0.9600
step [1800]: training accuracy 0.9600
step [1900]: training accuracy 0.9700
step [2000]: training accuracy 0.9400
step [2100]: training accuracy 1.0000
step [2200]: training accuracy 0.9800
step [2300]: training accuracy 1.0000
step [2400]: training accuracy 0.9700
step [2500]: training accuracy 1.0000
step [2600]: training ac[...]<jupyter_text>Vanilla CNNThis is the most basic CNN.  It has 2 convolutional layers and 2 fully connected layers.<jupyter_code>def conv2d(x, W):
    # [batch, in_height, in_width, in_channels]
    # [filter_height, filter_width, in_channels, out_channels]
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
%%time
tf.reset_default_graph()
# Placeholders
y_true = tf.placeholder(tf.float32, [None, 10])
x = tf.placeholder(tf.float32, [None, 784])

# Resize to appropriate input tensor:
# [batch, in_height, in_width, in_channels]
x_image = tf.reshape(x, [-1,28,28,1])

# First Convolutional Layer
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)

# Second Convolutional Layer
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_conv1, W_conv2) + b_conv2)

# Densely Connected Layer
W_fc1 = weight_variable([28*28*64, 1024])
b_fc1 = bias_variable([1024])
h_conv2_flat = tf.reshape(h_conv2, [-1, 28*28*64])
h_fc1 = tf.nn.relu(tf.matmul(h_conv2_flat, W_fc1) + b_fc1)

# Readout Layer
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.matmul(h_fc1, W_fc2) + b_fc2

# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_true))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(5000):
    batch = mnist.train.next_batch(50)
    if i % 100 == 0:
        _, acc = sess.run([train_step, accuracy], feed_dict={x: batch[0], y_true: batch[1]})
        print("step [%d]: training accuracy %.4f" % (i, acc))
    else:
        sess.run(train_step, feed_dict={x: batch[0], y_true: batch[1]})
        
print("--- RESULTS ---")
print("Test Accuracy %.4f" % sess.run(accuracy, feed_dict={
    x: mnist.test.images[:1000], y_true: mnist.test.labels[:1000]}))

sess.close()<jupyter_output>step [0]: training accuracy 0.1000
step [100]: training accuracy 0.8400
step [200]: training accuracy 0.9000
step [300]: training accuracy 0.9200
step [400]: training accuracy 0.9600
step [500]: training accuracy 0.9200
step [600]: training accuracy 0.9200
step [700]: training accuracy 0.9600
step [800]: training accuracy 0.9800
step [900]: training accuracy 0.9800
step [1000]: training accuracy 0.9400
step [1100]: training accuracy 0.9600
step [1200]: training accuracy 0.9600
step [1300]: training accuracy 0.9200
step [1400]: training accuracy 0.9600
step [1500]: training accuracy 1.0000
step [1600]: training accuracy 0.9600
step [1700]: training accuracy 1.0000
step [1800]: training accuracy 1.0000
step [1900]: training accuracy 0.9600
step [2000]: training accuracy 1.0000
step [2100]: training accuracy 1.0000
step [2200]: training accuracy 1.0000
step [2300]: training accuracy 1.0000
step [2400]: training accuracy 1.0000
step [2500]: training accuracy 0.9800
step [2600]: training ac[...]<jupyter_text>CNN + MaxPool + DropoutThis CNN is the same as above plus max pooling and dropout.<jupyter_code>def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')
%%time
tf.reset_default_graph()
# Placeholders
y_true = tf.placeholder(tf.float32, [None, 10])
x = tf.placeholder(tf.float32, [None, 784])
keep_prob = tf.placeholder(tf.float32)

# Resize to appropriate input tensor:
# [batch, in_height, in_width, in_channels]
x_image = tf.reshape(x, [-1,28,28,1])

# First Convolutional Layer
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1) # <------------------------- Added Pooling

# Second Convolutional Layer
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2) # <------------------------- Added Pooling

# Densely Connected Layer
W_fc1 = weight_variable([7*7*64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob) # <------------ Added Dropout

# Readout Layer
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2

# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_true))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for i in range(5000):
    batch = mnist.train.next_batch(50)
    if i % 100 == 0:
        _, acc = sess.run([train_step, accuracy], feed_dict={x: batch[0], y_true: batch[1], keep_prob: 1.0})
        print("step [%d]: training accuracy %g" % (i, acc))
    else:
        sess.run(train_step, feed_dict={x: batch[0], y_true: batch[1], keep_prob: .5})

print("--- RESULTS ---")
print("Test Accuracy %g" % accuracy.eval(feed_dict={
    x: mnist.test.images[:1000], y_true: mnist.test.labels[:1000], keep_prob: 1.0}))

sess.close()<jupyter_output>step [0]: training accuracy 0.14
step [100]: training accuracy 0.82
step [200]: training accuracy 0.88
step [300]: training accuracy 0.9
step [400]: training accuracy 0.94
step [500]: training accuracy 0.94
step [600]: training accuracy 0.9
step [700]: training accuracy 0.98
step [800]: training accuracy 0.92
step [900]: training accuracy 0.98
step [1000]: training accuracy 1
step [1100]: training accuracy 0.98
step [1200]: training accuracy 0.9
step [1300]: training accuracy 1
step [1400]: training accuracy 0.96
step [1500]: training accuracy 0.96
step [1600]: training accuracy 0.96
step [1700]: training accuracy 0.98
step [1800]: training accuracy 1
step [1900]: training accuracy 1
step [2000]: training accuracy 0.96
step [2100]: training accuracy 1
step [2200]: training accuracy 0.98
step [2300]: training accuracy 0.98
step [2400]: training accuracy 0.96
step [2500]: training accuracy 0.98
step [2600]: training accuracy 1
step [2700]: training accuracy 0.98
step [2800]: training acc[...]
StarcoderdataJupyterStructuredCleanDedup