<jupyter_start><jupyter_text><jupyter_code># Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

try:
    # %tensorflow_version only exists in Colab.
    %tensorflow_version 2.x
    IS_COLAB = True
except Exception:
    IS_COLAB = False

# TensorFlow ≥2.0 is required
import tensorflow as tf
from tensorflow import keras
assert tf.__version__ >= "2.0"

if not tf.config.list_physical_devices('GPU'):
    print("No GPU was detected. CNNs can be very slow without a GPU.")
    if IS_COLAB:
        print("Go to Runtime > Change runtime and select a GPU hardware accelerator.")

# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
np.random.seed(42)
tf.random.set_seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "cnn"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)<jupyter_output><empty_output><jupyter_text>9. High Accuracy CNN for MNIST_Exercise: Build your own CNN from scratch and try to achieve the highest possible accuracy on MNIST._ The following model uses 2 convolutional layers, followed by 1 pooling layer, then dropout 25%, then a dense layer, another dropout layer but with 50% dropout, and finally the output layer. It reaches about 99.2% accuracy on the test set. This places this model roughly in the top 20% in the [MNIST Kaggle competition](https://www.kaggle.com/c/digit-recognizer/) (if we ignore the models with an accuracy greater than 99.79% which were most likely trained on the test set, as explained by <NAME> in [this post](https://www.kaggle.com/c/digit-recognizer/discussion/61480)). Can you do better? To reach 99.5 to 99.7% accuracy on the test set, you need to add image augmentation, batch norm, use a learning schedule such as 1-cycle, and possibly create an ensemble.<jupyter_code>(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.mnist.load_data()
X_train_full = X_train_full / 255.
X_test = X_test / 255.
X_train, X_valid = X_train_full[:-5000], X_train_full[-5000:]
y_train, y_valid = y_train_full[:-5000], y_train_full[-5000:]

X_train = X_train[..., np.newaxis]
X_valid = X_valid[..., np.newaxis]
X_test = X_test[..., np.newaxis]
X_train.shape
X_train[0].shape
keras.backend.clear_session()
tf.random.set_seed(42)
np.random.seed(42)

model = keras.models.Sequential([
    keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
    keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
    keras.layers.MaxPool2D(),
    keras.layers.Flatten(),
    keras.layers.Dropout(0.25),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(10, activation="softmax")
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="nadam",
              metrics=["accuracy"])

model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))
model.evaluate(X_test, y_test)<jupyter_output>Epoch 1/10
1719/1719 [==============================] - 16s 5ms/step - loss: 0.3717 - accuracy: 0.8833 - val_loss: 0.0489 - val_accuracy: 0.9854
Epoch 2/10
1719/1719 [==============================] - 8s 5ms/step - loss: 0.0822 - accuracy: 0.9754 - val_loss: 0.0422 - val_accuracy: 0.9884
Epoch 3/10
1719/1719 [==============================] - 8s 5ms/step - loss: 0.0566 - accuracy: 0.9825 - val_loss: 0.0366 - val_accuracy: 0.9906
Epoch 4/10
1719/1719 [==============================] - 8s 5ms/step - loss: 0.0503 - accuracy: 0.9843 - val_loss: 0.0365 - val_accuracy: 0.9906
Epoch 5/10
1719/1719 [==============================] - 9s 5ms/step - loss: 0.0413 - accuracy: 0.9864 - val_loss: 0.0321 - val_accuracy: 0.9920
Epoch 6/10
1719/1719 [==============================] - 8s 5ms/step - loss: 0.0347 - accuracy: 0.9889 - val_loss: 0.0379 - val_accuracy: 0.9906
Epoch 7/10
1719/1719 [==============================] - 9s 5ms/step - loss: 0.0323 - accuracy: 0.9895 - val_loss: 0.0381 - val_accuracy[...]<jupyter_text>- Layer 1(Conv2D(32):  * Output Size: 28x28 image + 3x3 kernal + 1 stride = 28+3+1 = 32 => (28,28,32)  * number of parameters: 3 * 3 * 1 + 1 * 32 = 320- Layer 2(Conv2D(64):  * Output Size: 28x28 image + 3x3 kernal + 1 stride = (28+3+1)*2 = 64 => (28,28,64)  * number of parameters: 3 * 3 * 32 + 1 * 64 = 18496- Layer 3(MaxPool2D):  * Output Size: (14,14,64), dimensionality is reduced because of pooling  * number of parameters: 0- Layer 4(Flatten):  * Output Size: 14 * 14 * 64 = 12544  * number of parameters: 0- Layer 5(Dropout(0.25):  * Output Size: 12544, same of previous layer  * number of parameters: 0- Layer 6(Dense(128):  * Output Size: 64 + 64 = 128  * number of parameters: 128 * 12544 + 128 = 1605760- Layer 7(Dropout(0.5):  * Output Size: 128, same as previous layer  * number of parameters: 0- Layer 8(Dense(10):  * Output Size: 10, desired output size since there are 10 classes  * number of parameters: 128 * (3 * 3 + 1) = 1290<jupyter_code><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup