# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="djUvWu41mtXa" colab_type="text"
# ##### Copyright 2019 The TensorFlow Authors.

# + id="su2RaORHpReL" colab_type="code" colab={}
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# + [markdown] id="NztQK2uFpXT-" colab_type="text"
# # TensorBoard Profile: Profiling basic training metrics in Keras
#
# <table class="tfo-notebook-buttons" align="left">
#   <td>
#     <a target="_blank" href="https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
#   </td>
#   <td>
#     <a target="_blank" href="https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/tensorboard_profiling_keras.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
#   </td>
#   <td>
#     <a target="_blank" href="https://github.com/tensorflow/tensorboard/blob/master/docs/tensorboard_profiling_keras.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
#   </td>
# </table>

# + [markdown] id="eDXRFe_qp5C3" colab_type="text"
#
# ## Overview
# Performance is critical for machine learning. TensorFlow has a built-in profiler that allows you to record runtime of each ops with very little effort. Then you can visualize the profile result in TensorBoard's *Profile Plugin*. This tutorial focuses on GPU but the Profile Plugin can also be used with TPUs by following the [Cloud TPU Tools](https://cloud.google.com/tpu/docs/cloud-tpu-tools).
#
# This tutorial presents very basic examples to help you learn how to enable profiler when developing your Keras model. You will learn how to use the Keras TensorBoard callback to visualize profile result. *Profiler APIs* and *Profiler Server* mentioned in “Other ways for profiling” allow you to profile non-Keras TensorFlow job.

# + [markdown] id="dG-nnZK9qW9z" colab_type="text"
# ## Prerequisites
#
#
# *   Install latest [TensorBoard](https://www.tensorflow.org/tensorboard) on your local machine.
#
# *   Select “GPU” in the Accelerator drop-down in Notebook Settings (Assuming you run this notebook on Colab).
#
# >![Notebook Settings](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-notebook-settings.png?raw=1)
#
#

# + [markdown] id="DZhGh-G7KoKL" colab_type="text"
# ## Setup

# + id="3U5gdCw_nSG3" colab_type="code" outputId="ecbc68d1-3e87-42d1-8fb8-a23abdc3d59c" colab={"base_uri": "https://localhost:8080/", "height": 119}
# Ensure latest TensorFlow is installed.
# !pip install -q tf-nightly-gpu-2.0-preview
# Load the TensorBoard notebook extension.
# %load_ext tensorboard


# + id="1qIKtOBrqc9Y" colab_type="code" outputId="cba1a5a5-da4e-4df1-b80c-c83f446f8178" colab={"base_uri": "https://localhost:8080/", "height": 34}
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from datetime import datetime
from packaging import version

import functools
import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow.python.keras import backend
from tensorflow.python.keras import layers

import numpy as np

print("TensorFlow version: ", tf.__version__)


# + [markdown] id="8ZM-6NzYgPRn" colab_type="text"
# Confirm TensorFlow can see the GPU.

# + id="gp2p-MemgAIh" colab_type="code" outputId="d459de3e-c9cb-4336-874a-d544f6323ffe" colab={"base_uri": "https://localhost:8080/", "height": 34}
device_name = tf.test.gpu_device_name()
if not tf.test.is_gpu_available():
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))

# + [markdown] id="6YDAoNCN3ZNS" colab_type="text"
# ## Run a simple model with TensorBoard callback
#
# You're now going to use Keras to build a simple model for classifying [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) images using ResNet56 (Reference: [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)).
#
# Following ResNet model code is copied from [TensorFlow models garden](https://github.com/tensorflow/models/blob/master/official/resnet/keras/resnet_cifar_model.py).
#
#

# + id="ImCFrQ74eerE" colab_type="code" colab={}
BATCH_NORM_DECAY = 0.997
BATCH_NORM_EPSILON = 1e-5
L2_WEIGHT_DECAY = 2e-4


def identity_building_block(input_tensor,
                            kernel_size,
                            filters,
                            stage,
                            block,
                            training=None):
  """The identity block is the block that has no conv layer at shortcut.

  Arguments:
    input_tensor: input tensor
    kernel_size: default 3, the kernel size of
        middle conv layer at main path
    filters: list of integers, the filters of 3 conv layer at main path
    stage: integer, current stage label, used for generating layer names
    block: current block label, used for generating layer names
    training: Only used if training keras model with Estimator.  In other
      scenarios it is handled automatically.

  Returns:
    Output tensor for the block.
  """
  filters1, filters2 = filters
  if tf.keras.backend.image_data_format() == 'channels_last':
    bn_axis = 3
  else:
    bn_axis = 1
  conv_name_base = 'res' + str(stage) + block + '_branch'
  bn_name_base = 'bn' + str(stage) + block + '_branch'

  x = tf.keras.layers.Conv2D(filters1, kernel_size,
                             padding='same',
                             kernel_initializer='he_normal',
                             kernel_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             bias_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             name=conv_name_base + '2a')(input_tensor)
  x = tf.keras.layers.BatchNormalization(axis=bn_axis,
                                         name=bn_name_base + '2a',
                                         momentum=BATCH_NORM_DECAY,
                                         epsilon=BATCH_NORM_EPSILON)(
                                             x, training=training)
  x = tf.keras.layers.Activation('relu')(x)

  x = tf.keras.layers.Conv2D(filters2, kernel_size,
                             padding='same',
                             kernel_initializer='he_normal',
                             kernel_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             bias_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             name=conv_name_base + '2b')(x)
  x = tf.keras.layers.BatchNormalization(axis=bn_axis,
                                         name=bn_name_base + '2b',
                                         momentum=BATCH_NORM_DECAY,
                                         epsilon=BATCH_NORM_EPSILON)(
                                             x, training=training)

  x = tf.keras.layers.add([x, input_tensor])
  x = tf.keras.layers.Activation('relu')(x)
  return x


def conv_building_block(input_tensor,
                        kernel_size,
                        filters,
                        stage,
                        block,
                        strides=(2, 2),
                        training=None):
  """A block that has a conv layer at shortcut.

  Arguments:
    input_tensor: input tensor
    kernel_size: default 3, the kernel size of
        middle conv layer at main path
    filters: list of integers, the filters of 3 conv layer at main path
    stage: integer, current stage label, used for generating layer names
    block: current block label, used for generating layer names
    strides: Strides for the first conv layer in the block.
    training: Only used if training keras model with Estimator.  In other
      scenarios it is handled automatically.

  Returns:
    Output tensor for the block.

  Note that from stage 3,
  the first conv layer at main path is with strides=(2, 2)
  And the shortcut should have strides=(2, 2) as well
  """
  filters1, filters2 = filters
  if tf.keras.backend.image_data_format() == 'channels_last':
    bn_axis = 3
  else:
    bn_axis = 1
  conv_name_base = 'res' + str(stage) + block + '_branch'
  bn_name_base = 'bn' + str(stage) + block + '_branch'

  x = tf.keras.layers.Conv2D(filters1, kernel_size, strides=strides,
                             padding='same',
                             kernel_initializer='he_normal',
                             kernel_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             bias_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             name=conv_name_base + '2a')(input_tensor)
  x = tf.keras.layers.BatchNormalization(axis=bn_axis,
                                         name=bn_name_base + '2a',
                                         momentum=BATCH_NORM_DECAY,
                                         epsilon=BATCH_NORM_EPSILON)(
                                             x, training=training)
  x = tf.keras.layers.Activation('relu')(x)

  x = tf.keras.layers.Conv2D(filters2, kernel_size, padding='same',
                             kernel_initializer='he_normal',
                             kernel_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             bias_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             name=conv_name_base + '2b')(x)
  x = tf.keras.layers.BatchNormalization(axis=bn_axis,
                                         name=bn_name_base + '2b',
                                         momentum=BATCH_NORM_DECAY,
                                         epsilon=BATCH_NORM_EPSILON)(
                                             x, training=training)

  shortcut = tf.keras.layers.Conv2D(filters2, (1, 1), strides=strides,
                                    kernel_initializer='he_normal',
                                    kernel_regularizer=
                                    tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                                    bias_regularizer=
                                    tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                                    name=conv_name_base + '1')(input_tensor)
  shortcut = tf.keras.layers.BatchNormalization(
      axis=bn_axis, name=bn_name_base + '1',
      momentum=BATCH_NORM_DECAY, epsilon=BATCH_NORM_EPSILON)(
          shortcut, training=training)

  x = tf.keras.layers.add([x, shortcut])
  x = tf.keras.layers.Activation('relu')(x)
  return x


def resnet_block(input_tensor,
                 size,
                 kernel_size,
                 filters,
                 stage,
                 conv_strides=(2, 2),
                 training=None):
  """A block which applies conv followed by multiple identity blocks.

  Arguments:
    input_tensor: input tensor
    size: integer, number of constituent conv/identity building blocks.
    A conv block is applied once, followed by (size - 1) identity blocks.
    kernel_size: default 3, the kernel size of
        middle conv layer at main path
    filters: list of integers, the filters of 3 conv layer at main path
    stage: integer, current stage label, used for generating layer names
    conv_strides: Strides for the first conv layer in the block.
    training: Only used if training keras model with Estimator.  In other
      scenarios it is handled automatically.

  Returns:
    Output tensor after applying conv and identity blocks.
  """

  x = conv_building_block(input_tensor, kernel_size, filters, stage=stage,
                          strides=conv_strides, block='block_0',
                          training=training)
  for i in range(size - 1):
    x = identity_building_block(x, kernel_size, filters, stage=stage,
                                block='block_%d' % (i + 1), training=training)
  return x

def resnet(num_blocks, classes=10, training=None):
  """Instantiates the ResNet architecture.

  Arguments:
    num_blocks: integer, the number of conv/identity blocks in each block.
      The ResNet contains 3 blocks with each block containing one conv block
      followed by (layers_per_block - 1) number of idenity blocks. Each
      conv/idenity block has 2 convolutional layers. With the input
      convolutional layer and the pooling layer towards the end, this brings
      the total size of the network to (6*num_blocks + 2)
    classes: optional number of classes to classify images into
    training: Only used if training keras model with Estimator.  In other
    scenarios it is handled automatically.

  Returns:
    A Keras model instance.
  """

  input_shape = (32, 32, 3)
  img_input = layers.Input(shape=input_shape)

  if backend.image_data_format() == 'channels_first':
    x = layers.Lambda(lambda x: backend.permute_dimensions(x, (0, 3, 1, 2)),
                      name='transpose')(img_input)
    bn_axis = 1
  else:  # channel_last
    x = img_input
    bn_axis = 3

  x = tf.keras.layers.ZeroPadding2D(padding=(1, 1), name='conv1_pad')(x)
  x = tf.keras.layers.Conv2D(16, (3, 3),
                             strides=(1, 1),
                             padding='valid',
                             kernel_initializer='he_normal',
                             kernel_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             bias_regularizer=
                             tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                             name='conv1')(x)
  x = tf.keras.layers.BatchNormalization(axis=bn_axis, name='bn_conv1',
                                         momentum=BATCH_NORM_DECAY,
                                         epsilon=BATCH_NORM_EPSILON)(
                                             x, training=training)
  x = tf.keras.layers.Activation('relu')(x)

  x = resnet_block(x, size=num_blocks, kernel_size=3, filters=[16, 16],
                   stage=2, conv_strides=(1, 1), training=training)

  x = resnet_block(x, size=num_blocks, kernel_size=3, filters=[32, 32],
                   stage=3, conv_strides=(2, 2), training=training)

  x = resnet_block(x, size=num_blocks, kernel_size=3, filters=[64, 64],
                   stage=4, conv_strides=(2, 2), training=training)

  x = tf.keras.layers.GlobalAveragePooling2D(name='avg_pool')(x)
  x = tf.keras.layers.Dense(classes, activation='softmax',
                            kernel_initializer='he_normal',
                            kernel_regularizer=
                            tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                            bias_regularizer=
                            tf.keras.regularizers.l2(L2_WEIGHT_DECAY),
                            name='fc10')(x)

  inputs = img_input
  # Create model.
  model = tf.keras.models.Model(inputs, x, name='resnet56')

  return model


resnet20 = functools.partial(resnet, num_blocks=3)
resnet32 = functools.partial(resnet, num_blocks=5)
resnet56 = functools.partial(resnet, num_blocks=9)
resnet10 = functools.partial(resnet, num_blocks=110)

# + [markdown] id="1lAek-Lye8_q" colab_type="text"
# Download CIFAR-10 data from [TensorFlow Datasets](https://www.tensorflow.org/datasets).

# + id="H8A67-bNXzsx" colab_type="code" colab={}
cifar_builder = tfds.builder('cifar10')
cifar_builder.download_and_prepare()


# + [markdown] id="21jm6LOSq9EN" colab_type="text"
# Build data input pipeline and compile ResNet56 model.

# + id="j-ryO6OxnQH_" colab_type="code" colab={}
HEIGHT = 32
WIDTH = 32
NUM_CHANNELS = 3
NUM_CLASSES = 10
BATCH_SIZE = 128

def preprocess_data(record):
  image = record['image']
  label = record['label']
  
  # Resize the image to add four extra pixels on each side.
  image = tf.image.resize_with_crop_or_pad(
      image, HEIGHT + 8, WIDTH + 8)

  # Randomly crop a [HEIGHT, WIDTH] section of the image.
  image = tf.image.random_crop(image, [HEIGHT, WIDTH, NUM_CHANNELS])

  # Randomly flip the image horizontally.
  image = tf.image.random_flip_left_right(image)

  # Subtract off the mean and divide by the variance of the pixels.
  image = tf.image.per_image_standardization(image)
  
  label = tf.compat.v1.sparse_to_dense(label, (NUM_CLASSES,), 1)
  return image, label

train_data = cifar_builder.as_dataset(split=tfds.Split.TRAIN)
train_data = train_data.repeat()
train_data = train_data.map(
    lambda value: preprocess_data(value))
train_data = train_data.shuffle(1024)

train_data = train_data.batch(BATCH_SIZE)

model = resnet56(classes=NUM_CLASSES)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy',
              metrics=['categorical_accuracy'])


# + [markdown] id="_5llFQBKHFmA" colab_type="text"
# When creating TensorBoard callback, you can specify the batch num you want to profile. By default, TensorFlow will profile the second batch, because many one time graph optimizations run on the first batch. You can modify it by setting `profile_batch`. You can also turn off profiling by setting it to 0.
#
# This time, you will profile on the third batch.

# + id="WmY-2znGJxNY" colab_type="code" colab={}
log_dir="logs/profile/" + datetime.now().strftime("%Y%m%d-%H%M%S")

tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1, profile_batch = 3)

# + [markdown] id="ylDhh7zlJ273" colab_type="text"
# Start training by calling [Model.fit()](https://https://www.tensorflow.org/api_docs/python/tf/keras/models/Model#fit). 

# + id="LEb_1HETJ_tX" colab_type="code" outputId="5c22c9f5-6901-4ff6-a5fd-f2700010e85c" colab={"base_uri": "https://localhost:8080/", "height": 275}
model.fit(train_data,
          steps_per_epoch=20,
          epochs=5, 
          callbacks=[tensorboard_callback])

# + [markdown] id="042k7GMERVkx" colab_type="text"
# ## Visualizing profile result using TensorBoard
#
# Unfortunately, due to [#1913](https://github.com/tensorflow/tensorboard/issues/1913), you cannot use TensorBoard in Colab to visualize profile result. You are going to download the logdir and start TensorBoard on your local machine.
#
# Compress logdir:
#

# + id="6pck56gKReON" colab_type="code" colab={}
# !tar -zcvf logs.tar.gz logs/profile/

# + [markdown] id="TZOf_K4L-Nkv" colab_type="text"
# Download `logdir.tar.gz` by right-clicking it in “Files” tab.
#
# ![Download](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-download-logdir.png?raw=1)
#
# Please make sure you have the latest [TensorBoard](https://www.tensorflow.org/tensorboard) installed on you local machine as well. Execute following commands on your local machine:
#
# ```
# > cd download/directory
# > tar -zxvf logs.tar.gz
# > tensorboard --logdir=logs/ --port=6006
#
# ```
#
#
#

# + [markdown] id="ciSIRibhRi6N" colab_type="text"
# Open a new tab in your Chrome browser and navigate to [localhost:6006](http://localhost:6006) and then click “Profile” tab. You may see the profile result like this:
#
# ![Trace View](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-trace-viewer.png?raw=1)
#
#
#

# + [markdown] id="roE94vH9mJ6k" colab_type="text"
# ## Trace Viewer
# Once you click the profile tab, you will see Trace Viewer. The page displays a timeline of different events that happened on the CPU and the accelerator during the collection period.
#
# The Trace Viewer shows multiple *event groups* on the vertical axis. Each event group has multiple horizontal *tracks*, filled with trace events. The *track* is basically an event timeline for events executed on a thread or a GPU stream. Individual events are the colored, rectangular blocks on the timeline tracks. Time moves from left to right.
#
# You can navigate through the result using `w` (zoom in), `s` (zoom out), `a` (scroll left), `d` (scroll right).
#
# A single rectangle represents a *trace event*: when it began, and when it ended. To study an individual rectangle, you can click on it after selecting the mouse cursor icon in the floating tool bar. This will display information about the rectangle, such as its Start time and Duration.
#
# In addition to clicking, you can drag the mouse to select a rectangle covering a group of trace events. This will give you a list of events that intersect that rectangle and summarize them for you. The `m` key can be used to measure the time duration of the selected events.
#
# ![List of Events](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-trace-viewer-select.png?raw=1)
#
# The trace events are collected from three sources:
#
#
# *   **CPU:** CPU events are under event group named `/host:CPU`. Each track represents a thread on CPU. E.g. input pipeline events, GPU op scheduling events, CPU ops execution events, etc.
# *   **GPU:** GPU events are under event groups prefixed by `/device:GPU:`.  Except `stream:all`, each event group represents one stream on GPU. `stream::all` aggregates all events on one GPU. E.g. Memory copy events, Kernel execution events, etc.
# *   **TensorFlow Runtime:** Runtime events are under event groups prefixed by `/job:`. Runtime events represent the TensorFlow ops invoked by python program. E.g. tf.function execution events, etc.
#
#
#
#
#
#

# + [markdown] id="XAcO9sj4B2DK" colab_type="text"
# ## Debug Performance
# Now, you're going to use the Trace Viewer to improve your model's performance. 
#
# Let's go back to the profile result you have just captured.
#
# ![GPU kernel](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-idle-gpu.png?raw=1)
#
# GPU events show that GPU has nothing to do at all in the first harf of the step.
#
# ![CPU events](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-input-cpu.png?raw=1)
#
# CPU events show that CPU is occupied by data input pipeline in the beginning of this step.
#
# ![Runtime](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-blocking-runtime.png?raw=1)
#
# In TensorFlow runtime, there is a big block named `Iterator::GetNextSync`, which is a blocking call to get the next batch from data input pipeline. And it blocks the training step. So if you could prepare the input data for step `s` in `s-1` step, you can probably train this model faster.
#
# You can achieve it by using [tf.data.prefetch](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#prefetch).

# + id="JZ6UeYx9TT2T" colab_type="code" colab={}
train_data = cifar_builder.as_dataset(split=tfds.Split.TRAIN)
train_data = train_data.repeat()
train_data = train_data.map(
    lambda value: preprocess_data(value))
train_data = train_data.shuffle(1024)
train_data = train_data.batch(BATCH_SIZE)

# It will prefetch the data in (s-1) step
train_data = train_data.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)

# + [markdown] id="EfD6pnhgT7q3" colab_type="text"
# Re-run the model.

# + id="tgFqaHYBUADP" colab_type="code" outputId="a2e5ab1a-1390-4637-a630-c84035de3879" colab={"base_uri": "https://localhost:8080/", "height": 204}
log_dir="logs/profile/" + datetime.now().strftime("%Y%m%d-%H%M%S")

tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1, profile_batch = 3)

model.fit(train_data,
          steps_per_epoch=20,
          epochs=5, 
          callbacks=[tensorboard_callback])

# + [markdown] id="LFtVDt-9UVkn" colab_type="text"
# Woohoo! You have just improvd training performance from *~235ms/step* to *~200ms/step*.  

# + id="if5LuLl_pgna" colab_type="code" colab={}
# !tar -zcvf logs.tar.gz logs/profile/

# + [markdown] id="aBBKSVJVp4yk" colab_type="text"
#
# Download `logs` directory again to see the new profile result in TensorBoard.
#
# ![TF Runtime](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-prefetch-runtime.png?raw=1)
#
# The big `Iterator::GetNextSync` block is not there anymore.
#
# Good job!
#
# Apparently, this is still not the best performance yet. Please try by yourself to see if you can have further improvements.
#
# Some useful references for performance tuning:
#
#
# *   [Data input pipeline](https://www.tensorflow.org/guide/data_performance)
# *   [Training Performance: A user’s guide to converge faster (TensorFlow Dev Summit 2018)](https://www.youtube.com/watch?v=SxOsJPaxHME)
#
#
#

# + [markdown] id="pLfa4vMn626q" colab_type="text"
# ## Other ways for profiling
# In addition to TensorBoard callback, TensorFlow also provides two additional way to trigger profiler manually: *Profiler APIs* and *Profiler Service*.
#
# Note: Please don't run multiple profilers at the same time. If you want to use either Profiler APIs or Profiler Service with TensorBoard callback, ensure the `profile_batch` parameter is set to 0.
#
#
#
#
#

# + [markdown] id="gt9Dm8PkL1FI" colab_type="text"
# ### Profiler APIs

# + id="VYywGzC2GQ8w" colab_type="code" colab={}
# Context manager APIs
with tf.python.eager.profiler.Profiler('logdir_path'):
  # do your training here
  pass


# Function APIs
tf.python.eager.profiler.start()
# do your training here
profiler_result = tf.python.eager.profiler.stop()
tf.python.eager.profiler.save('logdir_path', profiler_result)

# + [markdown] id="NSHEq0rIHHBs" colab_type="text"
# ### Profiler Service
#
#

# + id="USTAe02KHcql" colab_type="code" colab={}
# This API will start a gRPC server with your TensorFlow job which can receive
# on-demand profiling request.
tf.python.eager.profiler.start_profiler_server(6009)

# Your TensorFlow program here

# + [markdown] id="AgIro3xQIXUa" colab_type="text"
# Then you can send profiling request to profiler server to perform on-demand profiling on TensorBoard by clicking the “Capture Profile” button:
#
# ![CAPTURE PROFILE](https://github.com/tensorflow/tensorboard/blob/master/docs/images/profiler-capture.png?raw=1)
#
# A message will show up after successfully captured. Then you can refresh TensorBoard to visualize the result.

StarcoderdataJupyterScriptsDedupFiltered