<jupyter_start><jupyter_text>ResNet-50 comparison: compiled vs uncompiled, featuring InferentiaIn this notebook we will see how to deploy a pretrained model from the PyTorch Vision library, in particular a ResNet50, to Amazon SageMaker. We will also test how it performs on different hardware configurations, and the effects of model compilation with Amazon SageMaker Neo. In a nutshell, we will test:- ResNet50 on a ml.c5.xlarge, uncompiled- ResNet50 on a ml.g4dn.xlarge, uncompiled- ResNet50 on a ml.c5.xlarge, compiled- ResNet50 on a ml.g4dn.xlarge, compiled- ResNet50 on a ml.inf1.xlarge, compiled  Set-up model and SageMaker helper functions<jupyter_code>import sagemaker
from sagemaker import Session, get_execution_role
from sagemaker.pytorch.model import PyTorchModel
from sagemaker.utils import name_from_base

print(sagemaker.__version__)

sess = Session()
bucket = sess.default_bucket()
role = get_execution_role()
endpoints = {}<jupyter_output>2.47.2.post0<jupyter_text>Let's download the model for the PyTorch Hub, and create an archive that can be used by SageMaker to deploy this model. For using PyTorch in Script Mode, Amazon SageMaker expects a single archive file in `.tar.gz` format, containing a model file and the code for inference in a `code` folder. The structure of the archive will be as follows: ```/model.tar.gz/--- model.pth/--- code//--- /--- inference.py/--- /--- requirements.txt (optional)```By setting the variable `download_the_model=False`, you can skip the download phase and provide your own path to S3 in the `model_data` variable.<jupyter_code>download_the_model = False

if download_the_model:
    import torch, tarfile
    # Load the model
    model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet50', pretrained=True)
    inp = torch.rand(1, 3, 224, 224)
    model_trace = torch.jit.trace(model, inp)
    # Save your model. The following code saves it with the .pth file extension
    model_trace.save('model.pth')
    with tarfile.open('model.tar.gz', 'w:gz') as f:
        f.add('model.pth')
        f.add('code/uncompiled-inference.py', 'code/inference.py')
    f.close()
    pytorch_resnet50_prefix = 'pytorch/resnet50'
    model_data = sess.upload_data('model.tar.gz', bucket, pytorch_resnet50_prefix)
else:
    pytorch_resnet50_prefix = 'pytorch/resnet50'
    model_data = f's3://{bucket}/{pytorch_resnet50_prefix}/model.tar.gz'
    
print(f'Model stored in {model_data}')<jupyter_output>Model stored in s3://sagemaker-us-east-1-859755744029/pytorch/resnet50/model.tar.gz<jupyter_text>Deploy and test on CPUIn our first test, we will deploy the model on a `ml.c5.xlarge` instance, without compiling the model. Although this is a CNN, it is still possible to run it on CPU, although the performances won't be that good. This can give us a nice baseline of the performances of our model.<jupyter_code>pth_model = PyTorchModel(model_data=model_data,
        entry_point='uncompiled-inference.py',
        source_dir='code',
        role=role,
        framework_version='1.7',
        py_version='py3'
)
predictor = pth_model.deploy(1, 'ml.c5.xlarge')
endpoints['cpu_uncompiled'] = predictor.endpoint_name
predictor.endpoint_name<jupyter_output><empty_output><jupyter_text>Deploy and test on GPUThe instance chosen this time is a `ml.g4dn.xlarge`. It has great throughput and the cheapest way of running GPU inferences on the AWS cloud.<jupyter_code>pth_model = PyTorchModel(model_data=model_data,
     entry_point='uncompiled-inference.py',
     source_dir='code',
     role=role,
     framework_version='1.6',
     py_version='py3'
)
predictor = pth_model.deploy(1, 'ml.g4dn.xlarge')
endpoints['gpu_uncompiled'] = predictor.endpoint_name
predictor.endpoint_name<jupyter_output><empty_output><jupyter_text>Compiled ModelsA common tactic in more advanced use cases is to improve model performances, in terms of latency and throughput, by compiling the model. Amazon SageMaker features its own compiler, Amazon SageMaker Neo, that enables data scientists to optimize ML models for inference on SageMaker in the cloud and supported devices at the edge.You start with a machine learning model already built with DarkNet, Keras, MXNet, PyTorch, TensorFlow, TensorFlow-Lite, ONNX, or XGBoost and trained in Amazon SageMaker or anywhere else. Then you choose your target hardware platform, which can be a SageMaker hosting instance or an edge device based on processors from Ambarella, Apple, ARM, Intel, MediaTek, Nvidia, NXP, Qualcomm, RockChip, Texas Instruments, or Xilinx. With a single click, SageMaker Neo optimizes the trained model and compiles it into an executable. For inference in the cloud, SageMaker Neo speeds up inference and saves cost by creating an inference optimized container in SageMaker hosting. For inference at the edge, SageMaker Neo saves developers months of manual tuning by automatically tuning the model for the selected operating system and processor hardware.  Create the `model_data` for compilationTo compile the model, we need to provide a `tar.gz` archive just like before, with very few changes. In particular, since SageMaker will use a different DL runtime for running compiled models, we will let it use the default function for serving the model, and only provide a script containing how to preprocess data. Let's create this archive and uplaod it to S3.<jupyter_code>import tarfile 

with tarfile.open('model-to-compile.tar.gz', 'w:gz') as f:
    f.add('model.pth')
    f.add('code/compiled-inference.py', 'code/inference.py')
f.close()
model_data = sess.upload_data('model-to-compile.tar.gz', bucket, pytorch_resnet50_prefix)
output_path = f's3://{bucket}/{pytorch_resnet50_prefix}/compiled'<jupyter_output><empty_output><jupyter_text>Compile for CPULet's run the same baseline test from before, and compile and deploy for CPU instances.<jupyter_code>pth_model = PyTorchModel(model_data=model_data,
        entry_point='compiled-inference.py',
        source_dir='code',
        role=role,
        framework_version='1.6',
        py_version='py3'
)
output_path = f's3://{bucket}/{pytorch_resnet50_prefix}/compiled'

compiled_model = pth_model.compile(
    target_instance_family='ml_c5',
    input_shape={"input0": [1, 3, 224, 224]},
    output_path=output_path,
    role=role,
    job_name=name_from_base(f'pytorch-resnet50-c5')
)
predictor = compiled_model.deploy(1, 'ml.c5.xlarge')
endpoints['cpu_compiled'] = predictor.endpoint_name
predictor.endpoint_name<jupyter_output><empty_output><jupyter_text>Compile for GPU<jupyter_code>pth_model = PyTorchModel(model_data=model_data,
                     entry_point='compiled-inference.py',
                     source_dir='code',
                     role=role,
                     framework_version='1.6',
                     py_version='py3'
                    )
output_path = f's3://{bucket}/{pytorch_resnet50_prefix}/compiled'

compiled_model = pth_model.compile(
    target_instance_family='ml_g4dn',
    input_shape={"input0": [1, 3, 224, 224]},
    output_path=output_path,
    role=role,
    job_name=name_from_base(f'pytorch-resnet50-g4dn')
)
predictor = compiled_model.deploy(1, 'ml.g4dn.xlarge')
endpoints['gpu_compiled'] = predictor.endpoint_name
predictor.endpoint_name
print(1)<jupyter_output>1<jupyter_text>Compile for Inferentia instancesThere is one more thing we can try to improve our model performances: using Inferentia instances.Amazon EC2 Inf1 instances deliver high-performance ML inference at the lowest cost in the cloud. They deliver up to 2.3x higher throughput and up to 70% lower cost per inference than comparable current generation GPU-based Amazon EC2 instances. Inf1 instances are built from the ground up to support machine learning inference applications. They feature up to 16 AWS Inferentia chips, high-performance machine learning inference chips designed and built by AWS.<jupyter_code>pth_model = PyTorchModel(
    model_data=model_data,
    entry_point='compiled-inference.py',
    source_dir='code',
    role=role,
    framework_version='1.7',
    py_version='py3'
)
compiled_model = pth_model.compile(
    target_instance_family='ml_inf1',
    input_shape={"input0": [1, 3, 224, 224]},
    output_path=output_path,
    role=role,
    job_name=name_from_base(f'pytorch-resnet50-inf1'),
    compile_max_run=1000
)
predictor = compiled_model.deploy(1, 'ml.inf1.xlarge')
endpoints['inferentia'] = predictor.endpoint_name
predictor.endpoint_name<jupyter_output><empty_output><jupyter_text>Test For testing our models and endpoints, we will use the following picture of a beagle pup, freely available on [Wikimedia](https://commons.wikimedia.org/wiki/File:Beagle_puppy_sitting_on_grass.jpg). We will pass it to our endpoints as `application/x-image`, and no particular pre-processing is needed on client-side.<jupyter_code>from IPython.display import Image

Image('doggo.jpg')<jupyter_output><empty_output><jupyter_text>Finally, we will use the following function to benchmark our SageMaker endpoints, measuring the latency of our predictions. This specific version uses both the `Predictor` from the SageMaker Python SDK and boto3's `invoke_endpoint()` function - just change the last parameter from `boto3` to `sm` if you want to use the Python SDK.<jupyter_code>from load_test import load_tester

num_thread = 10

from IPython.display import JSON
JSON(endpoints)
# CPU - Uncompiled
load_tester(num_thread, endpoints['cpu_uncompiled'], 'doggo.jpg', 'boto3')<jupyter_output>Using boto3 for requests.
 TPS  |  P50  |  P90  |  P95  |  P99  |  err  
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
4.100|154.09832|170.55326|171.40527|179.54471 |   0
5.900|167.96876|171.36371|171.82379|185.01005 |   0<jupyter_text>Let's discuss the above results:as we can from the latency tests, the model way too long to generate an inference, averaging about 6 transactions per second (TPS). This of course may be sufficient for some low throughput use cases, rarely used instances, but it's very likely that it won't be sufficient. Let's see how those numbers change when using a GPU.<jupyter_code># GPU - Uncompiled
load_tester(num_thread, endpoints['gpu_uncompiled'], 'doggo.jpg', 'boto3')<jupyter_output>Using boto3 for requests.
 TPS  |  P50  |  P90  |  P95  |  P99  |  err  
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
71.900|11.80012|14.37088|15.16426|71.54423 |   0
74.200|11.80911|14.11828|15.55170|80.74821 |   0
73.600|11.86891|13.91335|14.91341|74.74922 |   0
75.700|11.81529|13.47131|14.11105|79.58128 |   0
75.500|11.76775|13.63597|14.26841|65.81278 |   0
73.000|11.74483|13.89458|15.24351|75.62139 |   0
75.300|11.57395|13.56684|14.60281|95.03487 |   0
74.100|11.69744|13.78759|16.07176|77.24368 |   0
75.100|11.82399|13.74409|16.19095|77.27810 |   0
76.300|11.68339|13.35336|15.39730|82.90827 |   0
76.400|12.08593|13.35388|14.11228|70.84170 |   0
75.900|12.08845|13.37607|14.13612|64.37122 |   0
76.500|11.65937|13.01402|13.38935|73.97340 |   0
76.800|11.45495|13.07725|13.75206|79.20948 |   0
76.200|11.65578|13.37109|14.84504|68.58632 |   0
76.300|11.56737|13.29634|13.75376|84.87934 |   0
76.200|11.7[...]<jupyter_text>Now we're talking! The GPU helps us achieve 77 TPS on average, with a much lower latency percentile over the board. Nice!<jupyter_code># CPU - Compiled
load_tester(num_thread, endpoints['cpu_compiled'], 'doggo.jpg', 'boto3')<jupyter_output>Using boto3 for requests.
 TPS  |  P50  |  P90  |  P95  |  P99  |  err  
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
1.400|27.34789|43.17109|45.49675|46.66742 |   0
13.200|75.15947|77.39340|77.70184|78.02003 |   0
13.100|75.52688|77.68237|78.34686|79.13477 |   0
13.300|75.38521|77.44209|77.88107|79.08387 |   0
13.200|75.31815|77.52130|78.14982|79.18169 |   0
13.200|75.35261|77.46928|78.06024|79.13503 |   0
13.200|75.35843|77.41700|77.87167|79.08827 |   0
13.200|75.29122|77.43950|78.07704|79.71710 |   0
13.100|75.37787|77.48869|78.26608|81.16619 |   0
13.200|75.24554|77.36055|78.05424|81.42112 |   0
13.200|75.21513|77.36815|78.14131|81.42112 |   0
13.200|75.25020|77.41456|78.15042|81.[...]<jupyter_text>With a simple compilation job, we more than DOUBLED the performances of our model on our c5 instance, achieving 13 TPS and half the latency percentiles. Let's see if the same results can be seen on GPU.<jupyter_code># GPU - Compiled
load_tester(num_thread, endpoints['gpu_compiled'], 'doggo.jpg', 'boto3')<jupyter_output>Using boto3 for requests.
 TPS  |  P50  |  P90  |  P95  |  P99  |  err  
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
127.200|6.18587|7.55885|7.92916|46.42042 |   0
134.800|6.25129|7.63310|8.23157|43.91295 |   0
140.900|6.20703|7.22852|7.78228|53.26434 |   0
138.300|6.19562|6.83635|7.07245|58.19304 |   0
142.400|6.08956|6.92130|7.23731|46.92067 |   0
142.500|6.20135|7.51865|8.06668|51.33103 |   0
143.200|6.24374|6.97204|7.34329|30.09887 |   0
144.000|6.15883|6.95482|7.13942|48.93987 |   0
143.200|6.18212|6.93569|7.06721|47.13597 |   0
144.300|5.55439|6.83386|6.90111|56.21996 |   0
144.400|6.20829|6.94591|7.13983|9.06043 |   0
143.700|6.07319|6.87563|7.07919|73.52560 |   0
144.200|6.09211|6.83380|6.92895|49.54501 |   0
144.700|5.55294|6.85225|6.91850|63.36337 |   0
144.300|6.19800|6.91495|7.19666|29.78611 |   0
144.400|5.77911|6.87983|7.04087|61.14207 |   0
144.600|6.13239|6.87113|6.99401|52.21058 |   0
144.400|5.83435|6.88874|7.07376|53.57051 [...]<jupyter_text>Results are also consistent in the compiled version of the model on GPU. Almost double the TPS, with 7 ms latency. Let's take it one step further and test Inferentia.<jupyter_code># Inferentia
load_tester(num_thread, endpoints['inferentia'], 'doggo.jpg', 'boto3')<jupyter_output>Using boto3 for requests.
 TPS  |  P50  |  P90  |  P95  |  P99  |  err  
0.000|0.00000|0.00000|0.00000|0.00000 |   0
0.000|0.00000|0.00000|0.00000|0.00000 |   0
121.200|4.15289|5.93244|6.44113|7.80767 |   0
227.400|3.49829|4.67301|5.47904|8.46626 |   0
220.600|3.69167|5.06214|5.70987|7.27710 |   0
259.000|3.41871|4.28656|4.66167|8.99143 |   0
259.000|3.41253|4.19616|4.50734|5.42752 |   0
247.200|3.53762|4.24550|4.48812|5.24465 |   0
270.800|3.35592|4.15566|4.44887|5.29431 |   0
274.700|3.45520|4.20523|4.41165|4.96067 |   0
260.300|3.36056|4.15711|4.68477|9.28876 |   0
264.000|3.30304|4.11524|4.47264|9.47070 |   0
252.500|3.27674|3.91348|4.23365|5.38313 |   0
266.600|3.23882|3.94166|4.29280|5.42367 |   0
277.300|3.34935|4.16940|4.50558|5.50479 |   0
277.100|3.38901|4.08751|4.33727|4.99565 |   0
280.400|3.35106|4.01841|4.37606|5.20386 |   0
275.700|3.27066|3.95396|4.20007|4.99382 |   0
281.400|3.34144|4.08846|4.33117|4.84749 |   0
275.700|3.26554|3.98474|4.24265|4.85687 |   0<jupyter_text>The best results so far! Up to 290 TPS at the same latency percentiles than GPU, or lower. All of this for a fraction of the cost.  Reviewing the resultsLet's plot the results obtained from the previous tests, by taking into account the last printed line of each load testing. We will ignore the error rate, and take into account the TPS obtained with our tests dividing it by the cost of the machine in the region of testing (DUB) - a metric we will call "throughput per dollar".<jupyter_code>import matplotlib.pyplot as plt
from io import StringIO
import pandas as pd

data = StringIO('''
experiment|TPS|p50|cost/hour
c5|5.900|167.96876|0.23
c5 + Neo|13.200|75.25020|0.23
g4dn|76.200|11.75612|0.821
g4dn + Neo|144.400|5.83435|0.821
inf1|275.700|3.26554|0.33
''')

df = pd.read_csv(data, sep='|')
df['throughput per dollar'] = df['TPS']/df['cost/hour']

ax = df.plot(x="experiment", y="TPS", legend=False, linewidth=3)
df.plot(x="experiment", y="p50", ax=ax, legend=False, color="r", linewidth=3)
df.plot(x="experiment", y="throughput per dollar", ax=ax, legend=False, color="g", linewidth=3)
for x in df.experiment:
    plt.axvline(x=x, ls='--', color='black', linewidth=1)
ax.figure.legend()
plt.rcParams["figure.figsize"] = (8,5)
plt.show()<jupyter_output><empty_output><jupyter_text>The results speak for themselves. The `inf1` instance proves extremely competitive, thanks to its high TPS, incredibly low latency and undeniable value. Compiling the model for a GPU just like the `g4dn` is also a very interesting approach.  Clean-up<jupyter_code>for endpoint in endpoints:
    pred = sagemaker.predictor.Predictor(endpoint_name=endpoints[endpoint])
    pred.delete_endpoint()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup