  Quantization is a popular technique used in Deep Neural Networks (DNN)
inference to reduce the size of models and improve the overall numerical
performance by exploiting native hardware. This paper attempts to conduct an
elaborate performance characterization of the benefits of using quantization
techniques -- mainly FP16/INT8 variants with static and dynamic schemes --
using the MLPerf Edge Inference benchmarking methodology. The study is
conducted on Intel x86 processors and Raspberry Pi device with ARM processor.
The paper uses a number of DNN inference frameworks, including OpenVINO (for
Intel CPUs only), TensorFlow Lite (TFLite), ONNX, and PyTorch with MobileNetV2,
VGG-19, and DenseNet-121. The single-stream, multi-stream, and offline
scenarios of the MLPerf Edge Inference benchmarks are used for measuring
latency and throughput in our experiments. Our evaluation reveals that OpenVINO
and TFLite are the most optimized frameworks for Intel CPUs and Raspberry Pi
device, respectively. We observe no loss in accuracy except for the static
quantization techniques. We also observed the benefits of using quantization
for these optimized frameworks. For example, INT8-based quantized models
deliver $3.3\times$ and $4\times$ better performance over FP32 using OpenVINO
on Intel CPU and TFLite on Raspberry Pi device, respectively, for the MLPerf
offline scenario. To the best of our knowledge, this paper is the first one
that presents a unique characterization study characterizing the impact of
quantization for a range of DNN inference frameworks -- including OpenVINO,
TFLite, PyTorch, and ONNX -- on Intel x86 processors and Raspberry Pi device
with ARM processor using the MLPerf Edge Inference benchmark methodology.
