Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Efficient 8-Bit Quantization of Transformer Neural Machine Language
  Translation Model   In this work, we quantize a trained Transformer machine language translation
model leveraging INT8/VNNI instructions in the latest Intel$^\circledR$
Xeon$^\circledR$ Cascade Lake processors to improve inference performance while
maintaining less than 0.5$\%$ drop in accuracy. To the best of our knowledge,
this is the first attempt in the industry to quantize the Transformer model.
This has high impact as it clearly demonstrates the various complexities of
quantizing the language translation model. We present novel quantization
techniques directly in TensorFlow to opportunistically replace 32-bit floating
point (FP32) computations with 8-bit integers (INT8) and transform the FP32
computational graph. We also present a bin-packing parallel batching technique
to maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver
1.5X improvement over the best FP32 performance. Furthermore, it reveals the
opportunities and challenges to boost performance of quantized deep learning
inference and establishes best practices to run inference with high efficiency
on Intel CPUs.

7
The categories of this paper are: hep-th
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
27
The categories of this paper are: comp-gas
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th
29
The categories of this paper are: funct-an
2
The categories of this paper are: math
15
The categories of this paper are: nucl-ex
17
The categories of this paper are: q-fin
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
11
The categories of this paper are: hep-lat