Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Extremely Low Bit Transformer Quantization for On-Device Neural Machine
  Translation   The deployment of widely used Transformer architecture is challenging because
of heavy computation load and memory overhead during inference, especially when
the target device is limited in computational resources such as mobile or edge
devices. Quantization is an effective technique to address such challenges. Our
analysis shows that for a given number of quantization bits, each block of
Transformer contributes to translation quality and inference computations in
different manners. Moreover, even inside an embedding block, each word presents
vastly different contributions. Correspondingly, we propose a mixed precision
quantization strategy to represent Transformer weights by an extremely low
number of bits (e.g., under 3 bits). For example, for each word in an embedding
block, we assign different quantization bits based on statistical property. Our
quantized Transformer model achieves 11.8$\times$ smaller model size than the
baseline model, with less than -0.5 BLEU. We achieve 8.3$\times$ reduction in
run-time memory footprints and 3.5$\times$ speed up (Galaxy N10+) such that our
proposed compression strategy enables efficient implementation for on-device
NMT.

34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
11
The categories of this paper are: hep-lat
14
The categories of this paper are: math-ph
6
The categories of this paper are: astro-ph
10
The categories of this paper are: q-bio
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
18
The categories of this paper are: econ
37
The categories of this paper are: supr-con
8
The categories of this paper are: hep-ex
22
The categories of this paper are: chao-dyn
23
The categories of this paper are: solv-int