Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT   Transformer based architectures have become de-facto models used for a range
of Natural Language Processing tasks. In particular, the BERT based models
achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However,
BERT based models have a prohibitive memory footprint and latency. As a result,
deploying BERT based models in resource constrained environments has become a
challenging task. In this work, we perform an extensive analysis of fine-tuned
BERT models using second order Hessian information, and we use our results to
propose a novel method for quantizing BERT models to ultra low precision. In
particular, we propose a new group-wise quantization scheme, and we use a
Hessian based mix-precision method to compress the model further. We
extensively test our proposed method on BERT downstream tasks of SST-2, MNLI,
CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at
most $2.3\%$ performance degradation, even with ultra-low precision
quantization down to 2 bits, corresponding up to $13\times$ compression of the
model parameters, and up to $4\times$ compression of the embedding table as
well as activations. Among all tasks, we observed the highest performance loss
for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as
well as visualization, we show that this is related to the fact that current
training/fine-tuning strategy of BERT does not converge for SQuAD.

29
The categories of this paper are: funct-an
5
The categories of this paper are: gr-qc
33
The categories of this paper are: plasm-ph
8
The categories of this paper are: hep-ex
15
The categories of this paper are: nucl-ex
3
The categories of this paper are: physics
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
10
The categories of this paper are: q-bio
31
The categories of this paper are: ao-sci
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
32
The categories of this paper are: atom-ph
9
The categories of this paper are: nlin
21
The categories of this paper are: adap-org
6
The categories of this paper are: astro-ph