Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: I-BERT: Integer-only BERT Quantization   Transformer based models, like BERT and RoBERTa, have achieved
state-of-the-art results in many Natural Language Processing tasks. However,
their memory footprint, inference latency, and power consumption are
prohibitive efficient inference at the edge, and even at the data center. While
quantization can be a viable solution for this, previous work on quantizing
Transformer based models use floating-point arithmetic during inference, which
cannot efficiently utilize integer-only logical units such as the recent Turing
Tensor Cores, or traditional integer-only ARM processors. In this work, we
propose I-BERT, a novel quantization scheme for Transformer based models that
quantizes the entire inference with integer-only arithmetic. Based on
lightweight integer-only approximation methods for nonlinear operations, e.g.,
GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end
integer-only BERT inference without any floating point calculation. We evaluate
our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that
for both cases, I-BERT achieves similar (and slightly higher) accuracy as
compared to the full-precision baseline. Furthermore, our preliminary
implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4
GPU system as compared to FP32 inference. The framework has been developed in
PyTorch and has been open-sourced.

36
The categories of this paper are: cmp-lg
31
The categories of this paper are: ao-sci
7
The categories of this paper are: hep-th
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
21
The categories of this paper are: adap-org
5
The categories of this paper are: gr-qc
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
11
The categories of this paper are: hep-lat
16
The categories of this paper are: stat
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio