Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: TernaryBERT: Distillation-aware Ultra-low Bit BERT   Transformer-based pre-training models like BERT have achieved remarkable
performance in many natural language processing tasks.However, these models are
both computation and memory expensive, hindering their deployment to
resource-constrained devices. In this work, we propose TernaryBERT, which
ternarizes the weights in a fine-tuned BERT model. Specifically, we use both
approximation-based and loss-aware ternarization methods and empirically
investigate the ternarization granularity of different parts of BERT. Moreover,
to reduce the accuracy degradation caused by the lower capacity of low bits, we
leverage the knowledge distillation technique in the training process.
Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT
outperforms the other BERT quantization methods, and even achieves comparable
performance as the full-precision model while being 14.9x smaller.

21
The categories of this paper are: adap-org
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin
15
The categories of this paper are: nucl-ex
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
6
The categories of this paper are: astro-ph
5
The categories of this paper are: gr-qc
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
3
The categories of this paper are: physics
23
The categories of this paper are: solv-int
37
The categories of this paper are: supr-con