Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Compressing Sentence Representation for Semantic Retrieval via
  Homomorphic Projective Distillation   How to learn highly compact yet effective sentence representation?
Pre-trained language models have been effective in many NLP tasks. However,
these models are often huge and produce large sentence embeddings. Moreover,
there is a big performance gap between large and small models. In this paper,
we propose Homomorphic Projective Distillation (HPD) to learn compressed
sentence embeddings. Our method augments a small Transformer encoder model with
learnable projection layers to produce compact representations while mimicking
a large pre-trained language model to retain the sentence representation
quality. We evaluate our method with different model sizes on both semantic
textual similarity (STS) and semantic retrieval (SR) tasks. Experiments show
that our method achieves 2.7-4.5 points performance gain on STS tasks compared
with previous best representations of the same size. In SR tasks, our method
improves retrieval speed (8.2$\times$) and memory usage (8.0$\times$) compared
with state-of-the-art large models.

31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess
8
The categories of this paper are: hep-ex
7
The categories of this paper are: hep-th
4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
24
The categories of this paper are: dg-ga
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
35
The categories of this paper are: mtrl-th
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th
13
The categories of this paper are: quant-ph
33
The categories of this paper are: plasm-ph
16
The categories of this paper are: stat