Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Universal Text Representation from BERT: An Empirical Study   We present a systematic investigation of layer-wise BERT activations for
general-purpose text representations to understand what linguistic information
they capture and how transferable they are across different tasks.
Sentence-level embeddings are evaluated against two state-of-the-art models on
downstream and probing tasks from SentEval, while passage-level embeddings are
evaluated on four question-answering (QA) datasets under a learning-to-rank
problem setting. Embeddings from the pre-trained BERT model perform poorly in
semantic similarity and sentence surface information probing tasks. Fine-tuning
BERT on natural language inference data greatly improves the quality of the
embeddings. Combining embeddings from different BERT layers can further boost
performance. BERT embeddings outperform BM25 baseline significantly on factoid
QA datasets at the passage level, but fail to perform better than BM25 on
non-factoid datasets. For all QA datasets, there is a gap between
embedding-based method and in-domain fine-tuned BERT (we report new
state-of-the-art results on two datasets), which suggests deep interactions
between question and answer pairs are critical for those hard tasks.

7
The categories of this paper are: hep-th
6
The categories of this paper are: astro-ph
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
37
The categories of this paper are: supr-con
3
The categories of this paper are: physics
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
25
The categories of this paper are: patt-sol