Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Probing Biomedical Embeddings from Language Models   Contextualized word embeddings derived from pre-trained language models (LMs)
show significant improvements on downstream NLP tasks. Pre-training on
domain-specific corpora, such as biomedical articles, further improves their
performance. In this paper, we conduct probing experiments to determine what
additional information is carried intrinsically by the in-domain trained
contextualized embeddings. For this we use the pre-trained LMs as fixed feature
extractors and restrict the downstream task models to not have additional
sequence modeling layers. We compare BERT, ELMo, BioBERT and BioELMo, a
biomedical version of ELMo trained on 10M PubMed abstracts. Surprisingly, while
fine-tuned BioBERT is better than BioELMo in biomedical NER and NLI tasks, as a
fixed feature extractor BioELMo outperforms BioBERT in our probing tasks. We
use visualization and nearest neighbor analysis to show that better encoding of
entity-type and relational information leads to this superiority.

23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
5
The categories of this paper are: gr-qc
34
The categories of this paper are: chem-ph
9
The categories of this paper are: nlin
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas
37
The categories of this paper are: supr-con
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
17
The categories of this paper are: q-fin
19
The categories of this paper are: eess
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
36
The categories of this paper are: cmp-lg
32
The categories of this paper are: atom-ph