Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Measuring and Improving Consistency in Pretrained Language Models   Consistency of a model -- that is, the invariance of its behavior under
meaning-preserving alternations in its input -- is a highly desirable property
in natural language processing. In this paper we study the question: Are
Pretrained Language Models (PLMs) consistent with respect to factual knowledge?
To this end, we create ParaRel, a high-quality resource of cloze-style query
English paraphrases. It contains a total of 328 paraphrases for 38 relations.
Using ParaRel, we show that the consistency of all PLMs we experiment with is
poor -- though with high variance between relations. Our analysis of the
representational spaces of PLMs suggests that they have a poor structure and
are currently not suitable for representing knowledge robustly. Finally, we
propose a method for improving model consistency and experimentally demonstrate
its effectiveness.

18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
22
The categories of this paper are: chao-dyn
12
The categories of this paper are: nucl-th
30
The categories of this paper are: q-alg
33
The categories of this paper are: plasm-ph
0
The categories of this paper are: hep-ph
23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
35
The categories of this paper are: mtrl-th
32
The categories of this paper are: atom-ph