Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Accurate, yet inconsistent? Consistency Analysis on Language
  Understanding Models   Consistency, which refers to the capability of generating the same
predictions for semantically similar contexts, is a highly desirable property
for a sound language understanding model. Although recent pretrained language
models (PLMs) deliver outstanding performance in various downstream tasks, they
should exhibit consistent behaviour provided the models truly understand
language. In this paper, we propose a simple framework named consistency
analysis on language understanding models (CALUM)} to evaluate the model's
lower-bound consistency ability. Through experiments, we confirmed that current
PLMs are prone to generate inconsistent predictions even for semantically
identical inputs. We also observed that multi-task training with paraphrase
identification tasks is of benefit to improve consistency, increasing the
consistency by 13% on average.

9
The categories of this paper are: nlin
18
The categories of this paper are: econ
3
The categories of this paper are: physics
2
The categories of this paper are: math
12
The categories of this paper are: nucl-th
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
0
The categories of this paper are: hep-ph
15
The categories of this paper are: nucl-ex
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
37
The categories of this paper are: supr-con
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
24
The categories of this paper are: dg-ga