Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Testing the limits of natural language models for predicting human
  language judgments   Neural network language models can serve as computational hypotheses about
how humans process language. We compared the model-human consistency of diverse
language models using a novel experimental approach: controversial sentence
pairs. For each controversial sentence pair, two language models disagree about
which sentence is more likely to occur in natural text. Considering nine
language models (including n-gram, recurrent neural networks, and transformer
models), we created hundreds of such controversial sentence pairs by either
selecting sentences from a corpus or synthetically optimizing sentence pairs to
be highly controversial. Human subjects then provided judgments indicating for
each pair which of the two sentences is more likely. Controversial sentence
pairs proved highly effective at revealing model failures and identifying
models that aligned most closely with human judgments. The most
human-consistent model tested was GPT-2, although experiments also revealed
significant shortcomings of its alignment with human perception.

33
The categories of this paper are: plasm-ph
21
The categories of this paper are: adap-org
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
18
The categories of this paper are: econ
31
The categories of this paper are: ao-sci
36
The categories of this paper are: cmp-lg
34
The categories of this paper are: chem-ph
29
The categories of this paper are: funct-an
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
26
The categories of this paper are: bayes-an