Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word
  Understanding of Language Models   Recent progress in pretraining language models on large corpora has resulted
in large performance gains on many NLP tasks. These large models acquire
linguistic knowledge during pretraining, which helps to improve performance on
downstream tasks via fine-tuning. To assess what kind of knowledge is acquired,
language models are commonly probed by querying them with `fill in the blank'
style cloze questions. Existing probing datasets mainly focus on knowledge
about relations between words and entities. We introduce WDLMPro (Word
Definition Language Model Probing) to evaluate word understanding directly
using dictionary definitions of words. In our experiments, three popular
pretrained language models struggle to match words and their definitions. This
indicates that they understand many words poorly and that our new probing task
is a difficult challenge that could help guide research on LMs in the future.

27
The categories of this paper are: comp-gas
12
The categories of this paper are: nucl-th
17
The categories of this paper are: q-fin
26
The categories of this paper are: bayes-an
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
32
The categories of this paper are: atom-ph
31
The categories of this paper are: ao-sci
36
The categories of this paper are: cmp-lg
29
The categories of this paper are: funct-an
2
The categories of this paper are: math
3
The categories of this paper are: physics
14
The categories of this paper are: math-ph
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom