Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Information-Theoretic Measures of Dataset Difficulty   Estimating the difficulty of a dataset typically involves comparing
state-of-the-art models to humans; the bigger the performance gap, the harder
the dataset is said to be. Not only is this framework informal, but it also
provides little understanding of how difficult each instance is, or what
attributes make it difficult for a given model. To address these problems, we
propose an information-theoretic perspective, framing dataset difficulty as the
absence of $\textit{usable information}$. Measuring usable information is as
easy as measuring performance, but has certain theoretical advantages. While
the latter only allows us to compare different models w.r.t the same dataset,
the former also allows us to compare different datasets w.r.t the same model.
We then introduce $\textit{pointwise}$ $\mathcal{V}-$$\textit{information}$
(PVI) for measuring the difficulty of individual instances, where instances
with higher PVI are easier for model $\mathcal{V}$. By manipulating the input
before measuring usable information, we can understand $\textit{why}$ a dataset
is easy or difficult for a given model, which we use to discover annotation
artefacts in widely-used benchmarks.

23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
12
The categories of this paper are: nucl-th
0
The categories of this paper are: hep-ph
6
The categories of this paper are: astro-ph
13
The categories of this paper are: quant-ph
7
The categories of this paper are: hep-th
25
The categories of this paper are: patt-sol
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
8
The categories of this paper are: hep-ex
21
The categories of this paper are: adap-org
33
The categories of this paper are: plasm-ph
32
The categories of this paper are: atom-ph