Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Efficient Online Scalar Annotation with Bounded Support   We describe a novel method for efficiently eliciting scalar annotations for
dataset construction and system quality estimation by human judgments. We
contrast direct assessment (annotators assign scores to items directly), online
pairwise ranking aggregation (scores derive from annotator comparison of
items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels)
proposed here. Our proposal leads to increased correlation with ground truth,
at far greater annotator efficiency, suggesting this strategy as an improved
mechanism for dataset creation and manual system evaluation.

23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
3
The categories of this paper are: physics
8
The categories of this paper are: hep-ex
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
14
The categories of this paper are: math-ph
37
The categories of this paper are: supr-con
18
The categories of this paper are: econ
32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an