Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Dynamic Human Evaluation for Relative Model Comparisons   Collecting human judgements is currently the most reliable evaluation method
for natural language generation systems. Automatic metrics have reported flaws
when applied to measure quality aspects of generated text and have been shown
to correlate poorly with human judgements. However, human evaluation is time
and cost-intensive, and we lack consensus on designing and conducting human
evaluation experiments. Thus there is a need for streamlined approaches for
efficient collection of human judgements when evaluating natural language
generation systems. Therefore, we present a dynamic approach to measure the
required number of human annotations when evaluating generated outputs in
relative comparison settings. We propose an agent-based framework of human
evaluation to assess multiple labelling strategies and methods to decide the
better model in a simulation and a crowdsourcing case study. The main results
indicate that a decision about the superior model can be made with high
probability across different labelling strategies, where assigning a single
random worker per task requires the least overall labelling effort and thus the
least cost.

24
The categories of this paper are: dg-ga
33
The categories of this paper are: plasm-ph
29
The categories of this paper are: funct-an
22
The categories of this paper are: chao-dyn
28
The categories of this paper are: alg-geom
11
The categories of this paper are: hep-lat
8
The categories of this paper are: hep-ex
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
7
The categories of this paper are: hep-th
34
The categories of this paper are: chem-ph
31
The categories of this paper are: ao-sci
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin