Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy
  Evaluation Approach   Reliable automatic evaluation of dialogue systems under an interactive
environment has long been overdue. An ideal environment for evaluating dialog
systems, also known as the Turing test, needs to involve human interaction,
which is usually not affordable for large-scale experiments. Though researchers
have attempted to use metrics (e.g., perplexity, BLEU) in language generation
tasks or some model-based reinforcement learning methods (e.g., self-play
evaluation) for automatic evaluation, these methods only show a very weak
correlation with the actual human evaluation in practice. To bridge such a gap,
we propose a new framework named ENIGMA for estimating human evaluation scores
based on recent advances of off-policy evaluation in reinforcement learning.
ENIGMA only requires a handful of pre-collected experience data, and therefore
does not involve human interaction with the target policy during the
evaluation, making automatic evaluations feasible. More importantly, ENIGMA is
model-free and agnostic to the behavior policies for collecting the experience
data (see details in Section 2), which significantly alleviates the technical
difficulties of modeling complex dialogue environments and human behaviors. Our
experiments show that ENIGMA significantly outperforms existing methods in
terms of correlation with human evaluation scores.

26
The categories of this paper are: bayes-an
16
The categories of this paper are: stat
3
The categories of this paper are: physics
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
35
The categories of this paper are: mtrl-th
9
The categories of this paper are: nlin
23
The categories of this paper are: solv-int
37
The categories of this paper are: supr-con
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
15
The categories of this paper are: nucl-ex
7
The categories of this paper are: hep-th