Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for
  Machine Translation   Automatic metrics are commonly used as the exclusive tool for declaring the
superiority of one machine translation system's quality over another. The
community choice of automatic metric guides research directions and industrial
developments by deciding which models are deemed better. Evaluating metrics
correlations with sets of human judgements has been limited by the size of
these sets. In this paper, we corroborate how reliable metrics are in contrast
to human judgements on -- to the best of our knowledge -- the largest
collection of judgements reported in the literature. Arguably, pairwise
rankings of two systems are the most common evaluation tasks in research or
deployment scenarios. Taking human judgement as a gold standard, we investigate
which metrics have the highest accuracy in predicting translation quality
rankings for such system pairs. Furthermore, we evaluate the performance of
various metrics across different language pairs and domains. Lastly, we show
that the sole use of BLEU impeded the development of improved models leading to
bad deployment decisions. We release the collection of 2.3M sentence-level
human judgements for 4380 systems for further analysis and replication of our
work.

36
The categories of this paper are: cmp-lg
35
The categories of this paper are: mtrl-th
31
The categories of this paper are: ao-sci
23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
10
The categories of this paper are: q-bio
8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess
14
The categories of this paper are: math-ph
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
4
The categories of this paper are: cond-mat
2
The categories of this paper are: math
26
The categories of this paper are: bayes-an