Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Reproducibility Issues for BERT-based Evaluation Metrics   Reproducibility is of utmost concern in machine learning and natural language
processing (NLP). In the field of natural language generation (especially
machine translation), the seminal paper of Post (2018) has pointed out problems
of reproducibility of the dominant metric, BLEU, at the time of publication.
Nowadays, BERT-based evaluation metrics considerably outperform BLEU. In this
paper, we ask whether results and claims from four recent BERT-based metrics
can be reproduced. We find that reproduction of claims and results often fails
because of (i) heavy undocumented preprocessing involved in the metrics, (ii)
missing code and (iii) reporting weaker results for the baseline metrics. (iv)
In one case, the problem stems from correlating not to human scores but to a
wrong column in the csv file, inflating scores by 5 points. Motivated by the
impact of preprocessing, we then conduct a second study where we examine its
effects more closely (for one of the metrics). We find that preprocessing can
have large effects, especially for highly inflectional languages. In this case,
the effect of preprocessing may be larger than the effect of the aggregation
mechanism (e.g., greedy alignment vs. Word Mover Distance).

4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph
19
The categories of this paper are: eess
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
22
The categories of this paper are: chao-dyn
26
The categories of this paper are: bayes-an
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
21
The categories of this paper are: adap-org
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
10
The categories of this paper are: q-bio
16
The categories of this paper are: stat