Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Understanding the Extent to which Summarization Evaluation Metrics
  Measure the Information Quality of Summaries   Reference-based metrics such as ROUGE or BERTScore evaluate the content
quality of a summary by comparing the summary to a reference. Ideally, this
comparison should measure the summary's information quality by calculating how
much information the summaries have in common. In this work, we analyze the
token alignments used by ROUGE and BERTScore to compare summaries and argue
that their scores largely cannot be interpreted as measuring information
overlap, but rather the extent to which they discuss the same topics. Further,
we provide evidence that this result holds true for many other summarization
evaluation metrics. The consequence of this result is that it means the
summarization community has not yet found a reliable automatic metric that
aligns with its research goal, to generate summaries with high-quality
information. Then, we propose a simple and interpretable method of evaluating
summaries which does directly measure information overlap and demonstrate how
it can be used to gain insights into model behavior that could not be provided
by other methods alone.

36
The categories of this paper are: cmp-lg
14
The categories of this paper are: math-ph
17
The categories of this paper are: q-fin
6
The categories of this paper are: astro-ph
22
The categories of this paper are: chao-dyn
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
8
The categories of this paper are: hep-ex
16
The categories of this paper are: stat
31
The categories of this paper are: ao-sci
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
30
The categories of this paper are: q-alg
35
The categories of this paper are: mtrl-th
9
The categories of this paper are: nlin
23
The categories of this paper are: solv-int