[ROUGE] Different scores with `files2rouge`
MEMBER
@Colanim unfortunately there are different implementations of the ROUGE metric floating around online which yield different results, and we had to chose one for the package :) We ended up including the one from the google-research repository, which does minimal post-processing before computing the P/R/F scores. If I recall correctly, files2rouge relies on the Perl, script, which among other things normalizes all numbers to a special token: in the case you presented, this should account for a good chunk of the difference.

We may end up adding in more versions of the metric, but probably not for a while (@lhoestq correct me if I'm wrong). However, feel free to take a stab at adding it in yourself and submitting a PR if you're interested!
2020-06-01T13:45:37Z
https://github.com/huggingface/datasets/issues/225#issuecomment-636869718
https://api.github.com/repos/huggingface/datasets/issues/225
MDEyOklzc3VlQ29tbWVudDYzNjg2OTcxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/636869718/reactions
2020-06-01T13:45:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/636869718
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
NONE
Thank you for your kind answer.

As a side question : Isn't it better to have a package that normalize more ?

I understand to idea of having a package that does minimal post-processing for transparency.

But it means that people using different architecture (with different tokenizers for example) will have difference in ROUGE scores even if their predictions are actually similar.  
The goal of `nlp` is to have _one package to rule them all_, right ?

I will look into it but I'm not sure I have the required skill for this ^^ 
2020-06-01T23:41:29Z
https://github.com/huggingface/datasets/issues/225#issuecomment-637182938
https://api.github.com/repos/huggingface/datasets/issues/225
MDEyOklzc3VlQ29tbWVudDYzNzE4MjkzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/637182938/reactions
2020-06-01T23:42:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/637182938
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
MEMBER
You're right, there's a pretty interesting trade-off here between robustness and sensitivity :) The flip side of your argument is that we also still want the metric to be sensitive to model mistakes. How we think about number normalization for example has evolved a fair bit since the Perl script was written: at the time, ROUGE was used mostly to evaluate short-medium text summarization systems, where there were only a few numbers in the input and it was assumed that the most popular methods in use at the time would get those right. However, as your example showcases, that assumption does not hold any more, and we do want to be able to penalize a model that generates a wrong numerical value.

Also, we think that abstracting away tokenization differences is the role of the model/tokenizer: if you use the 🤗Tokenizers library for example, it will handle that for you ;)

Finally, there is a lot of active research on developing model-powered metrics that are both more sensitive and more robust than ROUGE. Check out for example BERTscore, which is implemented in this library!
2020-06-02T14:09:38Z
https://github.com/huggingface/datasets/issues/225#issuecomment-637568151
https://api.github.com/repos/huggingface/datasets/issues/225
MDEyOklzc3VlQ29tbWVudDYzNzU2ODE1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/637568151/reactions
2020-06-02T18:53:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/637568151
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
It seems that the ROUGE score of `nlp` is lower than the one of `files2rouge`.

Here is a self-contained notebook to reproduce both scores : https://colab.research.google.com/drive/14EyAXValB6UzKY9x4rs_T3pyL7alpw_F?usp=sharing

---

`nlp` : (Only mid F-scores)

>rouge1 0.33508031962733364
rouge2 0.14574333776191592
rougeL 0.2321187823256159

`files2rouge` :

>Running ROUGE...
===========================
1 ROUGE-1 Average_R: 0.48873 (95%-conf.int. 0.41192 - 0.56339)
1 ROUGE-1 Average_P: 0.29010 (95%-conf.int. 0.23605 - 0.34445)
1 ROUGE-1 Average_F: 0.34761 (95%-conf.int. 0.29479 - 0.39871)
===========================
1 ROUGE-2 Average_R: 0.20280 (95%-conf.int. 0.14969 - 0.26244)
1 ROUGE-2 Average_P: 0.12772 (95%-conf.int. 0.08603 - 0.17752)
1 ROUGE-2 Average_F: 0.14798 (95%-conf.int. 0.10517 - 0.19240)
===========================
1 ROUGE-L Average_R: 0.32960 (95%-conf.int. 0.26501 - 0.39676)
1 ROUGE-L Average_P: 0.19880 (95%-conf.int. 0.15257 - 0.25136)
1 ROUGE-L Average_F: 0.23619 (95%-conf.int. 0.19073 - 0.28663)

---

When using longer predictions/gold, the difference is bigger.  
**How can I reproduce same score as `files2rouge` ?**

@lhoestq 
