Compare different Rouge implementations  
NONE
Updates - the differences between the following three
(1) https://github.com/bheinzerling/pyrouge (previously popular. The one I trust the most)
(2) https://github.com/google-research/google-research/tree/master/rouge
(3) https://github.com/pltrdy/files2rouge (used in fairseq)
can be explained by two things, stemming and handling multiple sentences.

Stemming: 
(1), (2): default is no stemming. (3): default is with stemming ==> No stemming is the correct default as you did [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L84)

Multiple sentences:
(1) `rougeL` splits text using `\n`
(2) `rougeL` ignores `\n`. 
(2) `rougeLsum` splits text using `\n`
(3) `rougeL` splits text using `.`

For (2), `rougeL` and `rougeLsum` are identical if the sequence doesn't contain `\n`. With `\n`, it is `rougeLsum` that matches (1) not `rougeL`. 

Overall, and as far as I understand, for your implementation here https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L65 to match the default, you only need to change `rougeL` [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L86) to `rougeLsum` to correctly compute metrics for text with newlines.

Tagging @sshleifer who might be interested.
2020-09-13T05:10:53Z
https://github.com/huggingface/datasets/issues/617#issuecomment-691615081
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDY5MTYxNTA4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/691615081/reactions
2020-09-13T05:11:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/691615081
https://avatars.githubusercontent.com/u/2287797?v=4
https://api.github.com/users/ibeltagy/events{/privacy}
https://api.github.com/users/ibeltagy/followers
https://api.github.com/users/ibeltagy/following{/other_user}
https://api.github.com/users/ibeltagy/gists{/gist_id}
https://github.com/ibeltagy
ibeltagy
MDQ6VXNlcjIyODc3OTc=
https://api.github.com/users/ibeltagy/orgs
https://api.github.com/users/ibeltagy/received_events
https://api.github.com/users/ibeltagy/repos
https://api.github.com/users/ibeltagy/starred{/owner}{/repo}
https://api.github.com/users/ibeltagy/subscriptions
User
https://api.github.com/users/ibeltagy
public
MEMBER
Thanks for the clarification !
We're adding Rouge Lsum in #701 
2020-10-02T09:51:56Z
https://github.com/huggingface/datasets/issues/617#issuecomment-702635651
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDcwMjYzNTY1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/702635651/reactions
2020-10-02T09:51:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/702635651
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
This is a real issue, sorry for missing the mention @ibeltagy

We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines. 

Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

#### Sidebar: Wouldn't Deterministic Be Better?

`rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.

I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.


2020-10-08T02:02:18Z
https://github.com/huggingface/datasets/issues/617#issuecomment-705285843
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDcwNTI4NTg0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/705285843/reactions
2020-10-08T02:04:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/705285843
https://avatars.githubusercontent.com/u/6045025?v=4
https://api.github.com/users/sshleifer/events{/privacy}
https://api.github.com/users/sshleifer/followers
https://api.github.com/users/sshleifer/following{/other_user}
https://api.github.com/users/sshleifer/gists{/gist_id}
https://github.com/sshleifer
sshleifer
MDQ6VXNlcjYwNDUwMjU=
https://api.github.com/users/sshleifer/orgs
https://api.github.com/users/sshleifer/received_events
https://api.github.com/users/sshleifer/repos
https://api.github.com/users/sshleifer/starred{/owner}{/repo}
https://api.github.com/users/sshleifer/subscriptions
User
https://api.github.com/users/sshleifer
public
MEMBER
> This is a real issue, sorry for missing the mention @ibeltagy
> 
> We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines.
> 
> Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

Thanks for the details, I didn't know about that. Maybe we should consider adding this processing step or at least mention it somewhere in the library or the documentation

> #### Sidebar: Wouldn't Deterministic Be Better?
> `rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.
> 
> I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.

I think the default `n_samples` of the aggregator is 1000. We could increase it or at least allow users to change it if they want more precise results.
2020-10-08T08:58:40Z
https://github.com/huggingface/datasets/issues/617#issuecomment-705431992
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDcwNTQzMTk5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/705431992/reactions
2020-10-08T08:58:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/705431992
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, thanks for the solution. 

I am not sure if this is a bug, but on line [510](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L510), are pred, tgt supposed to be swapped?
2021-03-31T15:38:35Z
https://github.com/huggingface/datasets/issues/617#issuecomment-811164403
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDgxMTE2NDQwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/811164403/reactions
2021-03-31T15:38:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811164403
https://avatars.githubusercontent.com/u/9755416?v=4
https://api.github.com/users/dptam/events{/privacy}
https://api.github.com/users/dptam/followers
https://api.github.com/users/dptam/following{/other_user}
https://api.github.com/users/dptam/gists{/gist_id}
https://github.com/dptam
dptam
MDQ6VXNlcjk3NTU0MTY=
https://api.github.com/users/dptam/orgs
https://api.github.com/users/dptam/received_events
https://api.github.com/users/dptam/repos
https://api.github.com/users/dptam/starred{/owner}{/repo}
https://api.github.com/users/dptam/subscriptions
User
https://api.github.com/users/dptam
public
MEMBER
This looks like a bug in an old version of the examples in `transformers`
2021-03-31T17:28:32Z
https://github.com/huggingface/datasets/issues/617#issuecomment-811273179
https://api.github.com/repos/huggingface/datasets/issues/617
MDEyOklzc3VlQ29tbWVudDgxMTI3MzE3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811273179/reactions
2021-03-31T17:28:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811273179
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, so I took this example from the HF implementation. What I can see is that  the precision of `Hello there` being summarized to `general kenobi` is 1. I don't understand how this calculation is correct.
Is the comparison just counting the words?
and if Yes, then how does this translates to summarization evaluation?
```
    >>> rouge = datasets.load_metric('rouge')
    >>> predictions = ["hello there", "general kenobi"]
    >>> references = ["hello there", "general kenobi"]
    >>> results = rouge.compute(predictions=predictions, references=references)
    >>> print(list(results.keys()))
    ['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
    >>> print(results["rouge1"])
    AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))
    >>> print(results["rouge1"].mid.fmeasure)
    1.0
""", stored examples: 0)
```



2023-03-22T12:08:44Z
https://github.com/huggingface/datasets/issues/617#issuecomment-1479463389
https://api.github.com/repos/huggingface/datasets/issues/617
IC_kwDODunzps5YLtHd
https://api.github.com/repos/huggingface/datasets/issues/comments/1479463389/reactions
2023-03-22T12:08:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479463389
https://avatars.githubusercontent.com/u/48519427?v=4
https://api.github.com/users/owos/events{/privacy}
https://api.github.com/users/owos/followers
https://api.github.com/users/owos/following{/other_user}
https://api.github.com/users/owos/gists{/gist_id}
https://github.com/owos
owos
MDQ6VXNlcjQ4NTE5NDI3
https://api.github.com/users/owos/orgs
https://api.github.com/users/owos/received_events
https://api.github.com/users/owos/repos
https://api.github.com/users/owos/starred{/owner}{/repo}
https://api.github.com/users/owos/subscriptions
User
https://api.github.com/users/owos
public
I used RougeL implementation provided in `datasets` [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py) and it gives numbers that match those reported in the pegasus paper but very different from those reported in other papers, [this](https://arxiv.org/pdf/1909.03186.pdf) for example.
Can you make sure the google-research implementation you are using matches the official perl implementation? 
There are a couple of python wrappers around the perl implementation, [this](https://pypi.org/project/pyrouge/) has been commonly used, and [this](https://github.com/pltrdy/files2rouge) is used in fairseq). 
There's also a python reimplementation [here](https://github.com/pltrdy/rouge) but its RougeL numbers are way off. 
