🐛 [Metrics] ROUGE is non-deterministic
MEMBER
Hi, can you give a full self-contained example to reproduce this behavior?
2020-07-09T08:26:02Z
https://github.com/huggingface/datasets/issues/361#issuecomment-655986182
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY1NTk4NjE4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/655986182/reactions
2020-07-09T08:26:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655986182
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)
2020-07-09T08:29:30Z
https://github.com/huggingface/datasets/issues/361#issuecomment-655987936
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY1NTk4NzkzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/655987936/reactions
2020-07-09T08:29:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655987936
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.
2020-07-09T08:31:26Z
https://github.com/huggingface/datasets/issues/361#issuecomment-655988938
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY1NTk4ODkzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/655988938/reactions
2022-09-09T15:20:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655988938
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?
2020-07-09T08:33:40Z
https://github.com/huggingface/datasets/issues/361#issuecomment-655990243
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY1NTk5MDI0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/655990243/reactions
2020-07-09T08:33:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655990243
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
We should probably indeed!
2020-07-09T09:07:50Z
https://github.com/huggingface/datasets/issues/361#issuecomment-656008832
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY1NjAwODgzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/656008832/reactions
2020-07-09T09:07:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656008832
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```
2020-07-20T10:18:16Z
https://github.com/huggingface/datasets/issues/361#issuecomment-660940024
https://api.github.com/repos/huggingface/datasets/issues/361
MDEyOklzc3VlQ29tbWVudDY2MDk0MDAyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/660940024/reactions
2020-07-20T10:19:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660940024
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> 

Minor nit: Note that "aggregator" is misspelled in this command. Should be `use_aggregator=False`.  
2022-09-09T01:08:34Z
https://github.com/huggingface/datasets/issues/361#issuecomment-1241388251
https://api.github.com/repos/huggingface/datasets/issues/361
IC_kwDODunzps5J_hTb
https://api.github.com/repos/huggingface/datasets/issues/comments/1241388251/reactions
2022-09-09T01:08:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1241388251
https://avatars.githubusercontent.com/u/12689993?v=4
https://api.github.com/users/lxuechen/events{/privacy}
https://api.github.com/users/lxuechen/followers
https://api.github.com/users/lxuechen/following{/other_user}
https://api.github.com/users/lxuechen/gists{/gist_id}
https://github.com/lxuechen
lxuechen
MDQ6VXNlcjEyNjg5OTkz
https://api.github.com/users/lxuechen/orgs
https://api.github.com/users/lxuechen/received_events
https://api.github.com/users/lxuechen/repos
https://api.github.com/users/lxuechen/starred{/owner}{/repo}
https://api.github.com/users/lxuechen/subscriptions
User
https://api.github.com/users/lxuechen
public
MEMBER
Thanks, I fixed the code snippet
2022-09-09T15:20:55Z
https://github.com/huggingface/datasets/issues/361#issuecomment-1242115419
https://api.github.com/repos/huggingface/datasets/issues/361
IC_kwDODunzps5KCS1b
https://api.github.com/repos/huggingface/datasets/issues/comments/1242115419/reactions
2022-09-09T15:20:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1242115419
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?