BLEURT score calculation raises UnrecognizedFlagError
NONE
Upgrading tensorflow to version 2.4.0 solved the issue.
2021-01-13T00:12:05Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-759117368
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc1OTExNzM2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/759117368/reactions
2021-01-13T00:12:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/759117368
https://avatars.githubusercontent.com/u/6603920?v=4
https://api.github.com/users/nadavo/events{/privacy}
https://api.github.com/users/nadavo/followers
https://api.github.com/users/nadavo/following{/other_user}
https://api.github.com/users/nadavo/gists{/gist_id}
https://github.com/nadavo
nadavo
MDQ6VXNlcjY2MDM5MjA=
https://api.github.com/users/nadavo/orgs
https://api.github.com/users/nadavo/received_events
https://api.github.com/users/nadavo/repos
https://api.github.com/users/nadavo/starred{/owner}{/repo}
https://api.github.com/users/nadavo/subscriptions
User
https://api.github.com/users/nadavo
public
CONTRIBUTOR
I still have the same error even with TF 2.4.0.
2021-01-25T08:57:26Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-766660157
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc2NjY2MDE1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/766660157/reactions
2021-01-25T08:57:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766660157
https://avatars.githubusercontent.com/u/1479733?v=4
https://api.github.com/users/aleSuglia/events{/privacy}
https://api.github.com/users/aleSuglia/followers
https://api.github.com/users/aleSuglia/following{/other_user}
https://api.github.com/users/aleSuglia/gists{/gist_id}
https://github.com/aleSuglia
aleSuglia
MDQ6VXNlcjE0Nzk3MzM=
https://api.github.com/users/aleSuglia/orgs
https://api.github.com/users/aleSuglia/received_events
https://api.github.com/users/aleSuglia/repos
https://api.github.com/users/aleSuglia/starred{/owner}{/repo}
https://api.github.com/users/aleSuglia/subscriptions
User
https://api.github.com/users/aleSuglia
public
CONTRIBUTOR
And I have the same error with TF 2.4.1. I believe this issue should be reopened. Any ideas?!
2021-01-25T10:27:27Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-766715622
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc2NjcxNTYyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/766715622/reactions
2021-01-25T10:27:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766715622
https://avatars.githubusercontent.com/u/1479733?v=4
https://api.github.com/users/aleSuglia/events{/privacy}
https://api.github.com/users/aleSuglia/followers
https://api.github.com/users/aleSuglia/following{/other_user}
https://api.github.com/users/aleSuglia/gists{/gist_id}
https://github.com/aleSuglia
aleSuglia
MDQ6VXNlcjE0Nzk3MzM=
https://api.github.com/users/aleSuglia/orgs
https://api.github.com/users/aleSuglia/received_events
https://api.github.com/users/aleSuglia/repos
https://api.github.com/users/aleSuglia/starred{/owner}{/repo}
https://api.github.com/users/aleSuglia/subscriptions
User
https://api.github.com/users/aleSuglia
public
NONE
I'm seeing the same issue with TF 2.4.1 when running the following in https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb:
```
!pip install git+https://github.com/google-research/bleurt.git
references = ["foo bar baz", "one two three"]
bleurt_metric = load_metric('bleurt')
predictions =  ["foo bar", "four five six"]
bleurt_metric.compute(predictions=predictions, references=references)
```
2021-03-10T14:53:11Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-795543931
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc5NTU0MzkzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/795543931/reactions
2021-03-10T14:53:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/795543931
https://avatars.githubusercontent.com/u/1137755?v=4
https://api.github.com/users/oscartackstrom/events{/privacy}
https://api.github.com/users/oscartackstrom/followers
https://api.github.com/users/oscartackstrom/following{/other_user}
https://api.github.com/users/oscartackstrom/gists{/gist_id}
https://github.com/oscartackstrom
oscartackstrom
MDQ6VXNlcjExMzc3NTU=
https://api.github.com/users/oscartackstrom/orgs
https://api.github.com/users/oscartackstrom/received_events
https://api.github.com/users/oscartackstrom/repos
https://api.github.com/users/oscartackstrom/starred{/owner}{/repo}
https://api.github.com/users/oscartackstrom/subscriptions
User
https://api.github.com/users/oscartackstrom
public
NONE
@aleSuglia @oscartackstrom - Are you getting the error when running your code in a Jupyter notebook ?

I tried reproducing this error again, and was unable to do so from the python command line console in a virtual environment similar to the one I originally used (and unfortunately no longer have access to) when I first got the error. 
However, I've managed to reproduce the error by running the same code in a Jupyter notebook running a kernel from the same virtual environment.
This made me suspect that the problem is somehow related to the Jupyter notebook.

More environment details:
```
OS: Ubuntu Linux 18.04
conda==4.8.3
python==3.8.5
datasets==1.3.0
tensorflow==2.4.0
BLEURT==0.0.1
notebook==6.2.0
```
2021-03-12T01:24:37Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-797167896
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc5NzE2Nzg5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/797167896/reactions
2021-03-12T01:26:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797167896
https://avatars.githubusercontent.com/u/6603920?v=4
https://api.github.com/users/nadavo/events{/privacy}
https://api.github.com/users/nadavo/followers
https://api.github.com/users/nadavo/following{/other_user}
https://api.github.com/users/nadavo/gists{/gist_id}
https://github.com/nadavo
nadavo
MDQ6VXNlcjY2MDM5MjA=
https://api.github.com/users/nadavo/orgs
https://api.github.com/users/nadavo/received_events
https://api.github.com/users/nadavo/repos
https://api.github.com/users/nadavo/starred{/owner}{/repo}
https://api.github.com/users/nadavo/subscriptions
User
https://api.github.com/users/nadavo
public
NONE
This happens when running the notebook on colab. The issue seems to be that colab populates sys.argv with arguments not handled by bleurt.

Running this before calling bleurt fixes it:
```
import sys
sys.argv = sys.argv[:1]
```

Not the most elegant solution. Perhaps it needs to be fixed in the bleurt code itself rather than huggingface?

This is the output of `print(sys.argv)` when running on colab:
```
['/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py', '-f', '/root/.local/share/jupyter/runtime/kernel-a857a78c-44d6-4b9d-b18a-030b858ee327.json']
```
2021-03-12T10:12:35Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-797385993
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc5NzM4NTk5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/797385993/reactions
2021-03-12T10:16:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797385993
https://avatars.githubusercontent.com/u/1137755?v=4
https://api.github.com/users/oscartackstrom/events{/privacy}
https://api.github.com/users/oscartackstrom/followers
https://api.github.com/users/oscartackstrom/following{/other_user}
https://api.github.com/users/oscartackstrom/gists{/gist_id}
https://github.com/oscartackstrom
oscartackstrom
MDQ6VXNlcjExMzc3NTU=
https://api.github.com/users/oscartackstrom/orgs
https://api.github.com/users/oscartackstrom/received_events
https://api.github.com/users/oscartackstrom/repos
https://api.github.com/users/oscartackstrom/starred{/owner}{/repo}
https://api.github.com/users/oscartackstrom/subscriptions
User
https://api.github.com/users/oscartackstrom
public
CONTRIBUTOR
I got the error when running it from the command line. It looks more like an error that should be fixed in the BLEURT codebase.
2021-03-12T10:14:43Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-797387311
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc5NzM4NzMxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/797387311/reactions
2021-03-12T10:14:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797387311
https://avatars.githubusercontent.com/u/1479733?v=4
https://api.github.com/users/aleSuglia/events{/privacy}
https://api.github.com/users/aleSuglia/followers
https://api.github.com/users/aleSuglia/following{/other_user}
https://api.github.com/users/aleSuglia/gists{/gist_id}
https://github.com/aleSuglia
aleSuglia
MDQ6VXNlcjE0Nzk3MzM=
https://api.github.com/users/aleSuglia/orgs
https://api.github.com/users/aleSuglia/received_events
https://api.github.com/users/aleSuglia/repos
https://api.github.com/users/aleSuglia/starred{/owner}{/repo}
https://api.github.com/users/aleSuglia/subscriptions
User
https://api.github.com/users/aleSuglia
public
NONE
Seems to be a known issue in the bleurt codebase: https://github.com/google-research/bleurt/issues/24.
2021-03-12T10:17:54Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-797389090
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDc5NzM4OTA5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/797389090/reactions
2021-03-12T10:22:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797389090
https://avatars.githubusercontent.com/u/1137755?v=4
https://api.github.com/users/oscartackstrom/events{/privacy}
https://api.github.com/users/oscartackstrom/followers
https://api.github.com/users/oscartackstrom/following{/other_user}
https://api.github.com/users/oscartackstrom/gists{/gist_id}
https://github.com/oscartackstrom
oscartackstrom
MDQ6VXNlcjExMzc3NTU=
https://api.github.com/users/oscartackstrom/orgs
https://api.github.com/users/oscartackstrom/received_events
https://api.github.com/users/oscartackstrom/repos
https://api.github.com/users/oscartackstrom/starred{/owner}{/repo}
https://api.github.com/users/oscartackstrom/subscriptions
User
https://api.github.com/users/oscartackstrom
public
NONE
Hi, the problem should be solved now.
2021-04-12T22:21:41Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-818281307
https://api.github.com/repos/huggingface/datasets/issues/1727
MDEyOklzc3VlQ29tbWVudDgxODI4MTMwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/818281307/reactions
2021-04-12T22:21:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818281307
https://avatars.githubusercontent.com/u/6736052?v=4
https://api.github.com/users/tsellam/events{/privacy}
https://api.github.com/users/tsellam/followers
https://api.github.com/users/tsellam/following{/other_user}
https://api.github.com/users/tsellam/gists{/gist_id}
https://github.com/tsellam
tsellam
MDQ6VXNlcjY3MzYwNTI=
https://api.github.com/users/tsellam/orgs
https://api.github.com/users/tsellam/received_events
https://api.github.com/users/tsellam/repos
https://api.github.com/users/tsellam/starred{/owner}{/repo}
https://api.github.com/users/tsellam/subscriptions
User
https://api.github.com/users/tsellam
public
COLLABORATOR
Hi @tsellam! I can verify that the issue is indeed fixed now. Thanks!
2022-06-01T16:05:59Z
https://github.com/huggingface/datasets/issues/1727#issuecomment-1143801691
https://api.github.com/repos/huggingface/datasets/issues/1727
IC_kwDODunzps5ELQdb
https://api.github.com/repos/huggingface/datasets/issues/comments/1143801691/reactions
2022-06-01T16:05:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1143801691
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Calling the `compute` method for **bleurt** metric fails with an `UnrecognizedFlagError` for `FLAGS.bleurt_batch_size`. 

My environment:
```
python==3.8.5
datasets==1.2.0
tensorflow==2.3.1
cudatoolkit==11.0.221
```

Test code for reproducing the error:
```
from datasets import load_metric
bleurt = load_metric('bleurt')
gen_text = "I am walking on the promenade today"
ref_text = "I am walking along the promenade on this sunny day"
bleurt.compute(predictions=[test_text], references=[test_text])
```

Error Output:
```
Using default BLEURT-Base checkpoint for sequence maximum length 128. You can use a bigger model for better results with e.g.: datasets.load_metric('bleurt', 'bleurt-large-512').
INFO:tensorflow:Reading checkpoint /home/ubuntu/.cache/huggingface/metrics/bleurt/default/downloads/extracted/9aee35580225730ac5422599f35c4986e4c49cafd08082123342b1019720dac4/bleurt-base-128.
INFO:tensorflow:Config file found, reading.
INFO:tensorflow:Will load checkpoint bert_custom
INFO:tensorflow:Performs basic checks...
INFO:tensorflow:... name:bert_custom
INFO:tensorflow:... vocab_file:vocab.txt
INFO:tensorflow:... bert_config_file:bert_config.json
INFO:tensorflow:... do_lower_case:True
INFO:tensorflow:... max_seq_length:128
INFO:tensorflow:Creating BLEURT scorer.
INFO:tensorflow:Loading model...
INFO:tensorflow:BLEURT initialized.
---------------------------------------------------------------------------
UnrecognizedFlagError                     Traceback (most recent call last)
<ipython-input-12-8b3f4322318a> in <module>
      2 gen_text = "I am walking on the promenade today"
      3 ref_text = "I am walking along the promenade on this sunny day"
----> 4 bleurt.compute(predictions=[gen_text], references=[ref_text])

~/anaconda3/envs/noved/lib/python3.8/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399 
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/bleurt/b1de33e1cbbcb1dbe276c887efa1fad68c6aff913885108078fa1ad408908778/bleurt.py in _compute(self, predictions, references)
    103 
    104     def _compute(self, predictions, references):
--> 105         scores = self.scorer.score(references=references, candidates=predictions)
    106         return {"scores": scores}

~/anaconda3/envs/noved/lib/python3.8/site-packages/bleurt/score.py in score(self, references, candidates, batch_size)
    164     """
    165     if not batch_size:
--> 166       batch_size = FLAGS.bleurt_batch_size
    167 
    168     candidates, references = list(candidates), list(references)

~/anaconda3/envs/noved/lib/python3.8/site-packages/tensorflow/python/platform/flags.py in __getattr__(self, name)
     83     # a flag.
     84     if not wrapped.is_parsed():
---> 85       wrapped(_sys.argv)
     86     return wrapped.__getattr__(name)
     87 

~/anaconda3/envs/noved/lib/python3.8/site-packages/absl/flags/_flagvalues.py in __call__(self, argv, known_only)
    643     for name, value in unknown_flags:
    644       suggestions = _helpers.get_flag_suggestions(name, list(self))
--> 645       raise _exceptions.UnrecognizedFlagError(
    646           name, value, suggestions=suggestions)
    647 

UnrecognizedFlagError: Unknown command line flag 'f'
```

Possible Fix:
Modify `_compute` method https://github.com/huggingface/datasets/blob/7e64851a12263dc74d41c668167918484c8000ab/metrics/bleurt/bleurt.py#L104
to receive a `batch_size` argument, for example:
```
def _compute(self, predictions, references, batch_size=1):
    scores = self.scorer.score(references=references, candidates=predictions, batch_size=batch_size)
    return {"scores": scores}
```