[WER] Better error message for wer
MEMBER
Hi ! I don't think this would solve this issue.
Currently it looks like there's a bug that converts the list `["hello it's nice"]` to a string `'["hello it's nice"]'` since this is what the metric expects as input. The conversion is done before the data are passed to `_compute()`.

This is `Value("string").encode_example` that is called to do the conversion. Since `str()` encoding is too permissive we should consider raising an error if the example is not a string (even though it can be converted to string). 
2021-12-20T16:33:53Z
https://github.com/huggingface/datasets/pull/3456#issuecomment-998086685
https://api.github.com/repos/huggingface/datasets/issues/3456
IC_kwDODunzps47fZgd
https://api.github.com/repos/huggingface/datasets/issues/comments/998086685/reactions
2021-12-20T16:33:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998086685
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> called

Ah yeah you're right
2021-12-20T16:40:36Z
https://github.com/huggingface/datasets/pull/3456#issuecomment-998092114
https://api.github.com/repos/huggingface/datasets/issues/3456
IC_kwDODunzps47fa1S
https://api.github.com/repos/huggingface/datasets/issues/comments/998092114/reactions
2021-12-20T16:40:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998092114
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I just opened https://github.com/huggingface/datasets/pull/3460 to fix that. It now raises an error instead of computing the wrong WER
2021-12-20T16:51:31Z
https://github.com/huggingface/datasets/pull/3456#issuecomment-998100490
https://api.github.com/repos/huggingface/datasets/issues/3456
IC_kwDODunzps47fc4K
https://api.github.com/repos/huggingface/datasets/issues/comments/998100490/reactions
2021-12-20T16:51:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998100490
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thank you - that should be good enough!
2021-12-20T16:53:30Z
https://github.com/huggingface/datasets/pull/3456#issuecomment-998101941
https://api.github.com/repos/huggingface/datasets/issues/3456
IC_kwDODunzps47fdO1
https://api.github.com/repos/huggingface/datasets/issues/comments/998101941/reactions
2021-12-20T16:53:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998101941
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following: 


```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.