BigBench: NonMatchingSplitsSizesError when passing a dataset configuration parameter
MEMBER
Why not adding `max_examples` as part of the config name?
2022-06-09T05:55:10Z
https://github.com/huggingface/datasets/issues/4462#issuecomment-1150701622
https://api.github.com/repos/huggingface/datasets/issues/4462
IC_kwDODunzps5EllA2
https://api.github.com/repos/huggingface/datasets/issues/comments/1150701622/reactions
2022-06-09T05:55:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150701622
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yup it can also work, and maybe it's simpler this way. Opening a PR to fix bigbench instead of https://github.com/huggingface/datasets/pull/4463
2022-06-09T08:02:36Z
https://github.com/huggingface/datasets/issues/4462#issuecomment-1150801724
https://api.github.com/repos/huggingface/datasets/issues/4462
IC_kwDODunzps5El9c8
https://api.github.com/repos/huggingface/datasets/issues/comments/1150801724/reactions
2022-06-09T08:02:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150801724
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq,

Thank you for taking a look at this issue, and proposing a solution. 
Unfortunately, after trying the fix in #4465 I still see the same issue.

I think there is some subtlety where the config name gets overwritten somewhere when `BUILDER_CONFIGS`[(link)](https://github.com/huggingface/datasets/blob/master/datasets/bigbench/bigbench.py#L126) is defined. 

If I print out the `self.config.name` in the current version (with the fix in #4465), I see just the task name, but if I comment out `BUILDER_CONFIGS`, the `num_shots` and `max_examples` gets appended as was meant by #4465.

I haven't managed to track down where this happens, but I thought you might know? 

(Another comment on your fix: the `name` variable is used to fetch the task from the bigbench API, so modifying it causes an error if it's actually called. This can easily be fixed by having `config_name` variable in addition to the `task_name`)



2022-06-30T04:06:23Z
https://github.com/huggingface/datasets/issues/4462#issuecomment-1170730858
https://api.github.com/repos/huggingface/datasets/issues/4462
IC_kwDODunzps5Fx-9q
https://api.github.com/repos/huggingface/datasets/issues/comments/1170730858/reactions
2022-06-30T04:06:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170730858
https://avatars.githubusercontent.com/u/43357549?v=4
https://api.github.com/users/andersjohanandreassen/events{/privacy}
https://api.github.com/users/andersjohanandreassen/followers
https://api.github.com/users/andersjohanandreassen/following{/other_user}
https://api.github.com/users/andersjohanandreassen/gists{/gist_id}
https://github.com/andersjohanandreassen
andersjohanandreassen
MDQ6VXNlcjQzMzU3NTQ5
https://api.github.com/users/andersjohanandreassen/orgs
https://api.github.com/users/andersjohanandreassen/received_events
https://api.github.com/users/andersjohanandreassen/repos
https://api.github.com/users/andersjohanandreassen/starred{/owner}{/repo}
https://api.github.com/users/andersjohanandreassen/subscriptions
User
https://api.github.com/users/andersjohanandreassen
public
As noticed in https://github.com/huggingface/datasets/pull/4125 when a dataset config class has a parameter that reduces the number of examples (e.g. named `max_examples`), then loading the dataset and passing `max_examples` raises `NonMatchingSplitsSizesError`.

This is because it will check for expected the number of examples of the config with the same name without taking into account the `max_examples` parameter. This can be fixed by checking the expected number of examples using the **config id** instead of name. Indeed the config id corresponds to the config name + an optional suffix that depends on the config parameters