Autoeval config
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-27T05:42:47Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1110567146
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CMejq
https://api.github.com/repos/huggingface/datasets/issues/comments/1110567146/reactions
2022-05-05T18:27:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1110567146
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424
2022-04-27T12:07:01Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1110920965
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CN08F
https://api.github.com/repos/huggingface/datasets/issues/comments/1110920965/reactions
2022-04-27T12:44:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1110920965
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
The tests are failing due to the changed metadata:

```
got an unexpected keyword argument 'train-eval-index'
```

I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function

@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?
2022-04-27T12:47:23Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1110958683
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CN-Jb
https://api.github.com/repos/huggingface/datasets/issues/comments/1110958683/reactions
2022-04-27T19:23:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1110958683
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

* `config`
* `task`
* `task_id`
* `splits` (train / validation / eval)
* `col_mapping`
* `metrics` (checking that each one has `type`, `name`) 

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ?
2022-05-03T13:21:55Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1116091379
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5ChjPz
https://api.github.com/repos/huggingface/datasets/issues/comments/1116091379/reactions
2022-05-03T13:21:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1116091379
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it
2022-05-04T12:26:34Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1117252923
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5Cl-07
https://api.github.com/repos/huggingface/datasets/issues/comments/1117252923/reactions
2022-05-04T12:26:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117252923
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I ended up improving the metadata validation in this PR x)

In particular:
- I added support YAML keys with dashes instead of underscores for `train-eval-index`
- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR
2022-05-04T13:56:05Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1117345220
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CmVXE
https://api.github.com/repos/huggingface/datasets/issues/comments/1117345220/reactions
2022-05-04T13:57:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117345220
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)
2022-05-04T14:56:11Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1117426196
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CmpIU
https://api.github.com/repos/huggingface/datasets/issues/comments/1117426196/reactions
2022-05-04T14:56:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117426196
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Come on windows... I didn't do anything advanced...
> 
> Anyway, will try to fix this when I get back home x)

Hehe, thanks!
2022-05-04T15:08:27Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1117450966
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CmvLW
https://api.github.com/repos/huggingface/datasets/issues/comments/1117450966/reactions
2022-05-04T15:08:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117450966
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
NONE
Thanks, @lhoestq this is great! 
2022-05-04T15:10:02Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1117454844
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CmwH8
https://api.github.com/repos/huggingface/datasets/issues/comments/1117454844/reactions
2022-05-04T15:10:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117454844
https://avatars.githubusercontent.com/u/3278583?v=4
https://api.github.com/users/nazneenrajani/events{/privacy}
https://api.github.com/users/nazneenrajani/followers
https://api.github.com/users/nazneenrajani/following{/other_user}
https://api.github.com/users/nazneenrajani/gists{/gist_id}
https://github.com/nazneenrajani
nazneenrajani
MDQ6VXNlcjMyNzg1ODM=
https://api.github.com/users/nazneenrajani/orgs
https://api.github.com/users/nazneenrajani/received_events
https://api.github.com/users/nazneenrajani/repos
https://api.github.com/users/nazneenrajani/starred{/owner}{/repo}
https://api.github.com/users/nazneenrajani/subscriptions
User
https://api.github.com/users/nazneenrajani
public
MEMBER
Did I just fix it for windows and now it fails on linux ? xD
2022-05-05T15:21:49Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1118690531
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5Crdzj
https://api.github.com/repos/huggingface/datasets/issues/comments/1118690531/reactions
2022-05-05T15:21:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118690531
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅 
2022-05-05T15:27:18Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1118697231
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CrfcP
https://api.github.com/repos/huggingface/datasets/issues/comments/1118697231/reactions
2022-05-05T15:27:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118697231
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
The worst is that the tests pass locally both on my windows and my linux x)
2022-05-05T15:35:38Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1118710464
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CrirA
https://api.github.com/repos/huggingface/datasets/issues/comments/1118710464/reactions
2022-05-05T15:35:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118710464
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types
2022-05-05T17:55:12Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1118883066
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CsMz6
https://api.github.com/repos/huggingface/datasets/issues/comments/1118883066/reactions
2022-05-05T17:55:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118883066
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗 
2022-05-05T19:04:07Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1118950052
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5CsdKk
https://api.github.com/repos/huggingface/datasets/issues/comments/1118950052/reactions
2022-05-05T19:04:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118950052
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
CONTRIBUTOR
This is cool!
2022-05-06T13:20:31Z
https://github.com/huggingface/datasets/pull/4234#issuecomment-1119615490
https://api.github.com/repos/huggingface/datasets/issues/4234
IC_kwDODunzps5Cu_oC
https://api.github.com/repos/huggingface/datasets/issues/comments/1119615490/reactions
2022-05-06T13:20:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1119615490
https://avatars.githubusercontent.com/u/9808326?v=4
https://api.github.com/users/fxmarty/events{/privacy}
https://api.github.com/users/fxmarty/followers
https://api.github.com/users/fxmarty/following{/other_user}
https://api.github.com/users/fxmarty/gists{/gist_id}
https://github.com/fxmarty
fxmarty
MDQ6VXNlcjk4MDgzMjY=
https://api.github.com/users/fxmarty/orgs
https://api.github.com/users/fxmarty/received_events
https://api.github.com/users/fxmarty/repos
https://api.github.com/users/fxmarty/starred{/owner}{/repo}
https://api.github.com/users/fxmarty/subscriptions
User
https://api.github.com/users/fxmarty
public
Added autoeval config to imdb as pilot