Metadata validation
CONTRIBUTOR
> Also I was wondering this is really needed to have `utils.metadata` as a submodule of `datasets` ? This is only used by the CI so I'm not sure we should have this in the actual `datasets` package.

I'm unclear on the suggestion, would you rather have a root-level `./metadata.py` file? I think it's well where it is, if anything we could move it out of utils and into `datasets` as it could be used by e.g. `DatasetDict` so that users can pull the metadata easily rather than have to reparse the readme.

2021-03-24T15:09:24Z
https://github.com/huggingface/datasets/pull/2107#issuecomment-805902407
https://api.github.com/repos/huggingface/datasets/issues/2107
MDEyOklzc3VlQ29tbWVudDgwNTkwMjQwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/805902407/reactions
2021-03-24T15:09:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/805902407
https://avatars.githubusercontent.com/u/17948980?v=4
https://api.github.com/users/theo-m/events{/privacy}
https://api.github.com/users/theo-m/followers
https://api.github.com/users/theo-m/following{/other_user}
https://api.github.com/users/theo-m/gists{/gist_id}
https://github.com/theo-m
theo-m
MDQ6VXNlcjE3OTQ4OTgw
https://api.github.com/users/theo-m/orgs
https://api.github.com/users/theo-m/received_events
https://api.github.com/users/theo-m/repos
https://api.github.com/users/theo-m/starred{/owner}{/repo}
https://api.github.com/users/theo-m/subscriptions
User
https://api.github.com/users/theo-m
public
MEMBER
Ok that makes sense if we want to have functions that parse the metadata for users
2021-03-25T12:02:43Z
https://github.com/huggingface/datasets/pull/2107#issuecomment-806608300
https://api.github.com/repos/huggingface/datasets/issues/2107
MDEyOklzc3VlQ29tbWVudDgwNjYwODMwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/806608300/reactions
2021-03-25T12:02:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806608300
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @theo-m @lhoestq 

This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan
2021-03-25T19:35:03Z
https://github.com/huggingface/datasets/pull/2107#issuecomment-807344345
https://api.github.com/repos/huggingface/datasets/issues/2107
MDEyOklzc3VlQ29tbWVudDgwNzM0NDM0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/807344345/reactions
2021-03-25T19:35:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/807344345
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
> Hi @theo-m @lhoestq
> 
> This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)
> 
> Sorry for the delay in responding.
> 
> Thanks,
> Gunjan

Hi @gchhablani, yes I think at the moment the best solution is for you to write in `datasets-tagging`, as the PR will allow us to discuss and review, even though the work will be ported to this repo in the end. 
Or we wait for this to be merged and you reopen the PR here, your call :)
2021-03-26T11:21:03Z
https://github.com/huggingface/datasets/pull/2107#issuecomment-808134563
https://api.github.com/repos/huggingface/datasets/issues/2107
MDEyOklzc3VlQ29tbWVudDgwODEzNDU2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/808134563/reactions
2021-03-26T11:21:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808134563
https://avatars.githubusercontent.com/u/17948980?v=4
https://api.github.com/users/theo-m/events{/privacy}
https://api.github.com/users/theo-m/followers
https://api.github.com/users/theo-m/following{/other_user}
https://api.github.com/users/theo-m/gists{/gist_id}
https://github.com/theo-m
theo-m
MDQ6VXNlcjE3OTQ4OTgw
https://api.github.com/users/theo-m/orgs
https://api.github.com/users/theo-m/received_events
https://api.github.com/users/theo-m/repos
https://api.github.com/users/theo-m/starred{/owner}{/repo}
https://api.github.com/users/theo-m/subscriptions
User
https://api.github.com/users/theo-m
public
CONTRIBUTOR
cc @abhi1thakur 
2021-04-23T11:23:29Z
https://github.com/huggingface/datasets/pull/2107#issuecomment-825591576
https://api.github.com/repos/huggingface/datasets/issues/2107
MDEyOklzc3VlQ29tbWVudDgyNTU5MTU3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/825591576/reactions
2021-04-23T11:23:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/825591576
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
- `pydantic` metadata schema with dedicated validators against our taxonomy
- ci script to validate new changes against this schema and start a vertuous loop
- soft validation on tasks ids since we expect the taxonomy to undergo some changes in the near future

for reference with the current validation we have ~365~ 378 datasets with invalid metadata! full error report [_here_.](https://gist.github.com/theo-m/61b3c0c47fc6121d08d3174bd4c2a26b)