Check invalid config name
CONTRIBUTOR
I think that's not related to the config name but the filenames in the dummy data. Mostly it occurs with files downloaded from drive. In that case the dummy file name is extracted from the google drive link and it corresponds to what comes after `https://drive.google.com/`


2020-05-26T14:05:39Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634046922
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA0NjkyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/634046922/reactions
2020-05-26T14:05:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634046922
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
MEMBER
> I think that's not related to the config name but the filenames in the dummy data. Mostly it occurs with files downloaded from drive. In that case the dummy file name is extracted from the google drive link and it corresponds to what comes after `https://drive.google.com/`

The filenames of the dummy data are now encoded (see #173). So this is not a problem anymore.

The problem here is different and comes from the directory names where we save the arrow files (basically `dataset_name/config_name/version`). In this case we could have invalid directory names because of the config name

2020-05-26T14:08:57Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634048878
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA0ODg3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/634048878/reactions
2020-05-26T14:08:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634048878
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Okay great then.
2020-05-26T14:10:45Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634049936
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA0OTkzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/634049936/reactions
2020-05-26T14:10:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634049936
https://avatars.githubusercontent.com/u/38249783?v=4
https://api.github.com/users/mariamabarham/events{/privacy}
https://api.github.com/users/mariamabarham/followers
https://api.github.com/users/mariamabarham/following{/other_user}
https://api.github.com/users/mariamabarham/gists{/gist_id}
https://github.com/mariamabarham
mariamabarham
MDQ6VXNlcjM4MjQ5Nzgz
https://api.github.com/users/mariamabarham/orgs
https://api.github.com/users/mariamabarham/received_events
https://api.github.com/users/mariamabarham/repos
https://api.github.com/users/mariamabarham/starred{/owner}{/repo}
https://api.github.com/users/mariamabarham/subscriptions
User
https://api.github.com/users/mariamabarham
public
CONTRIBUTOR
I like the method, but I'm wondering whether it should just be a test method instead of a `__post_init__` function. From a logical point of view the only reason this error would be thrown is because of an invalid config name introduced when creating the dataset script / adding a new dataset => so I think it might be better to write a simple test for this in `test_dataset_common.py`...what do you think @lhoestq ?
2020-05-26T14:22:42Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634057022
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA1NzAyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/634057022/reactions
2020-05-26T14:22:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634057022
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
`test_dataset_common.py` only tests canonical datasets no ? What if users wants to create their own script ?
2020-05-26T14:25:46Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634058908
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA1ODkwOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/634058908/reactions
2020-05-26T14:25:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634058908
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> `test_dataset_common.py` only tests canonical datasets no ? What if users wants to create their own script ?

It tests all dataset that can be loaded either locally or on AWS (which includes all non-canonical datasets as well)...by their own script you mean like a private dataset script that they don't want to be public? I guess even then they could locally run the test functions to check...
2020-05-26T14:29:24Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634061129
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA2MTEyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634061129/reactions
2020-05-26T14:29:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634061129
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
We could have a bunch of simple consistency tests that run before uploading with the CLI (without loading data if we don't want to force the user to have dummy data)?
2020-05-26T14:39:24Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634067425
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA2NzQyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634067425/reactions
2020-05-26T14:39:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634067425
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Let's say someone want to create his own private script. As the script is not meant to be shared, it's not going to be placed in `/datasets` right ? Maybe the script is going to be inside another project. If I'm not wrong in this case the `test_dataset_common.py` is not going to test his script.

Raising an error in the post init is a sanity check that would tell the user immediately what's wrong.
The error is raised if he tried to load the script or if he uses `nlp-cli test`
2020-05-26T14:44:22Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634070477
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA3MDQ3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/634070477/reactions
2020-05-26T14:44:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634070477
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Let's say someone want to create his own private script. As the script is not meant to be shared, it's not going to be placed in `/datasets` right ? Maybe the script is going to be inside another project. If I'm not wrong in this case the `test_dataset_common.py` is not going to test his script.
> 
> Raising an error in the post init is a sanity check that would tell the user immediately what's wrong.
> The error is raised if he tried to load the script or if he uses `nlp-cli test`

OK, fair point! I'm good with this then :-) 
2020-05-26T14:45:11Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634070971
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA3MDk3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634070971/reactions
2020-05-26T14:45:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634070971
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I'm fine with this as well (even though I understand what you meant @patrickvonplaten, we can still change it later if needed)
2020-05-26T14:45:52Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634071354
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA3MTM1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/634071354/reactions
2020-05-26T14:45:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634071354
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
> We could have a bunch of simple consistency tests that run before uploading with the CLI (without loading data if we don't want to force the user to have dummy data)?

Yes! I guess that's a big question whether we should force the user to add dummy data. It's probably too tedious for the user...so when uploading to circle ci should we just check 
- 1) All configs can be instantiated (if there are any)
- 2) The BuilderClass can be instantiated ... 
- 3) ... maybe some more

and maybe suggest to the user to add dummy data using the dummy data command?
2020-05-26T14:47:14Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634072137
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDA3MjEzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/634072137/reactions
2020-05-26T14:47:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634072137
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I really like that we have a test with dummy data for canonical datasets. This is insurance that they'll keep working in the long run. 

On the other hand I understand that we will probably not force this practice for scripts uploaded on S3 by a user under his namespace (non-canonical), as it is tedious. As I understand right now the test is done for all the datasets on aws, even the non-canonical ? We should think about different tests for non-canonical datasets.

I also like the idea of a simple consistency test !
2020-05-26T21:02:52Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634277587
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDI3NzU4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/634277587/reactions
2020-05-26T21:02:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634277587
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Merging this one for now, we can think about the test for non-canonical datasets later
2020-05-26T21:04:24Z
https://github.com/huggingface/datasets/pull/196#issuecomment-634278255
https://api.github.com/repos/huggingface/datasets/issues/196
MDEyOklzc3VlQ29tbWVudDYzNDI3ODI1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634278255/reactions
2020-05-26T21:04:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634278255
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
As said in #194, we should raise an error if the config name has bad characters.
Bad characters are those that are not allowed for directory names on windows.