Fix URL with backslash in Windows
MEMBER
@lhoestq I was indeed working on that... to make another commit on this feature branch...
2020-11-27T16:38:26Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734914482
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxNDQ4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/734914482/reactions
2020-11-27T16:38:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734914482
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
But as you prefer... nevermind! :)
2020-11-27T16:40:42Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734915271
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxNTI3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/734915271/reactions
2020-11-27T16:40:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734915271
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Ah what do you have in mind for the tests ? I was thinking of adding a check in the MockDownloadManager used for tests based on dummy data. I'm creating a PR right now, I'd be happy to have your opinion
2020-11-27T16:43:48Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734916231
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxNjIzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/734916231/reactions
2020-11-27T16:43:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734916231
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Indeed I was thinking of something similar: monckeypatching the HTTP request...
2020-11-27T16:45:08Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734916704
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxNjcwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/734916704/reactions
2020-11-27T16:45:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734916704
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Therefore, if you agree, I am removing all the rest of `os.path.join`, both from the code and the docs...
2020-11-27T16:49:09Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734918153
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxODE1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/734918153/reactions
2020-11-27T16:49:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734918153
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
If you spot other `os.path.join` for urls in dataset scripts or metrics scripts feel free to fix them.
In the library itself (/src/datasets) it should be fine since there are tests and a windows CI, but if you have doubts of some usage of `os.path.join` somewhere, let me know.
2020-11-27T16:52:06Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734919138
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkxOTEzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/734919138/reactions
2020-11-27T16:52:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734919138
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright create the test in #905 .
The windows CI is failing for all the datasets that have bad usage of `os.path.join` for urls.
There are of course the ones you fixed in this PR (thanks again !) but I found others as well such as pg19 and blimp.
You can check the full list by looking at the CI failures of the commit 1ce3354
2020-11-27T17:43:13Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734934874
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkzNDg3NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/734934874/reactions
2020-11-27T17:55:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734934874
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I am merging this one as well as #906 that should fix all of the datasets.
Then I'll rebase #905 which adds the test that checks for bad urls and make sure it' all green now
2020-11-27T18:00:16Z
https://github.com/huggingface/datasets/pull/903#issuecomment-734939725
https://api.github.com/repos/huggingface/datasets/issues/903
MDEyOklzc3VlQ29tbWVudDczNDkzOTcyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/734939725/reactions
2020-11-27T18:00:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734939725
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
In Windows, `os.path.join` generates URLs containing backslashes, when the first "path" does not end with a slash. 

In general, `os.path.join` should be avoided to generate URLs.