Support remote cache_dir
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-13T14:38:37Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1126130976
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5DH2Ug
https://api.github.com/repos/huggingface/datasets/issues/comments/1126130976/reactions
2022-05-25T16:35:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1126130976
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
@lhoestq thanks for your review.

Please note that `xjoin` cannot be used in this context, as it always returns a POSIX path string and this is not suitable on Windows machines.
2022-05-20T11:22:01Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1132785549
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5DhO-N
https://api.github.com/repos/huggingface/datasets/issues/comments/1132785549/reactions
2022-05-20T11:26:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132785549
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
<s>`xjoin` returns windows paths (not posix) on windows, since it just extends`os.path.join` </s>

Actually you are right.

https://github.com/huggingface/datasets/blob/08ec04ccb59630a3029b2ecd8a14d327bddd0c4a/src/datasets/utils/streaming_download_manager.py#L104-L105

Though this is not an issue because posix paths (as returned by Path().as_posix()) work on windows. That's why we can replace `os.path.join` with `xjoin` in streaming mode. They look like `c:/Program Files/` or something (can't confirm right now, I don't have a windows with me)
2022-05-24T15:20:26Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1136066505
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5Dtv_J
https://api.github.com/repos/huggingface/datasets/issues/comments/1136066505/reactions
2022-05-24T15:29:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1136066505
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Until now, we have always replaced "/" in paths with `os.path.join` (`os.sep`,...) in order to support Windows paths (that contain r"\\").

Now, you suggest ignoring this and work with POSIX strings (with "/").

As an example, when passing `cache_dir=r"C:\Users\Username\.mycache"`:
- Until now, it results in `self._cache_downloaded_dir = r"C:\Users\Username\.mycache\downloads"`
- If we use `xjoin`, it will give `self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"`

You say this is OK and we don't care if we work with POSIX strings on Windows machines.

I'm incorporating your suggested changes then...
2022-05-25T07:02:03Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1136845502
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5DwuK-
https://api.github.com/repos/huggingface/datasets/issues/comments/1136845502/reactions
2022-05-25T07:03:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1136845502
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Also note that using `xjoin`, if we pass `cache_dir="C:\\Users\\Username\\.mycache"`, we get:
- `self._cache_dir_root = "C:\\Users\\Username\\.mycache"`
- `self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"`
2022-05-25T08:18:54Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1136938145
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5DxEyh
https://api.github.com/repos/huggingface/datasets/issues/comments/1136938145/reactions
2022-05-25T08:18:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1136938145
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
It looks like it broke the CI on windows :/ maybe this was not a good idea, sorry
2022-05-25T10:09:03Z
https://github.com/huggingface/datasets/pull/4347#issuecomment-1137052355
https://api.github.com/repos/huggingface/datasets/issues/4347
IC_kwDODunzps5DxgrD
https://api.github.com/repos/huggingface/datasets/issues/comments/1137052355/reactions
2022-05-25T10:09:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1137052355
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR implements complete support for remote `cache_dir`. Before, the support was just partial.

This is useful to create datasets using Apache Beam (parallel data processing) builder with `cache_dir` in a remote bucket, e.g., for Wikipedia dataset.