Allow load_dataset to take a working dir for intermediate data
MEMBER
Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.
2023-03-30T10:39:09Z
https://github.com/huggingface/datasets/issues/5679#issuecomment-1490077402
https://api.github.com/repos/huggingface/datasets/issues/5679
IC_kwDODunzps5Y0Mba
https://api.github.com/repos/huggingface/datasets/issues/comments/1490077402/reactions
2023-03-30T10:39:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1490077402
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache. 
2023-04-04T07:26:26Z
https://github.com/huggingface/datasets/issues/5679#issuecomment-1495478499
https://api.github.com/repos/huggingface/datasets/issues/5679
IC_kwDODunzps5ZIzDj
https://api.github.com/repos/huggingface/datasets/issues/comments/1495478499/reactions
2023-04-04T07:26:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495478499
https://avatars.githubusercontent.com/u/38018689?v=4
https://api.github.com/users/lu-wang-dl/events{/privacy}
https://api.github.com/users/lu-wang-dl/followers
https://api.github.com/users/lu-wang-dl/following{/other_user}
https://api.github.com/users/lu-wang-dl/gists{/gist_id}
https://github.com/lu-wang-dl
lu-wang-dl
MDQ6VXNlcjM4MDE4Njg5
https://api.github.com/users/lu-wang-dl/orgs
https://api.github.com/users/lu-wang-dl/received_events
https://api.github.com/users/lu-wang-dl/repos
https://api.github.com/users/lu-wang-dl/starred{/owner}{/repo}
https://api.github.com/users/lu-wang-dl/subscriptions
User
https://api.github.com/users/lu-wang-dl
public
MEMBER
You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk
2023-04-04T09:38:24Z
https://github.com/huggingface/datasets/issues/5679#issuecomment-1495657633
https://api.github.com/repos/huggingface/datasets/issues/5679
IC_kwDODunzps5ZJeyh
https://api.github.com/repos/huggingface/datasets/issues/comments/1495657633/reactions
2023-04-04T09:38:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495657633
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?
2023-04-12T22:30:25Z
https://github.com/huggingface/datasets/issues/5679#issuecomment-1506049006
https://api.github.com/repos/huggingface/datasets/issues/5679
IC_kwDODunzps5ZxHvu
https://api.github.com/repos/huggingface/datasets/issues/comments/1506049006/reactions
2023-04-12T22:30:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1506049006
https://avatars.githubusercontent.com/u/12763339?v=4
https://api.github.com/users/es94129/events{/privacy}
https://api.github.com/users/es94129/followers
https://api.github.com/users/es94129/following{/other_user}
https://api.github.com/users/es94129/gists{/gist_id}
https://github.com/es94129
es94129
MDQ6VXNlcjEyNzYzMzM5
https://api.github.com/users/es94129/orgs
https://api.github.com/users/es94129/received_events
https://api.github.com/users/es94129/repos
https://api.github.com/users/es94129/starred{/owner}{/repo}
https://api.github.com/users/es94129/subscriptions
User
https://api.github.com/users/es94129
public
### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.