Download and prepare as Parquet for cloud storage
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-07-20T13:45:30Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1190307937
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5G8qhh
https://api.github.com/repos/huggingface/datasets/issues/comments/1190307937/reactions
2022-09-05T17:27:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1190307937
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Added some docs for dask and took your comments into account

cc @philschmid if you also want to take a look :)
2022-07-29T14:35:21Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1199410101
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5HfYu1
https://api.github.com/repos/huggingface/datasets/issues/comments/1199410101/reactions
2022-07-29T14:35:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1199410101
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Just noticed that it would be more convenient to pass the output dir to download_and_prepare directly, to bypass the caching logic which prepares the dataset at `<cache_dir>/<name>/<version>/<hash>/`. And this way the cache is only used for the downloaded files. What do you think ?

```python 

builder = load_datadet_builder("squad")
# or with a custom cache
builder = load_datadet_builder("squad", cache_dir="path/to/local/cache/for/downloaded/files")

# download and prepare to s3
builder.download_and_prepare("s3://my_bucket/squad")
```
2022-07-30T17:30:14Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1200261694
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5Hioo-
https://api.github.com/repos/huggingface/datasets/issues/comments/1200261694/reactions
2022-07-30T17:38:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1200261694
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Might be of interest: 
PyTorch and AWS introduced better support for S3 streaming in `torchtext`. 
![image](https://user-images.githubusercontent.com/32632186/183354186-a7f005e3-4167-4d80-ad1a-c62dd51ad7b6.png)

2022-08-08T06:34:23Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1207724083
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5H_Ggz
https://api.github.com/repos/huggingface/datasets/issues/comments/1207724083/reactions
2022-08-08T06:34:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1207724083
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
COLLABORATOR
Having thought about it a bit more, I also agree with @philschmid in that it's important to follow the existing APIs (pandas/dask), which means we should support the following at some point:

* remote data files resolution for the packaged modules to support `load_dataset("<format>", data_files="<fs_url>")`
* `to_<format>("<fs_url>")`
* `load_from_disk` and `save_to_disk` already expose the `fs` param, but it would be cool to support specifying `fsspec` URLs directly as the source/destination path (perhaps we can then deprecate `fs` to be fully aligned with pandas/dask)

IMO these are the two main issues with the current approach:
* relying on the builder API to generate the formatted files results in a non-friendly format due to how our caching works (a lot of nested subdirectories)
* this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running `to_<format>` on streamed/iterable datasets
2022-08-10T12:56:56Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1210637020
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5IKNrc
https://api.github.com/repos/huggingface/datasets/issues/comments/1210637020/reactions
2022-08-10T12:58:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1210637020
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Alright I did the last change I wanted to do, here is the final API:

```python
builder = load_dataset_builder(...)
builder.download_and_prepare("s3://...", storage_options={"token": ...})
```

and it creates the arrow files directly in the specified directory, not in a nested subdirectory structure as we do in the cache !

> this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running to_<format> on streamed/iterable datasets

Yup this can be explored in some future work I think. Though to keep things simple and clear I would keep the streaming behaviors only when you load a dataset in streaming mode, and not include it in `download_and_prepare` (because it wouldn't be aligned with the name of the function, which imply to 1. download and 2. prepare ^^). Maybe an API like that can make sense for those who need full streaming

```python
ds = load_dataset(..., streaming=True)
ds.to_parquet("s3://...")
```
2022-08-26T12:34:47Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1228433789
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5JOGl9
https://api.github.com/repos/huggingface/datasets/issues/comments/1228433789/reactions
2022-08-26T12:34:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1228433789
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
totally agree with your comment on the meaning of "loading", I'll update the docs
2022-08-26T15:09:09Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1228614744
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5JOyxY
https://api.github.com/repos/huggingface/datasets/issues/comments/1228614744/reactions
2022-08-26T15:09:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1228614744
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I took your comments into account and reverted all the changes related to `cache_dir` to keep the support for remote `cache_dir` for beam datasets. I also updated the wording in the docs to not use "load" when it's not appropriate :)
2022-08-26T17:32:32Z
https://github.com/huggingface/datasets/pull/4724#issuecomment-1228750251
https://api.github.com/repos/huggingface/datasets/issues/4724
IC_kwDODunzps5JPT2r
https://api.github.com/repos/huggingface/datasets/issues/comments/1228750251/reactions
2022-08-26T17:32:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1228750251
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Download a dataset as Parquet in a cloud storage can be useful for streaming mode and to use with spark/dask/ray.

This PR adds support for `fsspec` URIs like `s3://...`, `gcs://...` etc. and ads the `file_format` to save as parquet instead of arrow:
```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("crime_and_punish", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet")
```

EDIT: actually changed the API to

```python
from datasets import *

builder = load_dataset_builder("crime_and_punish")
builder.download_and_prepare("s3://...", file_format="parquet")
```

credentials to cloud storage can be passed using the `storage_options` argument in

For consistency with the BeamBasedBuilder, I name the parquet files `{builder.name}-{split}-xxxxx-of-xxxxx.parquet`. I think this is fine since we'll need to implement parquet sharding after this PR, so that a dataset can be used efficiently with dask for example.

Note that images/audio files are not embedded yet in the parquet files, this will added in a subsequent PR

TODO:
- [x] docs
- [x] tests