Don't duplicate data when encoding audio or image
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-20T14:00:39Z
https://github.com/huggingface/datasets/pull/4187#issuecomment-1103969222
https://api.github.com/repos/huggingface/datasets/issues/4187
IC_kwDODunzps5BzTvG
https://api.github.com/repos/huggingface/datasets/issues/comments/1103969222/reactions
2022-04-21T09:17:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1103969222
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
I'm not familiar with the concept of streaming vs non-streaming in HF datasets. I just wonder that you have the distinction here. Why doesn't it work to always make use of `bytes`? "using a local file - which is often required for audio" - why would that be?

The `path` would always point to some location in the `cache_dir`? I think this can be problematic. I would have expected that after I did `dataset.save_to_disk(...)` that I can remove the cache dir. But maybe just because I'm not familiar with HF. Or maybe the docs can be improved to clarify this.

2022-04-20T15:13:01Z
https://github.com/huggingface/datasets/pull/4187#issuecomment-1104050057
https://api.github.com/repos/huggingface/datasets/issues/4187
IC_kwDODunzps5BzneJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1104050057/reactions
2022-04-20T15:13:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1104050057
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
CONTRIBUTOR
We could always load every data file into `bytes` and save it this way the audio as bytes in `arrow` format, but the problem then would be that it makes the `file` column useless, *i.e.* people cannot inspect the audio file locally anymore or else they would need to first save bytes as a file which is not evident. This either breaks backwards compatibility or forces the user to stored 2x the required size locally. There was a longer discussion here: https://github.com/huggingface/datasets/issues/3663

It's a good argument though that `dataset.save_to_disk(...)` should save everything that is needed to the disk and should be independent of other folders, but I do think the arguments of #3663 to not break backwards compatibility and to allow people to inspect the downloaded audio files locally are a bit more important here. 

But maybe, we could add a flag, `save_files_as_bytes` or `make_independent`, `make_self_contained` or a better name to `save_to_disk(...)` and `push_to_hub(...)` that would allow to make the resulting folder completely independent. 
2022-04-20T17:55:19Z
https://github.com/huggingface/datasets/pull/4187#issuecomment-1104247388
https://api.github.com/repos/huggingface/datasets/issues/4187
IC_kwDODunzps5B0Xpc
https://api.github.com/repos/huggingface/datasets/issues/comments/1104247388/reactions
2022-04-20T17:56:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1104247388
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
What do you think @mariosasko @lhoestq @polinaeterna @anton-l ?

2022-04-20T17:55:29Z
https://github.com/huggingface/datasets/pull/4187#issuecomment-1104247560
https://api.github.com/repos/huggingface/datasets/issues/4187
IC_kwDODunzps5B0XsI
https://api.github.com/repos/huggingface/datasets/issues/comments/1104247560/reactions
2022-04-20T17:55:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1104247560
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
For context: you can either store the path to local images or audio files, or the bytes of those files.

If your images and audio files are local files, then the arrow file from `save_to_disk` will store paths to these files.
If you want to include the bytes or your images or audio files instead, you must `read()` those files first.
This can be done by storing the "bytes" instead of the "path" of the images or audio files.

On the other hand, the resulting Parquet files from `push_to_hub` are self-contained, so that anyone can reload the dataset from the Hub. If your dataset contains image or audio data, the Parquet files will store the bytes of your images or audio files.

For now I just updated the documentation: https://github.com/huggingface/datasets/pull/4193. Maybe we can also embed the image and audio bytes in `save_to_disk` when we implement sharding, so that is can be done as efficiently as `push_to_hub`.

Anyway, merging this one :)
2022-04-21T09:10:30Z
https://github.com/huggingface/datasets/pull/4187#issuecomment-1104924122
https://api.github.com/repos/huggingface/datasets/issues/4187
IC_kwDODunzps5B283a
https://api.github.com/repos/huggingface/datasets/issues/comments/1104924122/reactions
2022-04-21T09:10:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1104924122
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Right now if you pass both the `bytes` and a local `path` for audio or image data, then the `bytes` are unnecessarily written in the Arrow file, while we could just keep the local `path`.

This PR discards the `bytes` when the audio or image file exists locally.

In particular it's common for audio datasets builders to provide both the bytes and the local path in order to work for both streaming (using the bytes) and non-streaming mode (using a local file - which is often required for audio).

cc @patrickvonplaten 