Sharded save_to_disk + multiprocessing
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-11-18T18:58:07Z
https://github.com/huggingface/datasets/pull/5268#issuecomment-1320413728
https://api.github.com/repos/huggingface/datasets/issues/5268
IC_kwDODunzps5Os-og
https://api.github.com/repos/huggingface/datasets/issues/comments/1320413728/reactions
2022-12-14T18:25:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1320413728
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Added both num_shards and max_shard_size in push_to_hub/save_to_disk. Will take care of updating the tests later
2022-12-08T17:38:56Z
https://github.com/huggingface/datasets/pull/5268#issuecomment-1343070636
https://api.github.com/repos/huggingface/datasets/issues/5268
IC_kwDODunzps5QDaGs
https://api.github.com/repos/huggingface/datasets/issues/comments/1343070636/reactions
2022-12-08T17:38:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1343070636
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It's ready for a final review @mariosasko and @albertvillanova, let me know what you think :)
2022-12-12T18:16:27Z
https://github.com/huggingface/datasets/pull/5268#issuecomment-1346991934
https://api.github.com/repos/huggingface/datasets/issues/5268
IC_kwDODunzps5QSXc-
https://api.github.com/repos/huggingface/datasets/issues/comments/1346991934/reactions
2022-12-12T18:16:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1346991934
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Took your comments into account, and also changed `iflatmap_unordered` to take an iterable of kwargs to make the code more redable :)
2022-12-14T17:31:11Z
https://github.com/huggingface/datasets/pull/5268#issuecomment-1351825088
https://api.github.com/repos/huggingface/datasets/issues/5268
IC_kwDODunzps5QkzbA
https://api.github.com/repos/huggingface/datasets/issues/comments/1351825088/reactions
2022-12-14T17:31:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1351825088
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263 
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351