Why max_shard_size is not supported in load_dataset and passed to download_and_prepare
COLLABORATOR
Can you explain your use case for `max_shard_size`? 

On some systems, there is a limit to the size of a memory-mapped file, so we could consider exposing this parameter in `load_dataset`.
2023-06-26T18:52:36Z
https://github.com/huggingface/datasets/issues/5987#issuecomment-1608055333
https://api.github.com/repos/huggingface/datasets/issues/5987
IC_kwDODunzps5f2Pol
https://api.github.com/repos/huggingface/datasets/issues/comments/1608055333/reactions
2023-06-26T18:52:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1608055333
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
In my use case, users may choose a proper size to balance the cost and benefit of using large shard size. (On azure blob or hdfs which may automatically download the shard from background)
2023-06-27T01:59:37Z
https://github.com/huggingface/datasets/issues/5987#issuecomment-1608590487
https://api.github.com/repos/huggingface/datasets/issues/5987
IC_kwDODunzps5f4SSX
https://api.github.com/repos/huggingface/datasets/issues/comments/1608590487/reactions
2023-06-27T01:59:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1608590487
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
COLLABORATOR
But `load_dataset` doesn't support caching (and reading) Arrow datasets from remote storage. 

`load_datset_builder` + `download_and_prepare` is not equal to `load_dataset`. The latter has one more step, `builder.as_dataset`, that memory-maps Arrow files, which only works for local files.
2023-06-28T17:44:40Z
https://github.com/huggingface/datasets/issues/5987#issuecomment-1611836049
https://api.github.com/repos/huggingface/datasets/issues/5987
IC_kwDODunzps5gEqqR
https://api.github.com/repos/huggingface/datasets/issues/comments/1611836049/reactions
2023-06-28T17:44:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1611836049
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Thanks. So if I want to use `IterableDataset` and control the size of single arrow file, how should I organize the data loader? Maybe `load_dataset_build` + `download_and_prepare` + `builder.as_dataset` + `dataset.to_iterable_dataset`?
2023-06-29T03:42:09Z
https://github.com/huggingface/datasets/issues/5987#issuecomment-1612388769
https://api.github.com/repos/huggingface/datasets/issues/5987
IC_kwDODunzps5gGxmh
https://api.github.com/repos/huggingface/datasets/issues/comments/1612388769/reactions
2023-06-29T03:42:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1612388769
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
COLLABORATOR
Yes, this should work.

I think we can expose `max_shard_size` in `load_dataset`, so feel free to open a PR.
2023-06-29T15:24:38Z
https://github.com/huggingface/datasets/issues/5987#issuecomment-1613387103
https://api.github.com/repos/huggingface/datasets/issues/5987
IC_kwDODunzps5gKlVf
https://api.github.com/repos/huggingface/datasets/issues/comments/1613387103/reactions
2023-06-29T15:24:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1613387103
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

What I can to is break the `load_dataset` and use `load_datset_builder` + `download_and_prepare` instead.

### Steps to reproduce the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

### Expected behavior

Users can define the max shard size.

### Environment info

datasets==2.13.1