Shard Dataset at specific indices
MEMBER
Hi ! if it's an option I'd suggest to have one sequence per row instead.

Otherwise you'd have to make your own save/load mechanism
2025-02-20T14:16:00Z
https://github.com/huggingface/datasets/issues/7415#issuecomment-2671631540
https://api.github.com/repos/huggingface/datasets/issues/7415
IC_kwDODunzps6fPdy0
https://api.github.com/repos/huggingface/datasets/issues/comments/2671631540/reactions
2025-02-20T14:16:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2671631540
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Saving one sequence per row is very difficult and heavy and makes all the optimizations pointless. How would a custom save/load mechanism look like?
2025-02-20T15:24:52Z
https://github.com/huggingface/datasets/issues/7415#issuecomment-2671832296
https://api.github.com/repos/huggingface/datasets/issues/7415
IC_kwDODunzps6fQOzo
https://api.github.com/repos/huggingface/datasets/issues/comments/2671832296/reactions
2025-02-20T15:24:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2671832296
https://avatars.githubusercontent.com/u/11044035?v=4
https://api.github.com/users/nikonikolov/events{/privacy}
https://api.github.com/users/nikonikolov/followers
https://api.github.com/users/nikonikolov/following{/other_user}
https://api.github.com/users/nikonikolov/gists{/gist_id}
https://github.com/nikonikolov
nikonikolov
MDQ6VXNlcjExMDQ0MDM1
https://api.github.com/users/nikonikolov/orgs
https://api.github.com/users/nikonikolov/received_events
https://api.github.com/users/nikonikolov/repos
https://api.github.com/users/nikonikolov/starred{/owner}{/repo}
https://api.github.com/users/nikonikolov/subscriptions
User
https://api.github.com/users/nikonikolov
public
MEMBER
You can use `pyarrow` for example to save/load individual arrow or parquet files and control what they contain
2025-02-24T11:06:43Z
https://github.com/huggingface/datasets/issues/7415#issuecomment-2678090494
https://api.github.com/repos/huggingface/datasets/issues/7415
IC_kwDODunzps6foGr-
https://api.github.com/repos/huggingface/datasets/issues/comments/2678090494/reactions
2025-02-24T11:06:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2678090494
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?