[Feature request] Add `shard()` method to dataset
MEMBER
Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?
2020-06-24T23:06:15Z
https://github.com/huggingface/datasets/issues/312#issuecomment-649118899
https://api.github.com/repos/huggingface/datasets/issues/312
MDEyOklzc3VlQ29tbWVudDY0OTExODg5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/649118899/reactions
2020-06-24T23:06:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/649118899
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.
2020-06-24T23:21:17Z
https://github.com/huggingface/datasets/issues/312#issuecomment-649123773
https://api.github.com/repos/huggingface/datasets/issues/312
MDEyOklzc3VlQ29tbWVudDY0OTEyMzc3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/649123773/reactions
2020-06-24T23:25:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/649123773
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.