Implement sharding on merged iterable datasets
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-04-11T10:06:23Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1503038765
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5Zlo0t
https://api.github.com/repos/huggingface/datasets/issues/comments/1503038765/reactions
2023-04-27T16:35:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503038765
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Hi ! What if one of the sub-iterables only has one shard ? In that case I don't think we'd end up with a correctly interleaved dataset, since only rank 0 would yield examples from this sub-iterable
2023-04-11T15:16:44Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1503588658
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5ZnvEy
https://api.github.com/repos/huggingface/datasets/issues/comments/1503588658/reactions
2023-04-11T15:16:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503588658
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi ! 
I just tested this out with the code below and it seems to be ok. Both datasets are alternating and we get all the examples with no duplicates.

On thing to keep in mind is that the max amount of workers is equal to the lowest amount of shard amongst the datasets to be merged (1 in this example).

 ```python
from torch.utils.data import DataLoader

from datasets import load_dataset, interleave_datasets


def process_dataset_train(batch):
    return {"input": f'train: {batch["review"][:20]}'}


def process_dataset_test(batch):
    return {"input": f'test: {batch["review"][:20]}'}


def identity_collator(x):
    return x


if __name__ == "__main__":
    ds = load_dataset("lhoestq/demo1")
    ds["train"] = ds["train"].map(process_dataset_train, remove_columns=ds["train"].column_names)
    ds["test"] = ds["test"].map(process_dataset_test, remove_columns=ds["test"].column_names)

    ds1 = ds["train"].to_iterable_dataset(num_shards=5)
    ds2 = ds["test"].to_iterable_dataset(num_shards=1)

    ds_merged = interleave_datasets([ds1, ds2], stopping_strategy="all_exhausted")

    dataloader = DataLoader(ds_merged, collate_fn=identity_collator, num_workers=1, batch_size=1)

    for i, element in enumerate(dataloader):
        print(i, element)

```

```
0 [{'input': 'train: Great app! The new v'}]
1 [{'input': 'test: Works with RTL and N'}]
2 [{'input': "train: Great It's not fully"}]
3 [{'input': 'test: Works with RTL SDR W'}]
4 [{'input': 'train: Works on a Nexus 6p '}]
5 [{'input': 'test: Awsome App! Easy to '}]
6 [{'input': 'train: The bandwidth seemed'}]
7 [{'input': "test: I'll forgo the refun"}]
8 [{'input': 'train: Works well with my H'}]
9 [{'input': 'test: looks like a great p'}]
```
2023-04-11T16:17:37Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1503701669
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5ZoKql
https://api.github.com/repos/huggingface/datasets/issues/comments/1503701669/reactions
2023-04-11T16:17:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503701669
https://avatars.githubusercontent.com/u/48770768?v=4
https://api.github.com/users/bruno-hays/events{/privacy}
https://api.github.com/users/bruno-hays/followers
https://api.github.com/users/bruno-hays/following{/other_user}
https://api.github.com/users/bruno-hays/gists{/gist_id}
https://github.com/bruno-hays
bruno-hays
MDQ6VXNlcjQ4NzcwNzY4
https://api.github.com/users/bruno-hays/orgs
https://api.github.com/users/bruno-hays/received_events
https://api.github.com/users/bruno-hays/repos
https://api.github.com/users/bruno-hays/starred{/owner}{/repo}
https://api.github.com/users/bruno-hays/subscriptions
User
https://api.github.com/users/bruno-hays
public
MEMBER
<s> Could you try with `num_workers>1` ? </s>

edit: Oh I see

> On thing to keep in mind is that the max amount of workers is equal to the lowest amount of shard amongst the datasets to be merged (1 in this example).
2023-04-11T18:18:30Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1503876178
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5Zo1RS
https://api.github.com/repos/huggingface/datasets/issues/comments/1503876178/reactions
2023-04-11T18:21:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503876178
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Great ! It's ok to have the max amount of workers is equal to the lowest amount of shard :)

So in the case of `num_workers>min(n_shards_per_dataset)` maybe some workers should turn off, and a warning can probably be shown. This is already the case if you use a single dataset with a single shard and `num_workers>1`.


Right now it seems to raise an error:

```python
  File "/Users/quentinlhoest/hf/datasets/src/datasets/iterable_dataset.py", line 979, in __iter__
    yield from self._iter_pytorch(ex_iterable)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/iterable_dataset.py", line 912, in _iter_pytorch
    for key, example in ex_iterable.shard_data_sources(worker_info.id, worker_info.num_workers):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/iterable_dataset.py", line 259, in shard_data_sources
    [iterable.shard_data_sources(worker_id, num_workers) for iterable in self.ex_iterables],
  File "/Users/quentinlhoest/hf/datasets/src/datasets/iterable_dataset.py", line 259, in <listcomp>
    [iterable.shard_data_sources(worker_id, num_workers) for iterable in self.ex_iterables],
  File "/Users/quentinlhoest/hf/datasets/src/datasets/iterable_dataset.py", line 125, in shard_data_sources
    requested_gen_kwargs = _merge_gen_kwargs([gen_kwargs_list[i] for i in shard_indices])
  File "/Users/quentinlhoest/hf/datasets/src/datasets/utils/sharding.py", line 76, in _merge_gen_kwargs
    for key in gen_kwargs_list[0]
IndexError: list index out of range
```
2023-04-11T18:39:05Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1503897991
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5Zo6mH
https://api.github.com/repos/huggingface/datasets/issues/comments/1503897991/reactions
2023-04-11T18:39:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503897991
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Good point. I have fixed the n_shards property of merged iterable datasets so that this warning is raised properly
2023-04-12T07:14:41Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1504783090
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5ZsSry
https://api.github.com/repos/huggingface/datasets/issues/comments/1504783090/reactions
2023-04-12T07:14:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1504783090
https://avatars.githubusercontent.com/u/48770768?v=4
https://api.github.com/users/bruno-hays/events{/privacy}
https://api.github.com/users/bruno-hays/followers
https://api.github.com/users/bruno-hays/following{/other_user}
https://api.github.com/users/bruno-hays/gists{/gist_id}
https://github.com/bruno-hays
bruno-hays
MDQ6VXNlcjQ4NzcwNzY4
https://api.github.com/users/bruno-hays/orgs
https://api.github.com/users/bruno-hays/received_events
https://api.github.com/users/bruno-hays/repos
https://api.github.com/users/bruno-hays/starred{/owner}{/repo}
https://api.github.com/users/bruno-hays/subscriptions
User
https://api.github.com/users/bruno-hays
public
CONTRIBUTOR
Hey @lhoestq, what do you think of the last modifications ? 
2023-04-26T11:10:14Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1523238763
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5aysdr
https://api.github.com/repos/huggingface/datasets/issues/comments/1523238763/reactions
2023-04-26T11:10:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1523238763
https://avatars.githubusercontent.com/u/48770768?v=4
https://api.github.com/users/bruno-hays/events{/privacy}
https://api.github.com/users/bruno-hays/followers
https://api.github.com/users/bruno-hays/following{/other_user}
https://api.github.com/users/bruno-hays/gists{/gist_id}
https://github.com/bruno-hays
bruno-hays
MDQ6VXNlcjQ4NzcwNzY4
https://api.github.com/users/bruno-hays/orgs
https://api.github.com/users/bruno-hays/received_events
https://api.github.com/users/bruno-hays/repos
https://api.github.com/users/bruno-hays/starred{/owner}{/repo}
https://api.github.com/users/bruno-hays/subscriptions
User
https://api.github.com/users/bruno-hays
public
CONTRIBUTOR
Hello! No problem :)

- About HorizontallyConcatenatedMultiSourcesExamplesIterable, I've haven't been able to create a bug with sharding. So either I missed something or it's working somehow:

```python
from torch.utils.data import DataLoader

from datasets import load_dataset, interleave_datasets, concatenate_datasets


def process_dataset_train(batch):
    return {"input": f'train: {batch["review"][:20]}'}


def process_dataset_test(batch):
    return {"input": f'test: {batch["review"][:20]}'}


def identity_collator(x):
    return x


if __name__ == "__main__":
    ds = load_dataset("lhoestq/demo1")
    ds["train"] = ds["train"].map(process_dataset_train, remove_columns=ds["train"].column_names)
    ds["test"] = ds["test"].map(process_dataset_test, remove_columns=ds["test"].column_names)
    ds["test"] = ds["test"].rename_columns({"input": "input2"})

    ds1 = ds["train"].to_iterable_dataset(num_shards=5)
    ds2 = ds["test"].to_iterable_dataset(num_shards=3)

    ds_merged = concatenate_datasets([ds1, ds2], axis=1)

    #n_shards is always 1 for HorizontallyConcatenatedMultiSourcesExamplesIterable
    dataloader = DataLoader(ds_merged, collate_fn=identity_collator, num_workers=1, batch_size=1)

    for i, element in enumerate(dataloader):
        print(i, element)
```

```
0 [{'input': 'train: Great app! The new v', 'input2': 'test: Works with RTL and N'}]
1 [{'input': "train: Great It's not fully", 'input2': 'test: Works with RTL SDR W'}]
2 [{'input': 'train: Works on a Nexus 6p ', 'input2': 'test: Awsome App! Easy to '}]
3 [{'input': 'train: The bandwidth seemed', 'input2': "test: I'll forgo the refun"}]
4 [{'input': 'train: Works well with my H', 'input2': 'test: looks like a great p'}]
```

- I've added a test but I'm not completely happy with it. My issue is that multiprocessing makes interleaving not completely deterministic as samples are yielded whenever ready by each process, if I'm correct.
As a result I opted to check for the amount of samples yielded and make that they are all unique, which should be equivalent.
But now my issue is that the "first_exhausted" method breaks the loop when one of the datasets of one of the shards is empty which means that all shards stop yielding and we could be missing up to n_workers samples. I don't know if this is the behaviour expected, but I had to modify the  test to accomodate this.

What are your thoughts about this ?
2023-04-27T13:20:30Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1525686116
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5a8B9k
https://api.github.com/repos/huggingface/datasets/issues/comments/1525686116/reactions
2023-04-27T13:20:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1525686116
https://avatars.githubusercontent.com/u/48770768?v=4
https://api.github.com/users/bruno-hays/events{/privacy}
https://api.github.com/users/bruno-hays/followers
https://api.github.com/users/bruno-hays/following{/other_user}
https://api.github.com/users/bruno-hays/gists{/gist_id}
https://github.com/bruno-hays
bruno-hays
MDQ6VXNlcjQ4NzcwNzY4
https://api.github.com/users/bruno-hays/orgs
https://api.github.com/users/bruno-hays/received_events
https://api.github.com/users/bruno-hays/repos
https://api.github.com/users/bruno-hays/starred{/owner}{/repo}
https://api.github.com/users/bruno-hays/subscriptions
User
https://api.github.com/users/bruno-hays
public
MEMBER
Ah indeed it works because it's set to be only 1 shard - my bad :)
2023-04-27T15:17:23Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1525884221
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5a8yU9
https://api.github.com/repos/huggingface/datasets/issues/comments/1525884221/reactions
2023-04-27T15:17:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1525884221
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> But now my issue is that the "first_exhausted" method breaks the loop when one of the datasets of one of the shards is empty which means that all shards stop yielding and we could be missing up to n_workers samples. I don't know if this is the behaviour expected, but I had to modify the test to accomodate this.

This looks reasonable, maybe this can be documented in the `interleave_datasets` docstring ?
```
Note for iterable datasets:

In a distributed setup or in PyTorch DataLoader workers, the stopping strategy is applied per process.
Therefore the "first_exhausted" strategy on an sharded iterable dataset can generate less samples in total (up to 1 missing sample per subdataset per worker).
```
2023-04-27T15:38:41Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1525919529
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5a868p
https://api.github.com/repos/huggingface/datasets/issues/comments/1525919529/reactions
2023-04-27T15:41:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1525919529
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006441 / 0.011353 (-0.004912) | 0.004551 / 0.011008 (-0.006457) | 0.099144 / 0.038508 (0.060636) | 0.028163 / 0.023109 (0.005054) | 0.386342 / 0.275898 (0.110444) | 0.398347 / 0.323480 (0.074867) | 0.004836 / 0.007986 (-0.003150) | 0.004724 / 0.004328 (0.000395) | 0.076277 / 0.004250 (0.072027) | 0.036305 / 0.037052 (-0.000747) | 0.377179 / 0.258489 (0.118690) | 0.410694 / 0.293841 (0.116853) | 0.030196 / 0.128546 (-0.098351) | 0.011436 / 0.075646 (-0.064211) | 0.325911 / 0.419271 (-0.093360) | 0.043709 / 0.043533 (0.000177) | 0.375801 / 0.255139 (0.120662) | 0.396511 / 0.283200 (0.113311) | 0.088346 / 0.141683 (-0.053337) | 1.483427 / 1.452155 (0.031272) | 1.553708 / 1.492716 (0.060992) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.190974 / 0.018006 (0.172968) | 0.451309 / 0.000490 (0.450819) | 0.004045 / 0.000200 (0.003845) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023814 / 0.037411 (-0.013597) | 0.096922 / 0.014526 (0.082396) | 0.101506 / 0.176557 (-0.075050) | 0.164694 / 0.737135 (-0.572441) | 0.106899 / 0.296338 (-0.189439) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.432164 / 0.215209 (0.216954) | 4.308076 / 2.077655 (2.230421) | 2.092434 / 1.504120 (0.588314) | 1.937405 / 1.541195 (0.396210) | 1.988030 / 1.468490 (0.519540) | 0.695476 / 4.584777 (-3.889301) | 3.436413 / 3.745712 (-0.309299) | 2.892954 / 5.269862 (-2.376908) | 1.519906 / 4.565676 (-3.045771) | 0.082579 / 0.424275 (-0.341696) | 0.012233 / 0.007607 (0.004626) | 0.531329 / 0.226044 (0.305284) | 5.365272 / 2.268929 (3.096344) | 2.391452 / 55.444624 (-53.053172) | 2.051116 / 6.876477 (-4.825361) | 2.140663 / 2.142072 (-0.001410) | 0.807262 / 4.805227 (-3.997966) | 0.151290 / 6.500664 (-6.349374) | 0.066137 / 0.075469 (-0.009333) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.193106 / 1.841788 (-0.648682) | 13.577240 / 8.074308 (5.502932) | 14.280126 / 10.191392 (4.088734) | 0.142538 / 0.680424 (-0.537886) | 0.016641 / 0.534201 (-0.517560) | 0.386318 / 0.579283 (-0.192965) | 0.385991 / 0.434364 (-0.048373) | 0.440712 / 0.540337 (-0.099625) | 0.524189 / 1.386936 (-0.862747) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006628 / 0.011353 (-0.004725) | 0.004664 / 0.011008 (-0.006344) | 0.077254 / 0.038508 (0.038746) | 0.028369 / 0.023109 (0.005259) | 0.343076 / 0.275898 (0.067178) | 0.376491 / 0.323480 (0.053011) | 0.005298 / 0.007986 (-0.002687) | 0.004853 / 0.004328 (0.000524) | 0.075927 / 0.004250 (0.071677) | 0.039951 / 0.037052 (0.002899) | 0.346225 / 0.258489 (0.087736) | 0.382367 / 0.293841 (0.088526) | 0.031133 / 0.128546 (-0.097413) | 0.011666 / 0.075646 (-0.063981) | 0.086383 / 0.419271 (-0.332889) | 0.042885 / 0.043533 (-0.000647) | 0.343885 / 0.255139 (0.088746) | 0.366840 / 0.283200 (0.083640) | 0.095942 / 0.141683 (-0.045741) | 1.528972 / 1.452155 (0.076817) | 1.586392 / 1.492716 (0.093676) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223952 / 0.018006 (0.205946) | 0.410767 / 0.000490 (0.410277) | 0.001014 / 0.000200 (0.000814) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024210 / 0.037411 (-0.013201) | 0.100308 / 0.014526 (0.085782) | 0.106899 / 0.176557 (-0.069658) | 0.156514 / 0.737135 (-0.580621) | 0.109548 / 0.296338 (-0.186790) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434763 / 0.215209 (0.219554) | 4.348485 / 2.077655 (2.270831) | 2.064255 / 1.504120 (0.560135) | 1.864394 / 1.541195 (0.323199) | 1.899732 / 1.468490 (0.431242) | 0.694147 / 4.584777 (-3.890630) | 3.357898 / 3.745712 (-0.387815) | 2.909155 / 5.269862 (-2.360707) | 1.424790 / 4.565676 (-3.140886) | 0.082597 / 0.424275 (-0.341678) | 0.012442 / 0.007607 (0.004835) | 0.538758 / 0.226044 (0.312713) | 5.390288 / 2.268929 (3.121359) | 2.532016 / 55.444624 (-52.912609) | 2.185724 / 6.876477 (-4.690753) | 2.274176 / 2.142072 (0.132104) | 0.804785 / 4.805227 (-4.000442) | 0.152649 / 6.500664 (-6.348015) | 0.067707 / 0.075469 (-0.007762) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.285219 / 1.841788 (-0.556568) | 13.958098 / 8.074308 (5.883790) | 14.043653 / 10.191392 (3.852261) | 0.144526 / 0.680424 (-0.535898) | 0.016813 / 0.534201 (-0.517388) | 0.390286 / 0.579283 (-0.188997) | 0.389184 / 0.434364 (-0.045180) | 0.470810 / 0.540337 (-0.069527) | 0.562391 / 1.386936 (-0.824545) |

</details>
</details>

![](https://cml.dev/watermark.png#4bb172c9772858c188f85ffc9a51f8cb1da292a0 "CML watermark")

2023-04-27T16:39:03Z
https://github.com/huggingface/datasets/pull/5735#issuecomment-1526011958
https://api.github.com/repos/huggingface/datasets/issues/5735
IC_kwDODunzps5a9Rg2
https://api.github.com/repos/huggingface/datasets/issues/comments/1526011958/reactions
2023-04-27T16:39:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1526011958
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
This PR allows sharding of merged iterable datasets.

Merged iterable datasets with for instance the `interleave_datasets` command are comprised of multiple sub-iterable, one for each dataset that has been merged.

With this PR, sharding a merged iterable will result in multiple merged datasets each comprised of sharded sub-iterable, ensuring that there is no duplication of data.

As a result it is now possible to set any amount of workers in the dataloader as long as it is lower or equal to the lowest amount of shards amongst the datasets. Before it had to be set to 0.

I previously talked about this issue on the forum [here](https://discuss.huggingface.co/t/interleaving-iterable-dataset-with-num-workers-0/35801) 