Question about iterable dataset and streaming
MEMBER
> If we have already loaded the dataset, why doing to_iterable_dataset? Does it go through the dataset faster than map-style dataset?

yes, it makes a faster DataLoader for example (otherwise DataLoader uses `__getitem__` which is slower than iterating)

> load_dataset(streaming=True) is useful for huge dataset, but the speed is slow. How to make it comparable to to_iterable_dataset without loading the whole dataset into RAM?

You can aim for saturating your bandwidth using a DataLoader with num_workers and prefetch_factor. The maximum speed will be your internet bandwidth (unless your CPU is a bottlenbeck for CPU operations like image decoding).
2025-07-15T10:18:29Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3073039226
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps63Ktt6
https://api.github.com/repos/huggingface/datasets/issues/comments/3073039226/reactions
2025-07-15T10:18:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3073039226
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> > If we have already loaded the dataset, why doing to_iterable_dataset? Does it go through the dataset faster than map-style dataset?
> 
> yes, it makes a faster DataLoader for example (otherwise DataLoader uses `__getitem__` which is slower than iterating)

Okay, but `__getitem__` seems suitable for distributed settings. A distributed sampler would dispatch distinct indexes to each rank (rank0 got 0,1,2,3, rank1 got 4,5,6,7), however, if we make it `to_iterable_dataset`, then each rank needs to iterate all the samples, making it slower (i,e, rank1 got 0,1,2,3, rank2 got 0,1,2,3,(4,5,6,7))

What's your opinion here?
2025-07-15T11:19:22Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3073211438
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps63LXwu
https://api.github.com/repos/huggingface/datasets/issues/comments/3073211438/reactions
2025-07-15T11:19:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3073211438
https://avatars.githubusercontent.com/u/73541181?v=4
https://api.github.com/users/Tavish9/events{/privacy}
https://api.github.com/users/Tavish9/followers
https://api.github.com/users/Tavish9/following{/other_user}
https://api.github.com/users/Tavish9/gists{/gist_id}
https://github.com/Tavish9
Tavish9
MDQ6VXNlcjczNTQxMTgx
https://api.github.com/users/Tavish9/orgs
https://api.github.com/users/Tavish9/received_events
https://api.github.com/users/Tavish9/repos
https://api.github.com/users/Tavish9/starred{/owner}{/repo}
https://api.github.com/users/Tavish9/subscriptions
User
https://api.github.com/users/Tavish9
public
MEMBER
> however, if we make it to_iterable_dataset, then each rank needs to iterate all the samples, making it slower (i,e, rank1 got 0,1,2,3, rank2 got 0,1,2,3,(4,5,6,7))

Actually if you specify `to_iterable_dataset(num_shards=world_size)` (or a factor of world_size) and use a `torch.utils.data.DataLoader` then each rank will get a subset of the data thanks to the sharding. E.g. rank0 gets 0,1,2,3 and rank1 gets 4,5,6,7.

This is because `datasets.IterableDataset` subclasses `torch.utils.data.IterableDataset` and is aware of the current rank.
2025-07-15T13:01:55Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3073511987
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps63MhIz
https://api.github.com/repos/huggingface/datasets/issues/comments/3073511987/reactions
2025-07-15T13:02:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3073511987
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Got it, very nice features `num_shards` 👍🏻 

I would benchmark `to_iterable_dataset(num_shards=world_size)` against traditional map-style one in distributed settings in the near future.
2025-07-15T13:39:37Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3073643360
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps63NBNg
https://api.github.com/repos/huggingface/datasets/issues/comments/3073643360/reactions
2025-07-15T13:39:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3073643360
https://avatars.githubusercontent.com/u/73541181?v=4
https://api.github.com/users/Tavish9/events{/privacy}
https://api.github.com/users/Tavish9/followers
https://api.github.com/users/Tavish9/following{/other_user}
https://api.github.com/users/Tavish9/gists{/gist_id}
https://github.com/Tavish9
Tavish9
MDQ6VXNlcjczNTQxMTgx
https://api.github.com/users/Tavish9/orgs
https://api.github.com/users/Tavish9/received_events
https://api.github.com/users/Tavish9/repos
https://api.github.com/users/Tavish9/starred{/owner}{/repo}
https://api.github.com/users/Tavish9/subscriptions
User
https://api.github.com/users/Tavish9
public
NONE
Hi @lhoestq , I run a test for the speed in single node. Things are not expected as you mentioned before.

```python
import time

import datasets
from torch.utils.data import DataLoader


def time_decorator(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"Time taken: {end_time - start_time} seconds")
        return result

    return wrapper


dataset = datasets.load_dataset(
    "parquet", data_dir="my_dir", split="train"
)


@time_decorator
def load_dataset1():
    for _ in dataset:
        pass


@time_decorator
def load_dataloader1():
    for _ in DataLoader(dataset, batch_size=100, num_workers=5):
        pass


@time_decorator
def load_dataset2():
    for _ in dataset.to_iterable_dataset():
        pass


@time_decorator
def load_dataloader2():
    for _ in DataLoader(dataset.to_iterable_dataset(num_shards=5), batch_size=100, num_workers=5):
        pass


load_dataset1()
load_dataloader1()
load_dataset2()
load_dataloader2()
```
```bash
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 53192/53192 [00:00<00:00, 227103.16it/s]
Time taken: 100.36162948608398 seconds
Time taken: 70.09702134132385 seconds
Time taken: 343.09229612350464 seconds
Time taken: 132.8996012210846 seconds
```

1. Why `for _ in dataset.to_iterable_dataset()` is much slower than `for _ in dataset`
2. The `70 < 132`, the dataloader is slower when `to_iterable_dataset`
2025-07-31T12:11:00Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3139714315
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps67JD0L
https://api.github.com/repos/huggingface/datasets/issues/comments/3139714315/reactions
2025-07-31T12:11:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3139714315
https://avatars.githubusercontent.com/u/73541181?v=4
https://api.github.com/users/Tavish9/events{/privacy}
https://api.github.com/users/Tavish9/followers
https://api.github.com/users/Tavish9/following{/other_user}
https://api.github.com/users/Tavish9/gists{/gist_id}
https://github.com/Tavish9
Tavish9
MDQ6VXNlcjczNTQxMTgx
https://api.github.com/users/Tavish9/orgs
https://api.github.com/users/Tavish9/received_events
https://api.github.com/users/Tavish9/repos
https://api.github.com/users/Tavish9/starred{/owner}{/repo}
https://api.github.com/users/Tavish9/subscriptions
User
https://api.github.com/users/Tavish9
public
MEMBER
Loading in batches is faster than one example at a time. In your test the dataset is loaded in batches while the iterable_dataset is loaded one example at a time and the dataloader has a buffer to turn the examples to batches.

can you try this ?

```
batched_dataset = dataset.batch(100, num_proc=5)

@time_decorator
def load_dataloader3():
    for _ in DataLoader(batched_dataset.to_iterable_dataset(num_shards=5), batch_size=None, num_workers=5):
        pass
```
2025-07-31T13:15:23Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3139938907
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps67J6pb
https://api.github.com/repos/huggingface/datasets/issues/comments/3139938907/reactions
2025-07-31T13:15:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3139938907
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
To be fair, I test the time including batching:
```python
@time_decorator
def load_dataloader3():
    for _ in DataLoader(dataset.batch(100, num_proc=5).to_iterable_dataset(num_shards=5), batch_size=None, num_workers=5):
        pass
```

```bash
Time taken: 49.722447633743286 seconds
```
2025-07-31T13:29:45Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3139984886
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps67KF32
https://api.github.com/repos/huggingface/datasets/issues/comments/3139984886/reactions
2025-07-31T13:29:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3139984886
https://avatars.githubusercontent.com/u/73541181?v=4
https://api.github.com/users/Tavish9/events{/privacy}
https://api.github.com/users/Tavish9/followers
https://api.github.com/users/Tavish9/following{/other_user}
https://api.github.com/users/Tavish9/gists{/gist_id}
https://github.com/Tavish9
Tavish9
MDQ6VXNlcjczNTQxMTgx
https://api.github.com/users/Tavish9/orgs
https://api.github.com/users/Tavish9/received_events
https://api.github.com/users/Tavish9/repos
https://api.github.com/users/Tavish9/starred{/owner}{/repo}
https://api.github.com/users/Tavish9/subscriptions
User
https://api.github.com/users/Tavish9
public
NONE
I run another test about shuffling.

```python
@time_decorator
def load_map_dataloader1():
    for _ in DataLoader(dataset, batch_size=100, num_workers=5, shuffle=True):
        pass

@time_decorator
def load_map_dataloader2():
    for _ in DataLoader(dataset.batch(100, num_proc=5), batch_size=None, num_workers=5, shuffle=True):
        pass


@time_decorator
def load_iter_dataloader1():
    for _ in DataLoader(dataset.batch(100, num_proc=5).to_iterable_dataset(num_shards=5).shuffle(buffer_size=1000), batch_size=None, num_workers=5):
        pass

load_map_dataloader1()
load_map_dataloader2()
load_iter_dataloader1()
```

```bash
Time taken: 43.8506863117218 seconds
Time taken: 38.02591300010681 seconds
Time taken: 53.38815689086914 seconds
```


- What if I have custom collate_fn when batching?

- And if I want to shuffle the dataset, what's the correct order for `to_iterable_dataset(num_shards=x)`, `batch()` and `shuffle()`. Is `dataset.batch().to_iterable_dataset().shuffle()`? This is not faster than map-style dataset
2025-08-01T01:10:09Z
https://github.com/huggingface/datasets/issues/7680#issuecomment-3141811538
https://api.github.com/repos/huggingface/datasets/issues/7680
IC_kwDODunzps67RD1S
https://api.github.com/repos/huggingface/datasets/issues/comments/3141811538/reactions
2025-08-01T13:01:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3141811538
https://avatars.githubusercontent.com/u/73541181?v=4
https://api.github.com/users/Tavish9/events{/privacy}
https://api.github.com/users/Tavish9/followers
https://api.github.com/users/Tavish9/following{/other_user}
https://api.github.com/users/Tavish9/gists{/gist_id}
https://github.com/Tavish9
Tavish9
MDQ6VXNlcjczNTQxMTgx
https://api.github.com/users/Tavish9/orgs
https://api.github.com/users/Tavish9/received_events
https://api.github.com/users/Tavish9/repos
https://api.github.com/users/Tavish9/starred{/owner}{/repo}
https://api.github.com/users/Tavish9/subscriptions
User
https://api.github.com/users/Tavish9
public
In the doc, I found the following example: https://github.com/huggingface/datasets/blob/611f5a592359ebac6f858f515c776aa7d99838b2/docs/source/stream.mdx?plain=1#L65-L78

I am confused, 
1. If we have already loaded the dataset, why doing `to_iterable_dataset`?  Does it go through the dataset faster than map-style dataset?
2. `load_dataset(streaming=True)` is useful for huge dataset, but the speed is slow. How to make it comparable to `to_iterable_dataset` without loading the whole dataset into RAM?