Prefetching for IterableDataset
CONTRIBUTOR
Very cool! Do you have a link to the code that you're using to eagerly fetch the data? Would also be interested in hacking around something here for pre-fetching iterable datasets
2023-05-22T15:41:27Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-1557452172
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps5c1NWM
https://api.github.com/repos/huggingface/datasets/issues/comments/1557452172/reactions
2023-05-22T15:41:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1557452172
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
NONE
I ended up just switching back to the pytorch dataloader and using it's multiprocessing functionality to handle this :(. I'm just not that familiar with python multiprocessing to get something to work in jupyter (kept having weird behaviors happening with zombies living after the cell finished).
2023-05-22T16:50:32Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-1557565926
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps5c1pHm
https://api.github.com/repos/huggingface/datasets/issues/comments/1557565926/reactions
2023-05-22T16:50:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1557565926
https://avatars.githubusercontent.com/u/30946190?v=4
https://api.github.com/users/vyeevani/events{/privacy}
https://api.github.com/users/vyeevani/followers
https://api.github.com/users/vyeevani/following{/other_user}
https://api.github.com/users/vyeevani/gists{/gist_id}
https://github.com/vyeevani
vyeevani
MDQ6VXNlcjMwOTQ2MTkw
https://api.github.com/users/vyeevani/orgs
https://api.github.com/users/vyeevani/received_events
https://api.github.com/users/vyeevani/repos
https://api.github.com/users/vyeevani/starred{/owner}{/repo}
https://api.github.com/users/vyeevani/subscriptions
User
https://api.github.com/users/vyeevani
public
NONE
Ultimately settled on using webdataset to circumvent huggingface datasets entirely. Would definitely switch back if: https://github.com/huggingface/datasets/issues/5337 was resolved.
2023-05-22T18:31:28Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-1557703340
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps5c2Kqs
https://api.github.com/repos/huggingface/datasets/issues/comments/1557703340/reactions
2023-05-22T18:31:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1557703340
https://avatars.githubusercontent.com/u/30946190?v=4
https://api.github.com/users/vyeevani/events{/privacy}
https://api.github.com/users/vyeevani/followers
https://api.github.com/users/vyeevani/following{/other_user}
https://api.github.com/users/vyeevani/gists{/gist_id}
https://github.com/vyeevani
vyeevani
MDQ6VXNlcjMwOTQ2MTkw
https://api.github.com/users/vyeevani/orgs
https://api.github.com/users/vyeevani/received_events
https://api.github.com/users/vyeevani/repos
https://api.github.com/users/vyeevani/starred{/owner}{/repo}
https://api.github.com/users/vyeevani/subscriptions
User
https://api.github.com/users/vyeevani
public
COLLABORATOR
Hi! You can combine `datasets` with `torchdata` to prefetch `IterableDataset`'s samples:
```python
from datasets import load_dataset
from torchdata.datapipes.iter import IterableWrapper, HuggingFaceHubReader
from torch.utils.data import DataLoader

ds = load_dataset("sst", split="train", streaming=True)
# processing...
dp = IterableWrapper(ds)
dp = dp.prefetch(100)
dl = DataLoader(dp, batch_size=8)

i = iter(dl)
next(i)
```
2023-05-23T16:45:55Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-1559809907
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps5c-M9z
https://api.github.com/repos/huggingface/datasets/issues/comments/1559809907/reactions
2023-05-23T16:45:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1559809907
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Hey @mariosasko! Thanks for the tip here - introducing prefetch with `torchdata` didn't really give me any performance difference vs not prefetching, but the concept is definitely one that could be really beneficial. Are there any benchmarks that show the speed-up you can get with `torchdata`'s prefetch just for comparison?
2023-06-01T17:40:00Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-1572514927
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps5duqxv
https://api.github.com/repos/huggingface/datasets/issues/comments/1572514927/reactions
2023-06-01T17:40:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1572514927
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
NONE
I want to perform mapping() in advance by prefetching within the IterableDataset. Are there any recent updates?
2024-11-12T17:41:52Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-2471182333
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps6TSz_9
https://api.github.com/repos/huggingface/datasets/issues/comments/2471182333/reactions
2024-11-12T17:41:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2471182333
https://avatars.githubusercontent.com/u/14285786?v=4
https://api.github.com/users/enze5088/events{/privacy}
https://api.github.com/users/enze5088/followers
https://api.github.com/users/enze5088/following{/other_user}
https://api.github.com/users/enze5088/gists{/gist_id}
https://github.com/enze5088
enze5088
MDQ6VXNlcjE0Mjg1Nzg2
https://api.github.com/users/enze5088/orgs
https://api.github.com/users/enze5088/received_events
https://api.github.com/users/enze5088/repos
https://api.github.com/users/enze5088/starred{/owner}{/repo}
https://api.github.com/users/enze5088/subscriptions
User
https://api.github.com/users/enze5088
public
NONE
> Ultimately settled on using webdataset to circumvent huggingface datasets entirely. Would definitely switch back if: [#5337](https://github.com/huggingface/datasets/issues/5337) was resolved.

The HF dataset now supports the webdataset format. On a custom dataset with multiple .tar files that is on a single HDD. The read speed is ~180 MB/s when building the buffer.

Using the following setup, I observe ~170MB/s read speed when training with hf accelerate (multi-gpu)

```
    train_dataset = load_dataset(
        "path_to_py",
        split="train",
        streaming=True,
        trust_remote_code=True
    )
    train_dataset = train_dataset.shuffle(seed=general_config.seed, buffer_size=1000)
    train_dataset = train_dataset.with_format("torch")
    train_dataloader = DataLoader(train_dataset, num_workers=1, batch_size=8, prefetch_factor=4)
   # Then accelerator.prepare (I used dispatch_batches=False as my custom dataset does not return tuple of tensors)
```

2025-01-24T17:13:04Z
https://github.com/huggingface/datasets/issues/5878#issuecomment-2613016503
https://api.github.com/repos/huggingface/datasets/issues/5878
IC_kwDODunzps6bv3e3
https://api.github.com/repos/huggingface/datasets/issues/comments/2613016503/reactions
2025-01-24T17:13:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2613016503
https://avatars.githubusercontent.com/u/100524561?v=4
https://api.github.com/users/tonyzyl/events{/privacy}
https://api.github.com/users/tonyzyl/followers
https://api.github.com/users/tonyzyl/following{/other_user}
https://api.github.com/users/tonyzyl/gists{/gist_id}
https://github.com/tonyzyl
tonyzyl
U_kgDOBf3iEQ
https://api.github.com/users/tonyzyl/orgs
https://api.github.com/users/tonyzyl/received_events
https://api.github.com/users/tonyzyl/repos
https://api.github.com/users/tonyzyl/starred{/owner}{/repo}
https://api.github.com/users/tonyzyl/subscriptions
User
https://api.github.com/users/tonyzyl
public
### Feature request

Add support for prefetching the next n batches through iterabledataset to reduce batch loading bottleneck in training loop.

### Motivation

The primary motivation behind this is to use hardware accelerators alongside a streaming dataset. This is required when you are in a low ram or low disk space setting as well as quick iteration where you're iterating though different accelerator environments (e.x changing ec2 instances quickly to figure out batch/sec for a particular architecture). 

Currently, using the IterableDataset results in accelerators becoming basically useless due to the massive bottleneck induced by the dataset lazy loading/transform/mapping.

I've considered two alternatives:
PyTorch dataloader that handles this. However, I'm using jax, and I believe this is a piece of functionality that should live in the stream class.

Replicating the "num_workers" part of the PyTorch DataLoader to eagerly load batches and apply the transform so Arrow caching will automatically cache results and make them accessible.

### Your contribution

I may or may not have time to do this. Currently, I've written the basic multiprocessor approach to handle the eager DataLoader for my own use case with code that's not integrated to datasets. I'd definitely see this as being the default over the regular Dataset for most people given that they wouldn't have to wait on the datasets while also not worrying about performance.