Load a cached dataset as iterable
NONE
Can I work on this issue? I am pretty new to this.
2023-01-29T20:45:18Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1407765289
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5T6Msp
https://api.github.com/repos/huggingface/datasets/issues/comments/1407765289/reactions
2023-01-29T20:45:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1407765289
https://avatars.githubusercontent.com/u/52814518?v=4
https://api.github.com/users/jalajk24/events{/privacy}
https://api.github.com/users/jalajk24/followers
https://api.github.com/users/jalajk24/following{/other_user}
https://api.github.com/users/jalajk24/gists{/gist_id}
https://github.com/jalajk24
jalajk24
MDQ6VXNlcjUyODE0NTE4
https://api.github.com/users/jalajk24/orgs
https://api.github.com/users/jalajk24/received_events
https://api.github.com/users/jalajk24/repos
https://api.github.com/users/jalajk24/starred{/owner}{/repo}
https://api.github.com/users/jalajk24/subscriptions
User
https://api.github.com/users/jalajk24
public
MEMBER
Hi ! Sure :) you can comment `#self-assign` to assign yourself to this issue.

I can give you some pointers to get started:

`load_dataset` works roughly this way:
1. it instantiate a dataset builder using `load_dataset_builder()`
2. the builder download and prepare the dataset as Arrow files in the cache using `download_and_prepare()`
3. the builder returns a Dataset object with `as_dataset()`

One way to approach this would be to implement `as_iterable_dataset()` in `builder.py`.

And similarly to `as_dataset()`, you can use the `ArrowReader`. It has a `get_file_instructions()` method that can be helpful. It gives you the files to read as list of dictionaries with those keys: `filename`, `skip` and `take`.

The `skip` and `take` arguments are used in case the user wants to load a subset of the dataset, e.g.
```python
load_dataset(..., split="train[:10]")
```

Let me know if you have questions or if I can help :)
2023-01-30T11:07:37Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1408423119
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5T8tTP
https://api.github.com/repos/huggingface/datasets/issues/comments/1408423119/reactions
2023-01-30T11:07:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1408423119
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
This use-case is a bit specific, and `load_dataset` already has enough parameters (plus, `streaming=True` also returns an iterable dataset, so we would have to explain the difference), so I think it would be better to add `IterableDataset.from_file` to the API (more flexible and aligned with the goal from https://github.com/huggingface/datasets/issues/3444) instead.
2023-01-30T13:07:30Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1408602106
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5T9Y_6
https://api.github.com/repos/huggingface/datasets/issues/comments/1408602106/reactions
2023-01-30T13:07:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1408602106
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
> This use-case is a bit specific

This allows to use `datasets` for large scale training where map-style datasets are too slow and use too much memory in PyTorch. So I would still consider adding it.

Alternatively we could add this feature one level bellow:
```python
builder = load_dataset_builder(...)
builder.download_and_prepare()
ids = builder.as_iterable_dataset()
```
2023-01-31T10:53:47Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1410146215
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UDR-n
https://api.github.com/repos/huggingface/datasets/issues/comments/1410146215/reactions
2023-01-31T10:53:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1410146215
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Yes, I see how this can be useful. Still, I think `Dataset.to_iterable` + `IterableDataset.from_file` would be much cleaner in terms of the API design (and more flexible since `load_dataset` can only access the "initial" (unprocessed) version of a dataset).

And since it can be tricky to manually find the "initial" version of a dataset in the cache, maybe `load_dataset` could return an iterable dataset streamed from the cache if `streaming=True` and the cache is up-to-date. 
2023-01-31T18:44:41Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1410892066
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UGIEi
https://api.github.com/repos/huggingface/datasets/issues/comments/1410892066/reactions
2023-01-31T18:44:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1410892066
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> This allows to use datasets for large scale training where map-style datasets are too slow and use too much memory in PyTorch.

I second that. e.g. In my last experiment Oscar-en uses 16GB RSS RAM per process and when using multiple processes the host quickly runs out cpu memory. 
2023-01-31T18:51:05Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1410898800
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UGJtw
https://api.github.com/repos/huggingface/datasets/issues/comments/1410898800/reactions
2023-01-31T18:51:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1410898800
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
CONTRIBUTOR
>And since it can be tricky to manually find the "initial" version of a dataset in the cache, maybe load_dataset could return an iterable dataset streamed from the cache if streaming=True and the cache is up-to-date.

This is exactly the need on JeanZay (HPC) - I have the dataset cache ready, but the compute node is offline, so making streaming work off a local cache would address that need.

If you will have a working POC I can be the tester. 
2023-01-31T18:53:25Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1410901550
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UGKYu
https://api.github.com/repos/huggingface/datasets/issues/comments/1410901550/reactions
2023-01-31T18:54:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1410901550
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
MEMBER
> Yes, I see how this can be useful. Still, I think Dataset.to_iterable + IterableDataset.from_file would be much cleaner in terms of the API design (and more flexible since load_dataset can only access the "initial" (unprocessed) version of a dataset).

I like `IterableDataset.from_file` as well. On the other hand `Dataset.to_iterable` first requires to load a Dataset object, which can take time depending on your hardware and your dataset size (sometimes 1h+).

> And since it can be tricky to manually find the "initial" version of a dataset in the cache, maybe load_dataset could return an iterable dataset streamed from the cache if streaming=True and the cache is up-to-date.

That would definitely do the job. I was suggesting a different parameter just to make explicit the difference between
- streaming from the raw data
- streaming from the local cache

But I'd be fine with streaming from cache is the cache is up-to-date since it's always faster. We could log a message as usual to make it explicit that the cache is used
2023-02-01T10:41:51Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1411845228
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UJwxs
https://api.github.com/repos/huggingface/datasets/issues/comments/1411845228/reactions
2023-02-01T10:41:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1411845228
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
> I was suggesting a different parameter just to make explicit the difference between

MosaicML's `streaming` library does the same (tries to stream from the local cache if possible), so logging a message should be explicit enough :).
2023-02-01T12:36:47Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1411989605
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UKUBl
https://api.github.com/repos/huggingface/datasets/issues/comments/1411989605/reactions
2023-02-01T12:36:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1411989605
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Ok ! Sounds good then :)
2023-02-01T12:53:17Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1412009252
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UKY0k
https://api.github.com/repos/huggingface/datasets/issues/comments/1412009252/reactions
2023-02-01T12:53:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1412009252
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi Both! It has been a while since my first issue so I am gonna go for this one ! #self-assign
2023-02-07T15:56:41Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1421008091
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5Ustzb
https://api.github.com/repos/huggingface/datasets/issues/comments/1421008091/reactions
2023-02-07T15:56:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1421008091
https://avatars.githubusercontent.com/u/56002455?v=4
https://api.github.com/users/hamid-vakilzadeh/events{/privacy}
https://api.github.com/users/hamid-vakilzadeh/followers
https://api.github.com/users/hamid-vakilzadeh/following{/other_user}
https://api.github.com/users/hamid-vakilzadeh/gists{/gist_id}
https://github.com/hamid-vakilzadeh
hamid-vakilzadeh
MDQ6VXNlcjU2MDAyNDU1
https://api.github.com/users/hamid-vakilzadeh/orgs
https://api.github.com/users/hamid-vakilzadeh/received_events
https://api.github.com/users/hamid-vakilzadeh/repos
https://api.github.com/users/hamid-vakilzadeh/starred{/owner}{/repo}
https://api.github.com/users/hamid-vakilzadeh/subscriptions
User
https://api.github.com/users/hamid-vakilzadeh
public
CONTRIBUTOR
#self-assign
2023-02-07T15:58:03Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1421010005
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5UsuRV
https://api.github.com/repos/huggingface/datasets/issues/comments/1421010005/reactions
2023-02-07T15:58:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1421010005
https://avatars.githubusercontent.com/u/56002455?v=4
https://api.github.com/users/hamid-vakilzadeh/events{/privacy}
https://api.github.com/users/hamid-vakilzadeh/followers
https://api.github.com/users/hamid-vakilzadeh/following{/other_user}
https://api.github.com/users/hamid-vakilzadeh/gists{/gist_id}
https://github.com/hamid-vakilzadeh
hamid-vakilzadeh
MDQ6VXNlcjU2MDAyNDU1
https://api.github.com/users/hamid-vakilzadeh/orgs
https://api.github.com/users/hamid-vakilzadeh/received_events
https://api.github.com/users/hamid-vakilzadeh/repos
https://api.github.com/users/hamid-vakilzadeh/starred{/owner}{/repo}
https://api.github.com/users/hamid-vakilzadeh/subscriptions
User
https://api.github.com/users/hamid-vakilzadeh
public
CONTRIBUTOR
I like idea of `IterableDataset.from_file`. 
2023-05-12T22:18:55Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1546368568
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5cK7Y4
https://api.github.com/repos/huggingface/datasets/issues/comments/1546368568/reactions
2023-05-12T22:18:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1546368568
https://avatars.githubusercontent.com/u/10278877?v=4
https://api.github.com/users/mariusz-jachimowicz-83/events{/privacy}
https://api.github.com/users/mariusz-jachimowicz-83/followers
https://api.github.com/users/mariusz-jachimowicz-83/following{/other_user}
https://api.github.com/users/mariusz-jachimowicz-83/gists{/gist_id}
https://github.com/mariusz-jachimowicz-83
mariusz-jachimowicz-83
MDQ6VXNlcjEwMjc4ODc3
https://api.github.com/users/mariusz-jachimowicz-83/orgs
https://api.github.com/users/mariusz-jachimowicz-83/received_events
https://api.github.com/users/mariusz-jachimowicz-83/repos
https://api.github.com/users/mariusz-jachimowicz-83/starred{/owner}{/repo}
https://api.github.com/users/mariusz-jachimowicz-83/subscriptions
User
https://api.github.com/users/mariusz-jachimowicz-83
public
MEMBER
https://github.com/huggingface/datasets/pull/5821 should be helpful to implement `IterableDataset.from_file`, since it defines a new ArrowExamplesIterable that takes an Arrow tables generator function (e.g. from a file) and can be used in an IterableDataset
2023-05-15T09:38:53Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1547519941
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5cPUfF
https://api.github.com/repos/huggingface/datasets/issues/comments/1547519941/reactions
2023-05-15T09:39:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1547519941
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq I have just started working on this issue. 
2023-05-15T19:21:06Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1548437561
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5cS0g5
https://api.github.com/repos/huggingface/datasets/issues/comments/1548437561/reactions
2023-05-15T19:21:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548437561
https://avatars.githubusercontent.com/u/10278877?v=4
https://api.github.com/users/mariusz-jachimowicz-83/events{/privacy}
https://api.github.com/users/mariusz-jachimowicz-83/followers
https://api.github.com/users/mariusz-jachimowicz-83/following{/other_user}
https://api.github.com/users/mariusz-jachimowicz-83/gists{/gist_id}
https://github.com/mariusz-jachimowicz-83
mariusz-jachimowicz-83
MDQ6VXNlcjEwMjc4ODc3
https://api.github.com/users/mariusz-jachimowicz-83/orgs
https://api.github.com/users/mariusz-jachimowicz-83/received_events
https://api.github.com/users/mariusz-jachimowicz-83/repos
https://api.github.com/users/mariusz-jachimowicz-83/starred{/owner}{/repo}
https://api.github.com/users/mariusz-jachimowicz-83/subscriptions
User
https://api.github.com/users/mariusz-jachimowicz-83
public
CONTRIBUTOR
@lhoestq Thank you for taking over.
2023-05-15T19:28:11Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1548449767
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5cS3fn
https://api.github.com/repos/huggingface/datasets/issues/comments/1548449767/reactions
2023-05-15T19:28:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548449767
https://avatars.githubusercontent.com/u/56002455?v=4
https://api.github.com/users/hamid-vakilzadeh/events{/privacy}
https://api.github.com/users/hamid-vakilzadeh/followers
https://api.github.com/users/hamid-vakilzadeh/following{/other_user}
https://api.github.com/users/hamid-vakilzadeh/gists{/gist_id}
https://github.com/hamid-vakilzadeh
hamid-vakilzadeh
MDQ6VXNlcjU2MDAyNDU1
https://api.github.com/users/hamid-vakilzadeh/orgs
https://api.github.com/users/hamid-vakilzadeh/received_events
https://api.github.com/users/hamid-vakilzadeh/repos
https://api.github.com/users/hamid-vakilzadeh/starred{/owner}{/repo}
https://api.github.com/users/hamid-vakilzadeh/subscriptions
User
https://api.github.com/users/hamid-vakilzadeh
public
CONTRIBUTOR
So what's recommanded usage of `IterableDataset.from_file` and `load_dataset`? How about I have multiple arrow files and `load_dataset` is often convenient to handle that.
2023-06-23T14:01:43Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1604329246
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5foB8e
https://api.github.com/repos/huggingface/datasets/issues/comments/1604329246/reactions
2023-06-23T14:15:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1604329246
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
MEMBER
If you have multiple Arrow files you can load them using

```python
from datasets import load_dataset

data_files = {"train": ["path/to/0.arrow", "path/to/1.arrow", ..., "path/to/n.arrow"]}

ds = load_dataset("arrow", data_files=data_files, streaming=True)
```

This is equivalent to calling `IterableDataset.from_file` and `concatenate_datasets`.
2023-06-26T10:48:52Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-1607200282
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps5fy-4a
https://api.github.com/repos/huggingface/datasets/issues/comments/1607200282/reactions
2023-06-26T10:48:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1607200282
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi! 👋 I’d love to help with this feature and was wondering if @mariusz-jachimowicz-83 is still working on it. If not, I’d be happy to pick it up and continue. Let me know! 🙌
2025-06-18T19:21:24Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-2985442826
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps6x8j4K
https://api.github.com/repos/huggingface/datasets/issues/comments/2985442826/reactions
2025-06-18T19:21:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2985442826
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
MEMBER
I don't think anyone is working on this at the moment, imo the simplest would be to do it one level below

```python
builder = load_dataset_builder(...)
builder.download_and_prepare()
ids = builder.as_iterable_dataset()
```
2025-06-19T15:00:27Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-2988421455
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps6yH7FP
https://api.github.com/repos/huggingface/datasets/issues/comments/2988421455/reactions
2025-06-19T15:00:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2988421455
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for the clarification @lhoestq 🙌 Will share a draft soon
2025-06-19T18:44:47Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-2988887014
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps6yJsvm
https://api.github.com/repos/huggingface/datasets/issues/comments/2988887014/reactions
2025-06-19T18:44:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2988887014
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
CONTRIBUTOR
Follow-up test added in #7629. for as_iterable_dataset() method.
I’ve added the unit test in a separate PR to keep this one focused on the feature implementation, as the test is optional and can be reviewed independently.
2025-06-19T19:24:57Z
https://github.com/huggingface/datasets/issues/5481#issuecomment-2988969900
https://api.github.com/repos/huggingface/datasets/issues/5481
IC_kwDODunzps6yKA-s
https://api.github.com/repos/huggingface/datasets/issues/comments/2988969900/reactions
2025-06-19T19:30:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2988969900
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
The idea would be to allow something like 

```python 
ds = load_dataset("c4", "en", as_iterable=True)
```

To be used to train models. It would load an IterableDataset from the cached Arrow files.

Cc @stas00 

Edit : from the discussions we may load from cache when streaming=True