apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets
CONTRIBUTOR
I think the problem is that the underlying ex_iterable will not use iter_arrow unless the formatting type is arrow, which leads to conversion from arrow -> python -> numpy in this case rather than arrow -> numpy.

Idea of updated fix is to use the ex_iterable's iter_arrow in any case where it's available and any formatting is specified. The formatter then works directly on arrow tables; the outputs of the formatter get passed to the function to be mapped.

With updated version:

```python
import numpy as np
import time
from datasets import Dataset, Features, Array3D

features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, batched=True, batch_size=10)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
Total time: < 0.01s (~30s on main)

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, batched=False)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
Time: ~0.02 s (~30s on main)

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
Time: ~0.02s
2024-10-08T21:37:29Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2400867954
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PGlZy
https://api.github.com/repos/huggingface/datasets/issues/comments/2400867954/reactions
2024-10-09T09:09:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2400867954
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
CONTRIBUTOR
also now working for filter with similar performance improvements:

```python
filtered_examples = []
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").filter(lambda x: [arr.shape[0]==2000 for arr in x["array0"]], batch_size=10, batched=True)
t0 = time.time()
for ex in ds:
    filtered_examples.append(ex)
t1 = time.time()
assert len(filtered_examples) == 25
```
0.01s vs 50s on main


```python
filtered_examples = []
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").filter(lambda x: x["array0"].shape[0]==2000, batched=False)
t0 = time.time()
for ex in ds:
    filtered_examples.append(ex)
t1 = time.time()
assert len(filtered_examples) == 25
```
0.04s vs 50s on main

2024-10-08T22:32:28Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2400933100
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PG1Ts
https://api.github.com/repos/huggingface/datasets/issues/comments/2400933100/reactions
2024-10-08T22:38:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2400933100
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7207). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-10-09T15:21:39Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402641614
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PNWbO
https://api.github.com/repos/huggingface/datasets/issues/comments/2402641614/reactions
2024-10-09T15:21:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402641614
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
(the distributed tests failing in the CI are unrelated)
2024-10-09T15:28:28Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402657232
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PNaPQ
https://api.github.com/repos/huggingface/datasets/issues/comments/2402657232/reactions
2024-10-09T15:28:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402657232
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
There also appears to be a separate? issue with chaining filter and map bc filter iter_arrow only returns _iter_arrow if arrow formatting is applied (and vv presumably)

I don't have a good minimal example atm
2024-10-09T15:37:04Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402677258
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PNfIK
https://api.github.com/repos/huggingface/datasets/issues/comments/2402677258/reactions
2024-10-09T15:37:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402677258
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
MEMBER
> issue with chaining filter and map bc filter iter_arrow only returns _iter_arrow if arrow formatting is applied (and vv presumably)

Maybe related to this issue ?

```python
ds = Dataset.from_dict({"a": range(10)}).to_iterable_dataset()
ds = ds.with_format("arrow").map(lambda x: x, features=Features({"a": Value("string")})).with_format(None)
print(list(ds))  # yields integers instead of strings
```
2024-10-09T15:52:15Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402710825
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PNnUp
https://api.github.com/repos/huggingface/datasets/issues/comments/2402710825/reactions
2024-10-09T15:52:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402710825
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I feel like we could get rid of TypedExampleIterable altogether and apply formatting with feature conversion with `formatted_python_examples_iterator ` and `formatted_arrow_examples_iterator`

btw you can pass `features=` in `get_formatter()` to get a formatter that does the feature conversion at the same time as formatting

(edit:

except maybe the arrow formatter doesn't use `features` yet, we can fix it like this if it's really needed
```diff
class ArrowFormatter(Formatter[pa.Table, pa.Array, pa.Table]):
    def format_row(self, pa_table: pa.Table) -> pa.Table:
-         return self.simple_arrow_extractor().extract_row(pa_table)
+         pa_table = self.simple_arrow_extractor().extract_row(pa_table)
+.        return cast_table_to_features(pa_table, self.features) if self.features else pa_table
            
```


)
2024-10-09T15:56:50Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402720647
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PNpuH
https://api.github.com/repos/huggingface/datasets/issues/comments/2402720647/reactions
2024-10-09T16:06:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402720647
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> I feel like we could get rid of TypedExampleIterable altogether and apply formatting with feature conversion with formatted_python_examples_iterator  and formatted_arrow_examples_iterator

Oh nice didn't know about the feature support in get_formatter. Haven't thought through whether this works but would a FormattedExampleIterable (with feature conversion) be able to solve this and fit the API better?
2024-10-09T16:39:39Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402809328
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PN_Xw
https://api.github.com/repos/huggingface/datasets/issues/comments/2402809328/reactions
2024-10-09T16:39:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402809328
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
MEMBER
> Oh nice didn't know about the feature support in get_formatter. Haven't thought through whether this works but would a FormattedExampleIterable (with feature conversion) be able to solve this and fit the API better?

Yes this is surely the way to go actually !
2024-10-09T17:14:52Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402874322
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6POPPS
https://api.github.com/repos/huggingface/datasets/issues/comments/2402874322/reactions
2024-10-09T17:14:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402874322
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
ok i've fixed the chaining issue with my last two commits.

Will see if I can refactor into a FormattedExampleIterable

The other issue you posted seems to be unrelated (maybe something to do with feature decoding?)
2024-10-09T17:52:09Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2402938819
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6POe_D
https://api.github.com/repos/huggingface/datasets/issues/comments/2402938819/reactions
2024-10-09T17:53:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402938819
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
CONTRIBUTOR
updated with FormattedExamplesIterable.

there might be a few unnecessary format calls once the data is already formatted - doesn't seem like a big performance bottleneck but could maybe be fixed with e.g. an is_formatted property

It also might be possible to do a wider refactor and use FormattedExamplesIterable elsewhere. But I'd personally prefer not to try that rn.
2024-10-09T22:15:57Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2403523575
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PQtv3
https://api.github.com/repos/huggingface/datasets/issues/comments/2403523575/reactions
2024-10-09T22:30:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2403523575
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
CONTRIBUTOR
Thinking about this in the context of #7210 - am wondering if it would make sense for Features to define their own extraction arrow->object logic? e.g. Arrays should *always* be extracted with NumpyArrowExtractor, not only in case with_format is set to numpy (which a user can easily forget or not know to do)

2024-10-12T16:43:18Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2408622566
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PkKnm
https://api.github.com/repos/huggingface/datasets/issues/comments/2408622566/reactions
2024-10-12T17:06:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2408622566
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
MEMBER
> Thinking about this in the context of https://github.com/huggingface/datasets/issues/7210 - am wondering if it would make sense for Features to define their own extraction arrow->object logic? e.g. Arrays should always be extracted with NumpyArrowExtractor, not only in case with_format is set to numpy (which a user can easily forget or not know to do)

For `ArrayND` they already implement `to_pylist` to decode arrow data and it can be updated to return a numpy array (see the `ArrayExtensionArray` class for more details)
2024-10-14T16:56:34Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2411789307
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6PwPv7
https://api.github.com/repos/huggingface/datasets/issues/comments/2411789307/reactions
2024-10-14T16:56:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2411789307
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq im no longer sure my specific concern about with_format(None) was well-founded - I didn't appreciate that the python formatter tries to do nothing to python objects including numpy arrays, so the existing with_format(None) should I *think* do what I want. Do you think with_format(None) is ok as is after all? If so think this is hopefully ready for final review!
2024-10-25T00:19:23Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2436559827
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6ROvPT
https://api.github.com/repos/huggingface/datasets/issues/comments/2436559827/reactions
2024-10-31T11:40:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2436559827
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
CONTRIBUTOR
@lhoestq I've updated to make compatible with latest changes on main, and think the current with_format None behaviour is probably fine - please let me know if there's anything else I can do!
2024-10-31T11:41:19Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2449649611
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6SAq_L
https://api.github.com/repos/huggingface/datasets/issues/comments/2449649611/reactions
2024-10-31T11:52:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2449649611
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
MEMBER
Hi Alex, I will be less available from today and for a week. I'll review your PR and play with it once I come back if you don't mind !
2024-11-05T15:19:09Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2457451896
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6Seb14
https://api.github.com/repos/huggingface/datasets/issues/comments/2457451896/reactions
2024-11-05T15:19:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2457451896
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
thanks for the reviews and extensions, happy to see this merged :)
2025-01-14T17:39:15Z
https://github.com/huggingface/datasets/pull/7207#issuecomment-2590678548
https://api.github.com/repos/huggingface/datasets/issues/7207
IC_kwDODunzps6aap4U
https://api.github.com/repos/huggingface/datasets/issues/comments/2590678548/reactions
2025-01-14T17:39:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2590678548
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
I got to this by hacking around a bit but it seems to solve #7206 

I have no idea if this approach makes sense or would break something else? 

Could maybe work on a full pr if this looks reasonable @lhoestq ? I imagine the same issue might affect other iterable dataset methods?