Align the Dataset and IterableDataset processing API
MEMBER
Yes I agree, these should be as aligned as possible. Maybe we can also check the feedback in the survey at http://hf.co/oss-survey and see if people mentioned related things on the API (in particular if we go the breaking change way, it would be good to be sure we are taking the right direction for the community).
2021-12-16T11:33:19Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-995709018
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps47WVBa
https://api.github.com/repos/huggingface/datasets/issues/comments/995709018/reactions
2021-12-16T11:33:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/995709018
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
COLLABORATOR
I like this proposal.

> There is also an important difference in terms of behavior:
Dataset.map adds new columns (with dict.update)
BUT
IterableDataset discards previous columns (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.

> The main breaking change would be the change of behavior of IterableDataset.map, because currently it discards all the previous columns instead of keeping them.

Yes, this behavior of `IterableDataset.map` was surprising to me the first time I used it because I was expecting the same behavior as `Dataset.map`, so I'm OK with the breaking change here.

> IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs

\+  it's also missing the actual formatting code (we return unformatted tensors)
> We could have a completely aligned map method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

> For information, TFDS does lazy map by default, and has an additional .cache() method.

If I understand this part correctly, the idea would be for `Dataset.map` to behave similarly to `Dataset.with_transform` (lazy processing) and to have an option to cache processed data (with `.cache()`). This idea is really nice because it can also be applied to `IterableDataset` to fix https://github.com/huggingface/datasets/issues/3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?) 
> If the two APIs are more aligned it would be awesome for the examples in transformers, and it would create a satisfactory experience for users that want to switch from one mode to the other.

Yes, it would be amazing to have an option to easily switch between these two modes.

I agree with the rest.

2021-12-16T13:52:18Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-995837012
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps47W0RU
https://api.github.com/repos/huggingface/datasets/issues/comments/995837012/reactions
2021-12-16T13:52:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/995837012
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
> If I understand this part correctly, the idea would be for Dataset.map to behave similarly to Dataset.with_transform (lazy processing) and to have an option to cache processed data (with .cache()). This idea is really nice because it can also be applied to IterableDataset to fix #3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?)

Yea this is too big of a change in my opinion. Anyway it's fine as it is right now with streaming=lazy and regular=eager.
2021-12-16T14:27:10Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-995867469
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps47W7tN
https://api.github.com/repos/huggingface/datasets/issues/comments/995867469/reactions
2021-12-16T14:27:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/995867469
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, IterableDataset is also missing set_format.
2022-03-20T08:44:58Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-1073200356
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps4_97zk
https://api.github.com/repos/huggingface/datasets/issues/comments/1073200356/reactions
2022-03-20T08:44:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1073200356
https://avatars.githubusercontent.com/u/13161779?v=4
https://api.github.com/users/tshu-w/events{/privacy}
https://api.github.com/users/tshu-w/followers
https://api.github.com/users/tshu-w/following{/other_user}
https://api.github.com/users/tshu-w/gists{/gist_id}
https://github.com/tshu-w
tshu-w
MDQ6VXNlcjEzMTYxNzc5
https://api.github.com/users/tshu-w/orgs
https://api.github.com/users/tshu-w/received_events
https://api.github.com/users/tshu-w/repos
https://api.github.com/users/tshu-w/starred{/owner}{/repo}
https://api.github.com/users/tshu-w/subscriptions
User
https://api.github.com/users/tshu-w
public
MEMBER
Yes indeed, thanks. I added it to the list of methods to align in the first post
2022-03-21T14:45:21Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-1073989338
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps5AA8ba
https://api.github.com/repos/huggingface/datasets/issues/comments/1073989338/reactions
2022-03-21T14:45:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1073989338
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I just encountered the problem of the missing `fn_kwargs` parameter in the `map` method. I am commenting to give a workaround in case someone has the same problem and does not find a solution.
You can wrap your function call inside a class that contains the other parameters needed by the function called by map, like this:

```python
def my_func(x, y, z):
    # Do things

class MyFuncWrapper:
    def __init__(self, y, z):
        self.y = y
        self.z = z

    def __call__(self, x):
        return my_func(x, self.y, self.z)
```

Then, give an instance of the `MyFuncWrapper` to the map function.
2022-06-16T14:59:10Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-1157760276
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps5FAgUU
https://api.github.com/repos/huggingface/datasets/issues/comments/1157760276/reactions
2022-06-16T14:59:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1157760276
https://avatars.githubusercontent.com/u/2855911?v=4
https://api.github.com/users/gcaillaut/events{/privacy}
https://api.github.com/users/gcaillaut/followers
https://api.github.com/users/gcaillaut/following{/other_user}
https://api.github.com/users/gcaillaut/gists{/gist_id}
https://github.com/gcaillaut
gcaillaut
MDQ6VXNlcjI4NTU5MTE=
https://api.github.com/users/gcaillaut/orgs
https://api.github.com/users/gcaillaut/received_events
https://api.github.com/users/gcaillaut/repos
https://api.github.com/users/gcaillaut/starred{/owner}{/repo}
https://api.github.com/users/gcaillaut/subscriptions
User
https://api.github.com/users/gcaillaut
public
CONTRIBUTOR
Any update on this? It's almost 2024😂 @lhoestq 
2023-08-12T08:21:59Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-1675786253
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps5j4ngN
https://api.github.com/repos/huggingface/datasets/issues/comments/1675786253/reactions
2023-08-16T04:40:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1675786253
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
MEMBER
The main differences have been addressed (map, formatting) but there are still a few things to implement like Dataset.take, Dataset.skip, IterableDataset.set_format, IterableDataset.formatted_as, IterableDataset.reset_format.

The rest cannot be implemented for the general case. E.g. train_test_split and select can only work on an iterable dataset if the underlying dataset format allows it (we need to know the number of rows and have some sort of random access)
2023-08-16T09:28:17Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-1680269675
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps5kJuFr
https://api.github.com/repos/huggingface/datasets/issues/comments/1680269675/reactions
2023-08-16T09:28:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1680269675
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It appears `IterableDataset` now supports all the formats apart from `pandas` but the documentation doesn't have any mention of it yet. The docstring of `with_format` seems like it's even older incorrectly saying it only supports `arrow`. Are there any plans to update the documentation and have some guides on best practices?
2024-10-08T11:51:19Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-2399630548
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps6PB3TU
https://api.github.com/repos/huggingface/datasets/issues/comments/2399630548/reactions
2024-10-08T11:51:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2399630548
https://avatars.githubusercontent.com/u/44096034?v=4
https://api.github.com/users/amin-nejad/events{/privacy}
https://api.github.com/users/amin-nejad/followers
https://api.github.com/users/amin-nejad/following{/other_user}
https://api.github.com/users/amin-nejad/gists{/gist_id}
https://github.com/amin-nejad
amin-nejad
MDQ6VXNlcjQ0MDk2MDM0
https://api.github.com/users/amin-nejad/orgs
https://api.github.com/users/amin-nejad/received_events
https://api.github.com/users/amin-nejad/repos
https://api.github.com/users/amin-nejad/starred{/owner}{/repo}
https://api.github.com/users/amin-nejad/subscriptions
User
https://api.github.com/users/amin-nejad
public
MEMBER
Thanks, I updated the docstrings. Would be cool to have more examples in the docs though, if this is something you'd like to contribute ;)
2024-10-08T13:06:19Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-2399796486
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps6PCf0G
https://api.github.com/repos/huggingface/datasets/issues/comments/2399796486/reactions
2024-10-08T13:06:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2399796486
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Now both `Dataset` and `IterableDataset` support all formats including pandas, arrow, polars, torch, tf, numpy, jax :)

```python
for df in ds.with_format("pandas").iter(batch_size=100):
    ...
```
will do a new release soon
2025-01-31T11:07:06Z
https://github.com/huggingface/datasets/issues/3444#issuecomment-2626945570
https://api.github.com/repos/huggingface/datasets/issues/3444
IC_kwDODunzps6clAIi
https://api.github.com/repos/huggingface/datasets/issues/comments/2626945570/reactions
2025-01-31T11:07:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2626945570
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Intro

items marked like <s>this</s> are done already :)

Currently the two classes have two distinct API for processing:

### The `.map()` method

Both have those parameters in common: function, batched, batch_size

- IterableDataset is missing those parameters:
<s>with_indices</s>, with_rank, <s>input_columns</s>, <s>drop_last_batch</s>, <s>remove_columns</s>, features, disable_nullable, fn_kwargs, num_proc

- Dataset also has additional parameters that are exclusive, due to caching:
keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, suffix_template, new_fingerprint

- <s>There is also an important difference in terms of behavior:
**Dataset.map adds new columns** (with dict.update)
BUT
**IterableDataset discards previous columns** (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.</s>

- Dataset.map is eager while IterableDataset.map is lazy

### The `.shuffle()` method

- <s>Both have an optional seed parameter, but IterableDataset requires a mandatory parameter buffer_size to control the size of the local buffer used for approximate shuffling.</s>

- <s>IterableDataset is missing the parameter generator</s>

- Also Dataset has exclusive parameters due to caching: keep_in_memory, load_from_cache_file, indices_cache_file_name, writer_batch_size, new_fingerprint

### The `.with_format()` method

- <s>IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow)</s> and is missing the parameters: columns, output_all_columns and format_kwargs
- other methods like `set_format`, `reset_format` or `formatted_as` are also missing

### Other methods

- Both have the same `remove_columns` method
- IterableDataset is missing: <s>cast</s>, <s>cast_column</s>, <s>filter</s>, <s>rename_column</s>, <s>rename_columns</s>, class_encode_column, flatten, train_test_split, <s>shard</s>
- Some other methods are missing but we can discuss them: set_transform, formatted_as, with_transform
- And others don't really make sense for an iterable dataset: select, sort, <s>add_column</s>, add_item
- Dataset is missing skip and take, that IterableDataset implements.

## Questions

I think it would be nice to be able to switch between streaming and regular dataset easily, without changing the processing code significantly.

1. What should be aligned and what shouldn't between those two APIs ?

IMO the minimum is to align the main processing methods.

It would mean aligning breaking the current `Iterable.map` to have the same behavior as `Dataset.map` (add columns with dict.update), and add multiprocessing as well as the missing parameters. DONE ✅

It would also mean implementing the missing methods: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard. WIP 🟠

2. What are the breaking changes for IterableDataset ?

The main breaking change would be the change of behavior of `IterableDataset.map`, because currently it discards all the previous columns instead of keeping them. DONE ✅

3. Shall we also do some changes for regular datasets ?

I agree the simplest would be to have the exact same methods for both Dataset and IterableDataset. However this is probably not a good idea because it would prevent users from using the best benefits of them. That's why we can keep some aspects of regular datasets as they are:
- keep the eager Dataset.map with caching
- keep the with_transform method for lazy processing
- keep Dataset.select (it could also be added to IterableDataset even though it's not recommended)

We could have a completely aligned `map` method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

For information, TFDS does lazy map by default, and has an additional `.cache()` method.

## Opinions ?

I'd love to gather some opinions about this here. If the two APIs are more aligned it would be awesome for the examples in `transformers`, and it would create a satisfactory experience for users that want to switch from one mode to the other.

cc @mariosasko @albertvillanova @thomwolf @patrickvonplaten @sgugger 