[FR] Transform Chaining, Lazy Mapping
COLLABORATOR
You can use `with_transform` to get a new dataset object.

Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex. 
2023-07-10T14:54:57Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1629137653
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hGqr1
https://api.github.com/repos/huggingface/datasets/issues/comments/1629137653/reactions
2023-07-10T14:54:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1629137653
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> You can use `with_transform` to get a new dataset object.
> 
> Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex.

I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

`with_transform` still does not chain AFAIU.
2023-07-10T14:58:24Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1629143476
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hGsG0
https://api.github.com/repos/huggingface/datasets/issues/comments/1629143476/reactions
2023-07-10T14:58:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1629143476
https://avatars.githubusercontent.com/u/36224762?v=4
https://api.github.com/users/NightMachinery/events{/privacy}
https://api.github.com/users/NightMachinery/followers
https://api.github.com/users/NightMachinery/following{/other_user}
https://api.github.com/users/NightMachinery/gists{/gist_id}
https://github.com/NightMachinery
NightMachinery
MDQ6VXNlcjM2MjI0NzYy
https://api.github.com/users/NightMachinery/orgs
https://api.github.com/users/NightMachinery/received_events
https://api.github.com/users/NightMachinery/repos
https://api.github.com/users/NightMachinery/starred{/owner}{/repo}
https://api.github.com/users/NightMachinery/subscriptions
User
https://api.github.com/users/NightMachinery
public
COLLABORATOR
> I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

You must cache an `IterableDataset` to disk to load it as a `Dataset`. One way to do this is with `Dataset.from_generator`:
```python
from functools import partial
from datasets import Dataset

def gen_from_iterable_dataset(iterable_ds)
    yield from iterable_ds

ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
```

> with_transform still does not chain AFAIU.

Yes, not supported yet - the solution is to combine the transforms into a single one.
2023-07-10T15:37:25Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1629212065
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hG82h
https://api.github.com/repos/huggingface/datasets/issues/comments/1629212065/reactions
2023-07-10T15:37:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1629212065
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
I wonder if it would be beneficial to have a dedicated method to do that ? Maybe a `.save_to_disk()` so that the user can reload the resulting dataset later ?
2023-07-11T16:19:57Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1631120611
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hOOzj
https://api.github.com/repos/huggingface/datasets/issues/comments/1631120611/reactions
2023-07-11T16:19:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1631120611
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> ```python
> from functools import partial
> from datasets import Dataset
> 
> def gen_from_iterable_dataset(iterable_ds)
>     yield from iterable_ds
> 
> ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
> ```

@mariosasko With these complex mapping functions, what hash will be used to cache this dataset?

2023-07-13T19:52:20Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1634820043
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hcV_L
https://api.github.com/repos/huggingface/datasets/issues/comments/1634820043/reactions
2023-07-13T19:52:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1634820043
https://avatars.githubusercontent.com/u/36224762?v=4
https://api.github.com/users/NightMachinery/events{/privacy}
https://api.github.com/users/NightMachinery/followers
https://api.github.com/users/NightMachinery/following{/other_user}
https://api.github.com/users/NightMachinery/gists{/gist_id}
https://github.com/NightMachinery
NightMachinery
MDQ6VXNlcjM2MjI0NzYy
https://api.github.com/users/NightMachinery/orgs
https://api.github.com/users/NightMachinery/received_events
https://api.github.com/users/NightMachinery/repos
https://api.github.com/users/NightMachinery/starred{/owner}{/repo}
https://api.github.com/users/NightMachinery/subscriptions
User
https://api.github.com/users/NightMachinery
public
COLLABORATOR
The params passed to `Dataset.from_generator` will be used to compute the hash (`partial` encapsulates the `iterable_ds` value, so changing it will also change the hash)
2023-07-14T13:12:39Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1635846752
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5hgQpg
https://api.github.com/repos/huggingface/datasets/issues/comments/1635846752/reactions
2023-07-14T13:12:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1635846752
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Hi, I think this feature would be very useful. I want to concatenate large datasets with heterogeneous columns. I dislike `map` since I don't want multiple copy of that datasets locally. I tried to use "set_transform" on each dataset to convert it to a standard features format, but `datasets.concatenate_datasets` ignores the updated format of the datasets.  A work around is to use `torch.utils.data.ConcatDataset`. Is there a neat way to do it using HF datasets?﻿
2023-11-23T10:08:57Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-1824116010
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps5suc0q
https://api.github.com/repos/huggingface/datasets/issues/comments/1824116010/reactions
2023-11-23T10:08:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1824116010
https://avatars.githubusercontent.com/u/51880718?v=4
https://api.github.com/users/kkoutini/events{/privacy}
https://api.github.com/users/kkoutini/followers
https://api.github.com/users/kkoutini/following{/other_user}
https://api.github.com/users/kkoutini/gists{/gist_id}
https://github.com/kkoutini
kkoutini
MDQ6VXNlcjUxODgwNzE4
https://api.github.com/users/kkoutini/orgs
https://api.github.com/users/kkoutini/received_events
https://api.github.com/users/kkoutini/repos
https://api.github.com/users/kkoutini/starred{/owner}{/repo}
https://api.github.com/users/kkoutini/subscriptions
User
https://api.github.com/users/kkoutini
public
NONE
@mariosasko These features would be handy for large datasets. A typical use case is video datasets: We have millions of videos, each stored in some OSS so they require some custom loading logic.

1) Due to the memory limit, loading the videos a priori into the memory is infeasible. But we can postpone video loading until they are needed with lazy mapping.
2) With chained transforms, we can allow the users to specify their custom video preprocessing logic while keeping the loading logic the same.
2025-01-18T17:31:20Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-2599801126
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps6a9dEm
https://api.github.com/repos/huggingface/datasets/issues/comments/2599801126/reactions
2025-01-18T17:32:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2599801126
https://avatars.githubusercontent.com/u/22806979?v=4
https://api.github.com/users/luowyang/events{/privacy}
https://api.github.com/users/luowyang/followers
https://api.github.com/users/luowyang/following{/other_user}
https://api.github.com/users/luowyang/gists{/gist_id}
https://github.com/luowyang
luowyang
MDQ6VXNlcjIyODA2OTc5
https://api.github.com/users/luowyang/orgs
https://api.github.com/users/luowyang/received_events
https://api.github.com/users/luowyang/repos
https://api.github.com/users/luowyang/starred{/owner}{/repo}
https://api.github.com/users/luowyang/subscriptions
User
https://api.github.com/users/luowyang
public
MEMBER
FYI lazy map is available for `IterableDataset`(map is applied on-the-fly when iterating on the dataset):

```python
ds = load_dataset(...streaming=True)
# or
ds = Dataset.from_list(...).to_iterable_dataset()
# or
ds = IterableDataset.from_generator(...)

# Then you can chain many map/filter/shuffle/etc.
ds = ds.map(...).filter(...).map(...)

# The map functions are applied on-the-fly when iterating on the dataset
for example in ds:
    ...
2025-01-20T14:06:27Z
https://github.com/huggingface/datasets/issues/6012#issuecomment-2602516344
https://api.github.com/repos/huggingface/datasets/issues/6012
IC_kwDODunzps6bHz94
https://api.github.com/repos/huggingface/datasets/issues/comments/2602516344/reactions
2025-01-20T14:06:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2602516344
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

Currently using a `map` call processes and duplicates the whole dataset, which takes both time and disk space.

The solution is to allow lazy mapping, which is essentially a saved chain of transforms that are applied on the fly whenever a slice of the dataset is requested.

The API should look like `map`, as `set_transform` changes the current dataset while `map` returns another dataset.

### Motivation

Lazy processing allows lower disk usage and faster experimentation.

### Your contribution

_