Feature request: IterableDataset.push_to_hub
NONE
+1
2024-05-31T15:49:55Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2142538684
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps5_tIu8
https://api.github.com/repos/huggingface/datasets/issues/comments/2142538684/reactions
2024-05-31T15:49:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2142538684
https://avatars.githubusercontent.com/u/4803565?v=4
https://api.github.com/users/ducha-aiki/events{/privacy}
https://api.github.com/users/ducha-aiki/followers
https://api.github.com/users/ducha-aiki/following{/other_user}
https://api.github.com/users/ducha-aiki/gists{/gist_id}
https://github.com/ducha-aiki
ducha-aiki
MDQ6VXNlcjQ4MDM1NjU=
https://api.github.com/users/ducha-aiki/orgs
https://api.github.com/users/ducha-aiki/received_events
https://api.github.com/users/ducha-aiki/repos
https://api.github.com/users/ducha-aiki/starred{/owner}{/repo}
https://api.github.com/users/ducha-aiki/subscriptions
User
https://api.github.com/users/ducha-aiki
public
NONE
+1
2024-07-07T07:17:07Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2212351261
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6D3c0d
https://api.github.com/repos/huggingface/datasets/issues/comments/2212351261/reactions
2024-07-07T07:17:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2212351261
https://avatars.githubusercontent.com/u/37548991?v=4
https://api.github.com/users/phineas-pta/events{/privacy}
https://api.github.com/users/phineas-pta/followers
https://api.github.com/users/phineas-pta/following{/other_user}
https://api.github.com/users/phineas-pta/gists{/gist_id}
https://github.com/phineas-pta
phineas-pta
MDQ6VXNlcjM3NTQ4OTkx
https://api.github.com/users/phineas-pta/orgs
https://api.github.com/users/phineas-pta/received_events
https://api.github.com/users/phineas-pta/repos
https://api.github.com/users/phineas-pta/starred{/owner}{/repo}
https://api.github.com/users/phineas-pta/subscriptions
User
https://api.github.com/users/phineas-pta
public
NONE
+1, should be possible now? :) https://huggingface.co/blog/xethub-joins-hf
2024-08-20T19:56:40Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2299656899
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6JEfrD
https://api.github.com/repos/huggingface/datasets/issues/comments/2299656899/reactions
2024-08-20T19:56:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2299656899
https://avatars.githubusercontent.com/u/64205064?v=4
https://api.github.com/users/Jourdelune/events{/privacy}
https://api.github.com/users/Jourdelune/followers
https://api.github.com/users/Jourdelune/following{/other_user}
https://api.github.com/users/Jourdelune/gists{/gist_id}
https://github.com/Jourdelune
Jourdelune
MDQ6VXNlcjY0MjA1MDY0
https://api.github.com/users/Jourdelune/orgs
https://api.github.com/users/Jourdelune/received_events
https://api.github.com/users/Jourdelune/repos
https://api.github.com/users/Jourdelune/starred{/owner}{/repo}
https://api.github.com/users/Jourdelune/subscriptions
User
https://api.github.com/users/Jourdelune
public
MEMBER
Haha we're working hard to integrate Xet in the HF back-end, it will enable cool use cases :)

Anyway about `IterableDataset.push_to_hub`, I'd be happy to to provide guidance and answer questions if anyone wants to start a first simple implementation of this
2024-08-21T15:59:41Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2302441304
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6JPHdY
https://api.github.com/repos/huggingface/datasets/issues/comments/2302441304/reactions
2025-01-30T10:33:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2302441304
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
+1
2024-08-29T23:21:37Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2319397929
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6KPzQp
https://api.github.com/repos/huggingface/datasets/issues/comments/2319397929/reactions
2024-08-29T23:21:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2319397929
https://avatars.githubusercontent.com/u/90473723?v=4
https://api.github.com/users/meg-huggingface/events{/privacy}
https://api.github.com/users/meg-huggingface/followers
https://api.github.com/users/meg-huggingface/following{/other_user}
https://api.github.com/users/meg-huggingface/gists{/gist_id}
https://github.com/meg-huggingface
meg-huggingface
MDQ6VXNlcjkwNDczNzIz
https://api.github.com/users/meg-huggingface/orgs
https://api.github.com/users/meg-huggingface/received_events
https://api.github.com/users/meg-huggingface/repos
https://api.github.com/users/meg-huggingface/starred{/owner}{/repo}
https://api.github.com/users/meg-huggingface/subscriptions
User
https://api.github.com/users/meg-huggingface
public
NONE
+1
2024-11-15T13:30:41Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2478820110
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6Tv8sO
https://api.github.com/repos/huggingface/datasets/issues/comments/2478820110/reactions
2024-11-15T13:30:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2478820110
https://avatars.githubusercontent.com/u/4660138?v=4
https://api.github.com/users/pkoperek/events{/privacy}
https://api.github.com/users/pkoperek/followers
https://api.github.com/users/pkoperek/following{/other_user}
https://api.github.com/users/pkoperek/gists{/gist_id}
https://github.com/pkoperek
pkoperek
MDQ6VXNlcjQ2NjAxMzg=
https://api.github.com/users/pkoperek/orgs
https://api.github.com/users/pkoperek/received_events
https://api.github.com/users/pkoperek/repos
https://api.github.com/users/pkoperek/starred{/owner}{/repo}
https://api.github.com/users/pkoperek/subscriptions
User
https://api.github.com/users/pkoperek
public
NONE
+1
2025-01-30T05:48:21Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2623572597
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6cYIp1
https://api.github.com/repos/huggingface/datasets/issues/comments/2623572597/reactions
2025-01-30T05:48:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2623572597
https://avatars.githubusercontent.com/u/71994237?v=4
https://api.github.com/users/girivad/events{/privacy}
https://api.github.com/users/girivad/followers
https://api.github.com/users/girivad/following{/other_user}
https://api.github.com/users/girivad/gists{/gist_id}
https://github.com/girivad
girivad
MDQ6VXNlcjcxOTk0MjM3
https://api.github.com/users/girivad/orgs
https://api.github.com/users/girivad/received_events
https://api.github.com/users/girivad/repos
https://api.github.com/users/girivad/starred{/owner}{/repo}
https://api.github.com/users/girivad/subscriptions
User
https://api.github.com/users/girivad
public
NONE
+1

Currently running into this when filtering [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) for Dutch entries.

Extra points for somehow making it resumable on error. 11 TB is a lot of data to stream on a home connection without encountering any sort of errors along the way.
2025-02-13T06:33:44Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2655658201
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6eSiDZ
https://api.github.com/repos/huggingface/datasets/issues/comments/2655658201/reactions
2025-02-13T06:33:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2655658201
https://avatars.githubusercontent.com/u/8604946?v=4
https://api.github.com/users/Rijgersberg/events{/privacy}
https://api.github.com/users/Rijgersberg/followers
https://api.github.com/users/Rijgersberg/following{/other_user}
https://api.github.com/users/Rijgersberg/gists{/gist_id}
https://github.com/Rijgersberg
Rijgersberg
MDQ6VXNlcjg2MDQ5NDY=
https://api.github.com/users/Rijgersberg/orgs
https://api.github.com/users/Rijgersberg/received_events
https://api.github.com/users/Rijgersberg/repos
https://api.github.com/users/Rijgersberg/starred{/owner}{/repo}
https://api.github.com/users/Rijgersberg/subscriptions
User
https://api.github.com/users/Rijgersberg
public
MEMBER
If it can help, IterableDataset does implement `.state_dict()` and `.load_state_dict()` that you can use to resume a stream already
2025-02-13T10:49:51Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2656211544
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6eUpJY
https://api.github.com/repos/huggingface/datasets/issues/comments/2656211544/reactions
2025-02-13T10:49:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2656211544
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
+1
2025-03-12T13:20:58Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2717867746
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6h_17i
https://api.github.com/repos/huggingface/datasets/issues/comments/2717867746/reactions
2025-03-12T13:20:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2717867746
https://avatars.githubusercontent.com/u/19579420?v=4
https://api.github.com/users/tolgadur/events{/privacy}
https://api.github.com/users/tolgadur/followers
https://api.github.com/users/tolgadur/following{/other_user}
https://api.github.com/users/tolgadur/gists{/gist_id}
https://github.com/tolgadur
tolgadur
MDQ6VXNlcjE5NTc5NDIw
https://api.github.com/users/tolgadur/orgs
https://api.github.com/users/tolgadur/received_events
https://api.github.com/users/tolgadur/repos
https://api.github.com/users/tolgadur/starred{/owner}{/repo}
https://api.github.com/users/tolgadur/subscriptions
User
https://api.github.com/users/tolgadur
public
NONE
+1
2025-04-05T15:10:04Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2780814928
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6lv95Q
https://api.github.com/repos/huggingface/datasets/issues/comments/2780814928/reactions
2025-04-05T15:10:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2780814928
https://avatars.githubusercontent.com/u/21249611?v=4
https://api.github.com/users/andstor/events{/privacy}
https://api.github.com/users/andstor/followers
https://api.github.com/users/andstor/following{/other_user}
https://api.github.com/users/andstor/gists{/gist_id}
https://github.com/andstor
andstor
MDQ6VXNlcjIxMjQ5NjEx
https://api.github.com/users/andstor/orgs
https://api.github.com/users/andstor/received_events
https://api.github.com/users/andstor/repos
https://api.github.com/users/andstor/starred{/owner}{/repo}
https://api.github.com/users/andstor/subscriptions
User
https://api.github.com/users/andstor
public
NONE
+1
2025-05-15T14:45:52Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2884095540
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6r5840
https://api.github.com/repos/huggingface/datasets/issues/comments/2884095540/reactions
2025-05-15T14:45:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2884095540
https://avatars.githubusercontent.com/u/49399312?v=4
https://api.github.com/users/Peter-Devine/events{/privacy}
https://api.github.com/users/Peter-Devine/followers
https://api.github.com/users/Peter-Devine/following{/other_user}
https://api.github.com/users/Peter-Devine/gists{/gist_id}
https://github.com/Peter-Devine
Peter-Devine
MDQ6VXNlcjQ5Mzk5MzEy
https://api.github.com/users/Peter-Devine/orgs
https://api.github.com/users/Peter-Devine/received_events
https://api.github.com/users/Peter-Devine/repos
https://api.github.com/users/Peter-Devine/starred{/owner}{/repo}
https://api.github.com/users/Peter-Devine/subscriptions
User
https://api.github.com/users/Peter-Devine
public
MEMBER
Just added a first implementation for `IterableDataset.push_to_hub()` :)

I'll do a new release soon, in the meantime feel free to install `datasets` from source to try it out !
2025-06-06T16:13:21Z
https://github.com/huggingface/datasets/issues/5665#issuecomment-2949768481
https://api.github.com/repos/huggingface/datasets/issues/5665
IC_kwDODunzps6v0eUh
https://api.github.com/repos/huggingface/datasets/issues/comments/2949768481/reactions
2025-06-06T16:13:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2949768481
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

It'd be great to have a lazy push to hub, similar to the lazy loading we have with `IterableDataset`.

Suppose you'd like to filter [LAION](https://huggingface.co/datasets/laion/laion400m) based on certain conditions, but as LAION doesn't fit into your disk, you'd like to leverage streaming:
```
from datasets import load_dataset

dataset = load_dataset("laion/laion400m", streaming=True, split="train")
```
Then you could filter the dataset based on certain conditions:
```
filtered_dataset = dataset.filter(lambda example: example['HEIGHT'] > 400)
```

In order to persist this dataset and push it back to the hub, one currently needs to first load the entire filtered dataset on disk and then push:

```
from datasets import Dataset

Dataset.from_generator(filtered_dataset.__iter__).push_to_hub(...)
```
It would be great if we can instead lazy push to the data to the hub (basically stream the data to the hub), not being limited by our disk size:
```
filtered_dataset.push_to_hub("my-filtered-dataset")
```

### Motivation

This feature would be very useful for people that want to filter huge datasets without having to load the entire dataset or a filtered version thereof on their local disk.

### Your contribution

Happy to test out a PR :)