Support `fsspec` in `Dataset.to_<format>` methods
MEMBER
Hi @mariosasko unless someone's already working on it, I guess I can tackle it!
2023-07-28T09:07:01Z
https://github.com/huggingface/datasets/issues/6086#issuecomment-1655333818
https://api.github.com/repos/huggingface/datasets/issues/6086
IC_kwDODunzps5iqmO6
https://api.github.com/repos/huggingface/datasets/issues/comments/1655333818/reactions
2023-07-28T09:07:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1655333818
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
COLLABORATOR
Hi! Sure, feel free to tackle this.
2023-07-28T09:52:20Z
https://github.com/huggingface/datasets/issues/6086#issuecomment-1655405883
https://api.github.com/repos/huggingface/datasets/issues/6086
IC_kwDODunzps5iq307
https://api.github.com/repos/huggingface/datasets/issues/comments/1655405883/reactions
2023-07-28T09:52:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1655405883
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
#self-assign
2023-07-28T14:16:17Z
https://github.com/huggingface/datasets/issues/6086#issuecomment-1655762698
https://api.github.com/repos/huggingface/datasets/issues/6086
IC_kwDODunzps5isO8K
https://api.github.com/repos/huggingface/datasets/issues/comments/1655762698/reactions
2023-07-28T14:16:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1655762698
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
I'm assuming this should just cover `to_csv`, `to_parquet`, and `to_json`, right? As `to_list` and `to_dict` just return Python objects, `to_pandas` returns a `pandas.DataFrame` and `to_sql` just inserts into a SQL DB, is that right?
2023-07-28T15:28:26Z
https://github.com/huggingface/datasets/issues/6086#issuecomment-1655887081
https://api.github.com/repos/huggingface/datasets/issues/6086
IC_kwDODunzps5istTp
https://api.github.com/repos/huggingface/datasets/issues/comments/1655887081/reactions
2023-07-28T15:28:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1655887081
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Fixed by #6096. 
2024-03-07T07:22:42Z
https://github.com/huggingface/datasets/issues/6086#issuecomment-1982722516
https://api.github.com/repos/huggingface/datasets/issues/6086
IC_kwDODunzps52LfHU
https://api.github.com/repos/huggingface/datasets/issues/comments/1982722516/reactions
2024-03-07T07:22:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1982722516
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Supporting this should be fairly easy.

Requested on the forum [here](https://discuss.huggingface.co/t/how-can-i-convert-a-loaded-dataset-in-to-a-parquet-file-and-save-it-to-the-s3/48353).