Integrate Polars library
MEMBER
If possible, a neat API could be something like `Dataset.to_polars()`, as well as `Dataset.set_format("polars")`
2021-11-29T12:55:50Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-981608382
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps46gie-
https://api.github.com/repos/huggingface/datasets/issues/comments/981608382/reactions
2021-11-29T12:55:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/981608382
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
Note they use a "custom" implementation of Arrow: [Arrow2](https://github.com/jorgecarleitao/arrow2).
2021-11-29T13:01:11Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-981612222
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps46gja-
https://api.github.com/repos/huggingface/datasets/issues/comments/981612222/reactions
2021-11-29T13:01:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/981612222
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Polars has grown rapidly in popularity over the last year - could you consider integrating the Polars functionality again?

I don't think the "custom" implementation should be a barrier, it still conforms to the Arrow specification 
2022-11-01T15:07:07Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-1298654844
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps5NZ-Z8
https://api.github.com/repos/huggingface/datasets/issues/comments/1298654844/reactions
2022-11-01T15:07:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1298654844
https://avatars.githubusercontent.com/u/10512793?v=4
https://api.github.com/users/braaannigan/events{/privacy}
https://api.github.com/users/braaannigan/followers
https://api.github.com/users/braaannigan/following{/other_user}
https://api.github.com/users/braaannigan/gists{/gist_id}
https://github.com/braaannigan
braaannigan
MDQ6VXNlcjEwNTEyNzkz
https://api.github.com/users/braaannigan/orgs
https://api.github.com/users/braaannigan/received_events
https://api.github.com/users/braaannigan/repos
https://api.github.com/users/braaannigan/starred{/owner}{/repo}
https://api.github.com/users/braaannigan/subscriptions
User
https://api.github.com/users/braaannigan
public
NONE
Is there some direction regarding this from the HF team @lewtun ? Can conversion from polars to HF dataset be implemented with limited/zero copy? So, something like ``Dataset.from_polars()`` and ``Dataset.to_polars()`` like you mentioned. Happy to contribute if I can get some pointers on how this may be implemented.
2023-09-12T14:53:27Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-1715883608
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps5mRk5Y
https://api.github.com/repos/huggingface/datasets/issues/comments/1715883608/reactions
2023-09-12T14:53:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1715883608
https://avatars.githubusercontent.com/u/8986523?v=4
https://api.github.com/users/amrit110/events{/privacy}
https://api.github.com/users/amrit110/followers
https://api.github.com/users/amrit110/following{/other_user}
https://api.github.com/users/amrit110/gists{/gist_id}
https://github.com/amrit110
amrit110
MDQ6VXNlcjg5ODY1MjM=
https://api.github.com/users/amrit110/orgs
https://api.github.com/users/amrit110/received_events
https://api.github.com/users/amrit110/repos
https://api.github.com/users/amrit110/starred{/owner}{/repo}
https://api.github.com/users/amrit110/subscriptions
User
https://api.github.com/users/amrit110
public
CONTRIBUTOR
Hi, is there any updates? Thanks!
2024-03-16T01:34:59Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-2001009865
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps53RPzJ
https://api.github.com/repos/huggingface/datasets/issues/comments/2001009865/reactions
2024-03-16T01:34:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2001009865
https://avatars.githubusercontent.com/u/5236035?v=4
https://api.github.com/users/fzyzcjy/events{/privacy}
https://api.github.com/users/fzyzcjy/followers
https://api.github.com/users/fzyzcjy/following{/other_user}
https://api.github.com/users/fzyzcjy/gists{/gist_id}
https://github.com/fzyzcjy
fzyzcjy
MDQ6VXNlcjUyMzYwMzU=
https://api.github.com/users/fzyzcjy/orgs
https://api.github.com/users/fzyzcjy/received_events
https://api.github.com/users/fzyzcjy/repos
https://api.github.com/users/fzyzcjy/starred{/owner}{/repo}
https://api.github.com/users/fzyzcjy/subscriptions
User
https://api.github.com/users/fzyzcjy
public
NONE
> Hi, is there any updates? Thanks!

The feature has been there for a bit 😊 You can call `dataset.to_polars()` (on a `Dataset`, not a `DatasetDict`). The issue can be closed, I guess! @lhoestq 
2024-08-30T20:29:51Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-2322291726
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps6Ka1wO
https://api.github.com/repos/huggingface/datasets/issues/comments/2322291726/reactions
2024-08-30T20:29:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2322291726
https://avatars.githubusercontent.com/u/57922983?v=4
https://api.github.com/users/baggiponte/events{/privacy}
https://api.github.com/users/baggiponte/followers
https://api.github.com/users/baggiponte/following{/other_user}
https://api.github.com/users/baggiponte/gists{/gist_id}
https://github.com/baggiponte
baggiponte
MDQ6VXNlcjU3OTIyOTgz
https://api.github.com/users/baggiponte/orgs
https://api.github.com/users/baggiponte/received_events
https://api.github.com/users/baggiponte/repos
https://api.github.com/users/baggiponte/starred{/owner}{/repo}
https://api.github.com/users/baggiponte/subscriptions
User
https://api.github.com/users/baggiponte
public
CONTRIBUTOR
Looks great and thanks!
2024-08-30T23:55:38Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-2322609416
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps6KcDUI
https://api.github.com/repos/huggingface/datasets/issues/comments/2322609416/reactions
2024-08-30T23:55:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2322609416
https://avatars.githubusercontent.com/u/5236035?v=4
https://api.github.com/users/fzyzcjy/events{/privacy}
https://api.github.com/users/fzyzcjy/followers
https://api.github.com/users/fzyzcjy/following{/other_user}
https://api.github.com/users/fzyzcjy/gists{/gist_id}
https://github.com/fzyzcjy
fzyzcjy
MDQ6VXNlcjUyMzYwMzU=
https://api.github.com/users/fzyzcjy/orgs
https://api.github.com/users/fzyzcjy/received_events
https://api.github.com/users/fzyzcjy/repos
https://api.github.com/users/fzyzcjy/starred{/owner}{/repo}
https://api.github.com/users/fzyzcjy/subscriptions
User
https://api.github.com/users/fzyzcjy
public
MEMBER
Thank you.
2024-08-31T05:31:27Z
https://github.com/huggingface/datasets/issues/3334#issuecomment-2322780558
https://api.github.com/repos/huggingface/datasets/issues/3334
IC_kwDODunzps6KctGO
https://api.github.com/repos/huggingface/datasets/issues/comments/2322780558/reactions
2024-08-31T05:31:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2322780558
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Check potential integration of the Polars library: https://github.com/pola-rs/polars
- Benchmark: https://h2oai.github.io/db-benchmark/

CC: @thomwolf @lewtun 
