Support Dataset conversion from/to Spark
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5401). All of your documentation changes will be reflected on that endpoint.
2023-01-03T10:02:29Z
https://github.com/huggingface/datasets/pull/5401#issuecomment-1369581592
https://api.github.com/repos/huggingface/datasets/issues/5401
IC_kwDODunzps5RoigY
https://api.github.com/repos/huggingface/datasets/issues/comments/1369581592/reactions
2023-01-03T10:02:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1369581592
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Cool thanks !

Spark DataFrame are usually quite big, and I believe here `from_spark` would load everything in the driver node's RAM, which is quite limiting. Same for `to_spark` which would load everything in the driver node's RAM before sending the data to the executor. Maybe we can mention this in the docstring ?

To transfer big datasets from/into the HF ecosystem using Spark maybe we can just make sure that `pyspark` can read/write to the HF Hub, and that `datasets` can read from HDFS/S3/etc.
2023-01-03T16:43:49Z
https://github.com/huggingface/datasets/pull/5401#issuecomment-1369988664
https://api.github.com/repos/huggingface/datasets/issues/5401
IC_kwDODunzps5RqF44
https://api.github.com/repos/huggingface/datasets/issues/comments/1369988664/reactions
2023-01-03T16:43:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1369988664
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes @lhoestq , consider this as a first integration of the Datasets library with Spark.
- This PR implements the basic conversion between both.
  - And yes, we are using the Spark's `pandas` API (that uses `pyarrow` under the hood): everything is transferred to the driver.
  - Note that we are converting from/to a Datasets dataset: this is not distributed

The next step is to support the integration of the HF Hub with Spark, that I think should be done using `hffs`.
2023-01-03T18:01:59Z
https://github.com/huggingface/datasets/pull/5401#issuecomment-1370070015
https://api.github.com/repos/huggingface/datasets/issues/5401
IC_kwDODunzps5RqZv_
https://api.github.com/repos/huggingface/datasets/issues/comments/1370070015/reactions
2023-01-03T18:04:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1370070015
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thinking more about it I don't really see how those two methods help in practice, since one can already do `datasets` <-> pandas <-> spark and those two methods don't add value over this.

However I think it can be good documentation to explain that it's possible to do it and it's super simple
2023-01-05T14:21:33Z
https://github.com/huggingface/datasets/pull/5401#issuecomment-1372277934
https://api.github.com/repos/huggingface/datasets/issues/5401
IC_kwDODunzps5Ry0yu
https://api.github.com/repos/huggingface/datasets/issues/comments/1372277934/reactions
2023-01-05T14:21:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1372277934
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR implements Spark integration by supporting `Dataset` conversion from/to Spark `DataFrame`.