Add support to create a Dataset from spark dataframe
NONE
if i read spark Dataframe , got an error on  multi-node Spark cluster.
Did the Api (Dataset.from_spark) support Spark cluster, read dataframe and save_to_disk?

Error: 
_pickle.PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforma
tion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
23/06/16 21:17:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)


2023-06-16T13:18:05Z
https://github.com/huggingface/datasets/issues/5678#issuecomment-1594664163
https://api.github.com/repos/huggingface/datasets/issues/5678
IC_kwDODunzps5fDKTj
https://api.github.com/repos/huggingface/datasets/issues/comments/1594664163/reactions
2023-06-16T13:19:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1594664163
https://avatars.githubusercontent.com/u/112800614?v=4
https://api.github.com/users/yanzia12138/events{/privacy}
https://api.github.com/users/yanzia12138/followers
https://api.github.com/users/yanzia12138/following{/other_user}
https://api.github.com/users/yanzia12138/gists{/gist_id}
https://github.com/yanzia12138
yanzia12138
U_kgDOBrkzZg
https://api.github.com/users/yanzia12138/orgs
https://api.github.com/users/yanzia12138/received_events
https://api.github.com/users/yanzia12138/repos
https://api.github.com/users/yanzia12138/starred{/owner}{/repo}
https://api.github.com/users/yanzia12138/subscriptions
User
https://api.github.com/users/yanzia12138
public
NONE
How to perform predictions on Dataset object in Spark with multi-node cluster parallelism?
2023-06-16T15:16:01Z
https://github.com/huggingface/datasets/issues/5678#issuecomment-1594857391
https://api.github.com/repos/huggingface/datasets/issues/5678
IC_kwDODunzps5fD5ev
https://api.github.com/repos/huggingface/datasets/issues/comments/1594857391/reactions
2023-06-16T15:16:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1594857391
https://avatars.githubusercontent.com/u/102869204?v=4
https://api.github.com/users/oakkas84/events{/privacy}
https://api.github.com/users/oakkas84/followers
https://api.github.com/users/oakkas84/following{/other_user}
https://api.github.com/users/oakkas84/gists{/gist_id}
https://github.com/oakkas84
oakkas84
U_kgDOBiGo1A
https://api.github.com/users/oakkas84/orgs
https://api.github.com/users/oakkas84/received_events
https://api.github.com/users/oakkas84/repos
https://api.github.com/users/oakkas84/starred{/owner}{/repo}
https://api.github.com/users/oakkas84/subscriptions
User
https://api.github.com/users/oakkas84
public
COLLABORATOR
Addressed in #5701
2023-07-21T14:15:38Z
https://github.com/huggingface/datasets/issues/5678#issuecomment-1645662906
https://api.github.com/repos/huggingface/datasets/issues/5678
IC_kwDODunzps5iFtK6
https://api.github.com/repos/huggingface/datasets/issues/comments/1645662906/reactions
2023-07-21T14:15:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1645662906
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Hi ! for your information we are working on some more documentation on how to use Spark with HF Datasets repositories (without the need for the `datasets` library) <s>https://github.com/huggingface/datasets/issues/5678</s>
Cc @lu-wang-dl @maddiedawson let me know what you think !
2024-08-27T14:41:35Z
https://github.com/huggingface/datasets/issues/5678#issuecomment-2312762573
https://api.github.com/repos/huggingface/datasets/issues/5678
IC_kwDODunzps6J2fTN
https://api.github.com/repos/huggingface/datasets/issues/comments/2312762573/reactions
2024-08-27T14:43:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2312762573
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
sorry, wrong link: https://github.com/huggingface/hub-docs/pull/1392
2024-08-27T14:42:58Z
https://github.com/huggingface/datasets/issues/5678#issuecomment-2312765757
https://api.github.com/repos/huggingface/datasets/issues/5678
IC_kwDODunzps6J2gE9
https://api.github.com/repos/huggingface/datasets/issues/comments/2312765757/reactions
2024-08-27T14:42:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2312765757
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.