Easy way to create a Kaggle dataset from a Huggingface dataset?
CONTRIBUTOR
Hi @hrbigelow , I'm no expert for such a question so I'll ping @lhoestq from the `datasets` library (also this issue could be moved there if someone with permission can do it :) )
2023-05-03T07:05:32Z
https://github.com/huggingface/datasets/issues/5815#issuecomment-1532707776
https://api.github.com/repos/huggingface/datasets/issues/5815
IC_kwDODunzps5bW0PA
https://api.github.com/repos/huggingface/datasets/issues/comments/1532707776/reactions
2023-05-03T07:05:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1532707776
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
MEMBER
Hi ! Many datasets are made of several files, and how they are parsed often requires a python script. Because of that, datasets like wmt14 are not available as a single file on HF. Though you can create this file using `datasets`:

```python
from datasets import load_dataset

ds = load_dataset("wmt14", "de-en", split="train")

ds.to_json("wmt14-train.json")
# OR to parquet, which is compressed:
# ds.to_parquet("wmt14-train.parquet")
```

We are also working on providing parquet exports for all datasets, but wmt14 is not supported yet (we're rolling it out for datasets <1GB first). They're usually available in the `refs/convert/parquet` branch (empty for wmt14):

<img width="267" alt="image" src="https://user-images.githubusercontent.com/42851186/235878909-7339f5a4-be19-4ada-85d8-8a50d23acf35.png">

2023-05-03T09:23:22Z
https://github.com/huggingface/datasets/issues/5815#issuecomment-1532707779
https://api.github.com/repos/huggingface/datasets/issues/5815
IC_kwDODunzps5bW0PD
https://api.github.com/repos/huggingface/datasets/issues/comments/1532707779/reactions
2023-05-03T09:23:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1532707779
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
also cc @nateraw for visibility on this (and cc @osanseviero too)
2023-05-03T10:07:50Z
https://github.com/huggingface/datasets/issues/5815#issuecomment-1532766528
https://api.github.com/repos/huggingface/datasets/issues/5815
IC_kwDODunzps5bXClA
https://api.github.com/repos/huggingface/datasets/issues/comments/1532766528/reactions
2023-05-03T10:07:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1532766528
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
COLLABORATOR
I've requested support for creating a Kaggle dataset from an imported HF dataset repo on their "forum" here: https://www.kaggle.com/discussions/product-feedback/427142 (upvotes appreciated 🙂)
2023-07-26T16:13:31Z
https://github.com/huggingface/datasets/issues/5815#issuecomment-1652122766
https://api.github.com/repos/huggingface/datasets/issues/5815
IC_kwDODunzps5ieWSO
https://api.github.com/repos/huggingface/datasets/issues/comments/1652122766/reactions
2023-07-26T16:13:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1652122766
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I'm not sure whether this is more appropriately addressed with HuggingFace or Kaggle.  I would like to somehow directly create a Kaggle dataset from a HuggingFace Dataset.

While Kaggle does provide the option to create a dataset from a URI, that URI must point to a single file.  For example:

![image](https://user-images.githubusercontent.com/5355286/235792394-7c559d07-4aff-45b7-ad2b-9c5280c88415.png)


Is there some mechanism from huggingface to represent a dataset (such as that from `load_dataset('wmt14', 'de-en', split='train')` as a single file?  Or, some other way to get that into a Kaggle dataset so that I can use the huggingface `datasets` module to process and consume it inside of a Kaggle notebook?

Thanks in advance!
