Add DatasetDict.to_pandas
NONE
The current implementation is what I had in mind, i.e. concatenate all splits by default.

However, I think most tabular datasets would come as a single split. So for that usecase, it wouldn't change UX if we raise when there are more than one splits.

And for multiple splits, the user either passes a list, or they can pass `splits="all"` to have all splits concatenated.
2022-11-29T16:46:55Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1330947615
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PVKYf
https://api.github.com/repos/huggingface/datasets/issues/comments/1330947615/reactions
2022-11-29T16:46:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330947615
https://avatars.githubusercontent.com/u/1663864?v=4
https://api.github.com/users/adrinjalali/events{/privacy}
https://api.github.com/users/adrinjalali/followers
https://api.github.com/users/adrinjalali/following{/other_user}
https://api.github.com/users/adrinjalali/gists{/gist_id}
https://github.com/adrinjalali
adrinjalali
MDQ6VXNlcjE2NjM4NjQ=
https://api.github.com/users/adrinjalali/orgs
https://api.github.com/users/adrinjalali/received_events
https://api.github.com/users/adrinjalali/repos
https://api.github.com/users/adrinjalali/starred{/owner}{/repo}
https://api.github.com/users/adrinjalali/subscriptions
User
https://api.github.com/users/adrinjalali
public
CONTRIBUTOR
I think it's better to raise an error in cases when there are multiple splits but no split is specified so that users know for sure with which data they are working. I imagine a case when a user loads a dataset that they don't know much about (like what splits it has), and if they get a concatenation of everything, it might lead to incorrect processing or interpretations and it would be hard to notice it.
("explicit is better than implicit")
2022-11-29T17:04:57Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1330979253
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PVSG1
https://api.github.com/repos/huggingface/datasets/issues/comments/1330979253/reactions
2022-11-29T17:08:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330979253
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
I just changed to raise an error if there are multiple splits. The error shows an example of how to choose a split to convert.
2022-11-29T17:26:37Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1331017087
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PVbV_
https://api.github.com/repos/huggingface/datasets/issues/comments/1331017087/reactions
2022-11-29T17:26:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331017087
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5312). All of your documentation changes will be reflected on that endpoint.
2022-11-29T17:30:31Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1331022027
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PVcjL
https://api.github.com/repos/huggingface/datasets/issues/comments/1331022027/reactions
2022-11-29T17:30:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331022027
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Thanks for the review, I've updated the type hint and added a line to raise an error on bad splits :)
2022-12-05T14:44:09Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1337503671
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PuK-3
https://api.github.com/repos/huggingface/datasets/issues/comments/1337503671/reactions
2022-12-05T14:44:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1337503671
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Merging https://github.com/huggingface/datasets/pull/5301 would eliminate the need for this PR, no?

In the meantime, I find the current API cleaner.
2022-12-05T15:17:19Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1337558490
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5PuYXa
https://api.github.com/repos/huggingface/datasets/issues/comments/1337558490/reactions
2022-12-05T15:17:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1337558490
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
This solution is simpler than https://github.com/huggingface/datasets/pull/5301 and covers most cases for tabular datasets, so I'm in favor of merging this one and put https://github.com/huggingface/datasets/pull/5301 on stand by
2022-12-06T12:09:48Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1339225726
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5P0vZ-
https://api.github.com/repos/huggingface/datasets/issues/comments/1339225726/reactions
2022-12-06T12:09:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1339225726
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Let me know if it sounds good to you @mariosasko @albertvillanova :)
2022-12-07T12:03:52Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1340869112
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5P7An4
https://api.github.com/repos/huggingface/datasets/issues/comments/1340869112/reactions
2022-12-07T12:03:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1340869112
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
I'm still not convinced. If `DatasetDict` needs this method and there is no other way, then IMO it would make more sense to return a dictionary with the splits converted to `pd.DataFrame`. 
2022-12-07T14:15:01Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1341030471
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5P7oBH
https://api.github.com/repos/huggingface/datasets/issues/comments/1341030471/reactions
2022-12-07T14:15:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341030471
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko the issue we're dealing with is that in tabular scenarios, we often don't have splits in the dataset, and imposing that concept to people dealing with the library hampers adoption.
2022-12-07T14:36:08Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1341058735
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5P7u6v
https://api.github.com/repos/huggingface/datasets/issues/comments/1341058735/reactions
2022-12-07T14:36:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341058735
https://avatars.githubusercontent.com/u/1663864?v=4
https://api.github.com/users/adrinjalali/events{/privacy}
https://api.github.com/users/adrinjalali/followers
https://api.github.com/users/adrinjalali/following{/other_user}
https://api.github.com/users/adrinjalali/gists{/gist_id}
https://github.com/adrinjalali
adrinjalali
MDQ6VXNlcjE2NjM4NjQ=
https://api.github.com/users/adrinjalali/orgs
https://api.github.com/users/adrinjalali/received_events
https://api.github.com/users/adrinjalali/repos
https://api.github.com/users/adrinjalali/starred{/owner}{/repo}
https://api.github.com/users/adrinjalali/subscriptions
User
https://api.github.com/users/adrinjalali
public
COLLABORATOR
@adrinjalali This PR proposes a solution inconsistent with the existing API (in other words, a solution that clutters our API 🙂). Moreover, our library primarily focuses on larger-than-RAM datasets, and tabular datasets don't (directly) fall into this group.

Instead of the temporary "fix" proposed here, it makes much more sense to align `load_dataset` with both tabular and DL workflows "in a consistent way", so I suggest we continue our discussion from https://github.com/huggingface/datasets/issues/5189 to have this resolved by version 3.0.
2022-12-12T18:50:42Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1347076122
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5QSsAa
https://api.github.com/repos/huggingface/datasets/issues/comments/1347076122/reactions
2022-12-12T18:50:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1347076122
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
closing this one for now
2023-01-25T17:33:42Z
https://github.com/huggingface/datasets/pull/5312#issuecomment-1403986550
https://api.github.com/repos/huggingface/datasets/issues/5312
IC_kwDODunzps5TryJ2
https://api.github.com/repos/huggingface/datasets/issues/comments/1403986550/reactions
2023-01-25T17:33:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1403986550
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
From discussions in https://github.com/huggingface/datasets/issues/5189, for tabular data it doesn't really make sense to have to do
```python
df = load_dataset(...)["train"].to_pandas()
```
because many datasets are not split.

In this PR I added `to_pandas` to `DatasetDict` which returns the DataFrame:

If there's only one split, you don't need to specify the split name:
```python
df = load_dataset(...).to_pandas()
```
EDIT: and if a dataset has multiple splits:
```python
df = load_dataset(...).to_pandas(splits=["train", "test"])
# or
df = load_dataset(...).to_pandas(splits="all")

# raises an error because you need to select the split(s) to convert
load_dataset(...).to_pandas()
```

I do have one question though @merveenoyan @adrinjalali @mariosasko:

Should we raise an error if there are multiple splits and ask the user to choose one explicitly ?

