Allow `list_datasets` to include private datasets
MEMBER
Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)
2022-07-26T10:32:31Z
https://github.com/huggingface/datasets/issues/4745#issuecomment-1195307045
https://api.github.com/repos/huggingface/datasets/issues/4745
IC_kwDODunzps5HPvAl
https://api.github.com/repos/huggingface/datasets/issues/comments/1195307045/reactions
2022-07-26T10:33:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1195307045
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)
2022-07-26T10:44:52Z
https://github.com/huggingface/datasets/issues/4745#issuecomment-1195319228
https://api.github.com/repos/huggingface/datasets/issues/4745
IC_kwDODunzps5HPx-8
https://api.github.com/repos/huggingface/datasets/issues/comments/1195319228/reactions
2022-07-26T10:44:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1195319228
https://avatars.githubusercontent.com/u/1528523?v=4
https://api.github.com/users/ola13/events{/privacy}
https://api.github.com/users/ola13/followers
https://api.github.com/users/ola13/following{/other_user}
https://api.github.com/users/ola13/gists{/gist_id}
https://github.com/ola13
ola13
MDQ6VXNlcjE1Mjg1MjM=
https://api.github.com/users/ola13/orgs
https://api.github.com/users/ola13/received_events
https://api.github.com/users/ola13/repos
https://api.github.com/users/ola13/starred{/owner}{/repo}
https://api.github.com/users/ola13/subscriptions
User
https://api.github.com/users/ola13
public
MEMBER
i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)
2022-07-26T11:59:25Z
https://github.com/huggingface/datasets/issues/4745#issuecomment-1195390206
https://api.github.com/repos/huggingface/datasets/issues/4745
IC_kwDODunzps5HQDT-
https://api.github.com/repos/huggingface/datasets/issues/comments/1195390206/reactions
2022-07-26T11:59:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1195390206
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
COLLABORATOR
`datasets.list_datasets` is now deprecated in favor of `huggingface_hub.list_datasets` (returns private datasets when `token` is present), so I'm closing this issue.
2023-07-25T15:01:48Z
https://github.com/huggingface/datasets/issues/4745#issuecomment-1650011833
https://api.github.com/repos/huggingface/datasets/issues/4745
IC_kwDODunzps5iWS65
https://api.github.com/repos/huggingface/datasets/issues/comments/1650011833/reactions
2023-07-25T15:01:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1650011833
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.