Add list_github_datasets api for retrieving dataset name list in github repo
MEMBER
We can look into removing some of the attributes from `GET /api/datasets` to make it smaller/faster, what do you think @lhoestq?
2020-11-30T08:50:23Z
https://github.com/huggingface/datasets/pull/914#issuecomment-735646401
https://api.github.com/repos/huggingface/datasets/issues/914
MDEyOklzc3VlQ29tbWVudDczNTY0NjQwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/735646401/reactions
2020-11-30T08:50:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/735646401
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
MEMBER
> We can look into removing some of the attributes from `GET /api/datasets` to make it smaller/faster, what do you think @lhoestq?

yes at least remove all the `dummy_data.zip`
2020-11-30T08:55:02Z
https://github.com/huggingface/datasets/pull/914#issuecomment-735648747
https://api.github.com/repos/huggingface/datasets/issues/914
MDEyOklzc3VlQ29tbWVudDczNTY0ODc0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/735648747/reactions
2020-11-30T08:55:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/735648747
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
`GET /api/datasets` should now be much faster. @zhuzilin can you check if `list_datasets` is now faster for you?
2020-12-01T19:58:01Z
https://github.com/huggingface/datasets/pull/914#issuecomment-736783384
https://api.github.com/repos/huggingface/datasets/issues/914
MDEyOklzc3VlQ29tbWVudDczNjc4MzM4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/736783384/reactions
2020-12-01T19:58:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736783384
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
NONE
> `GET /api/datasets` should now be much faster. @zhuzilin can you check if `list_datasets` is now faster for you?

Yes, much faster! Thank you!
2020-12-02T07:21:15Z
https://github.com/huggingface/datasets/pull/914#issuecomment-737045559
https://api.github.com/repos/huggingface/datasets/issues/914
MDEyOklzc3VlQ29tbWVudDczNzA0NTU1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737045559/reactions
2020-12-02T07:21:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737045559
https://avatars.githubusercontent.com/u/10428324?v=4
https://api.github.com/users/zhuzilin/events{/privacy}
https://api.github.com/users/zhuzilin/followers
https://api.github.com/users/zhuzilin/following{/other_user}
https://api.github.com/users/zhuzilin/gists{/gist_id}
https://github.com/zhuzilin
zhuzilin
MDQ6VXNlcjEwNDI4MzI0
https://api.github.com/users/zhuzilin/orgs
https://api.github.com/users/zhuzilin/received_events
https://api.github.com/users/zhuzilin/repos
https://api.github.com/users/zhuzilin/starred{/owner}{/repo}
https://api.github.com/users/zhuzilin/subscriptions
User
https://api.github.com/users/zhuzilin
public
Thank you for your great effort on unifying data processing for NLP!

This pr is trying to add a new api `list_github_datasets` in the `inspect` module. The reason for it is that the current `list_datasets` api need to access https://huggingface.co/api/datasets to get a large json. However, this connection can be really slow... (I was visiting from China) and from my own experience, most of the time `requests.get` failed to download the whole json after a long wait and will trigger fault in `r.json()`.
I also noticed that the current implementation will first try to download from github, which makes me be able to smoothly run `load_dataset('squad')` in the example.
Therefore, I think it would be better if we can have an api to get the list of datasets that are available on github, and it will also improve newcomers' experience (it is a little frustrating if one cannot successfully run the first function in the README example.) before we have faster source for huggingface.co.

As for the implementation, I've added a `dataset_infos.json` file under the `datasets` folder, and it has the following structure:
```json
    {
        "id": "aeslc",
        "folder": "datasets/aeslc",
        "dataset_infos": "datasets/aeslc/dataset_infos.json"
    },
    ...
    {
        "id": "json",
        "folder": "datasets/json"
    },
   ...
```
The script I used to get this file is:
```python
import json
import os

DATASETS_BASE_DIR = "/root/datasets" 
DATASET_INFOS_JSON = "dataset_infos.json"

datasets = []
for item in os.listdir(os.path.join(DATASETS_BASE_DIR, "datasets")):
    if os.path.isdir(os.path.join(DATASETS_BASE_DIR, "datasets", item)):
        datasets.append(item)

datasets.sort()

total_ds_info = []
for ds in datasets:
    ds_dir = os.path.join("datasets", ds)
    ds_info_dir = os.path.join(ds_dir, DATASET_INFOS_JSON)
    if os.path.isfile(os.path.join(DATASETS_BASE_DIR, ds_info_dir)):
        total_ds_info.append({"id": ds,
                              "folder": ds_dir,
                              "dataset_infos": ds_info_dir})
    else:
        total_ds_info.append({"id": ds,
                              "folder": ds_dir})

with open(DATASET_INFOS_JSON, "w") as f:
    json.dump(total_ds_info, f)
```
The new `dataset_infos.json` was saved as a formated json so that it will be easy to add new dataset.

When calling `list_github_datasets`, the user will get the list of dataset names in this github repo and if `with_details` is set to be `True`, they can get the url of specific dataset info.

Thank you for your time on reviewing this pr :).