Getting configs names takes too long
COLLABORATOR
maybe related to https://github.com/huggingface/datasets/issues/2859

2021-12-13T14:28:54Z
https://github.com/huggingface/datasets/issues/3425#issuecomment-992536177
https://api.github.com/repos/huggingface/datasets/issues/3425
IC_kwDODunzps47KOZx
https://api.github.com/repos/huggingface/datasets/issues/comments/992536177/reactions
2021-12-13T14:28:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/992536177
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
It looks like it's currently calling `HfFileSystem.ls()` ~8 times at the root and for each subdirectory:
- ""
- "en.noblocklist"
- "en.noclean"
- "en"
- "multilingual"
- "realnewslike"

Currently `ls` is slow because it iterates on all the files inside the repository.

An easy optimization would be to cache the result of each call to `ls`.
We can also optimize `ls` by using a tree structure per directory instead of a list of all the files.

2021-12-13T14:39:55Z
https://github.com/huggingface/datasets/issues/3425#issuecomment-992546029
https://api.github.com/repos/huggingface/datasets/issues/3425
IC_kwDODunzps47KQzt
https://api.github.com/repos/huggingface/datasets/issues/comments/992546029/reactions
2021-12-13T14:39:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/992546029
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
ok

2021-12-13T14:53:33Z
https://github.com/huggingface/datasets/issues/3425#issuecomment-992559004
https://api.github.com/repos/huggingface/datasets/issues/3425
IC_kwDODunzps47KT-c
https://api.github.com/repos/huggingface/datasets/issues/comments/992559004/reactions
2021-12-13T14:53:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/992559004
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public

## Steps to reproduce the bug
```python
from datasets import get_dataset_config_names
get_dataset_config_names("allenai/c4")
```

## Expected results

I would expect to get the answer quickly, at least in less than 10s

## Actual results

It takes about 45s on my environment

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1