Loading allenai/c4 in streaming mode does too many HEAD requests
COLLABORATOR
https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/src/datasets/builder.py#L179-L186
2021-09-01T08:16:34Z
https://github.com/huggingface/datasets/issues/2859#issuecomment-910052897
https://api.github.com/repos/huggingface/datasets/issues/2859
IC_kwDODunzps42Pk4h
https://api.github.com/repos/huggingface/datasets/issues/comments/910052897/reactions
2021-09-01T08:16:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/910052897
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
Thanks a lot!!!
2021-10-12T07:35:52Z
https://github.com/huggingface/datasets/issues/2859#issuecomment-940745453
https://api.github.com/repos/huggingface/datasets/issues/2859
IC_kwDODunzps44EqLt
https://api.github.com/repos/huggingface/datasets/issues/comments/940745453/reactions
2021-10-12T07:35:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/940745453
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
This does 60,000+ HEAD requests to get all the ETags of all the data files:
```python
from datasets import load_dataset
load_dataset("allenai/c4", streaming=True)
```
It makes loading the dataset completely impractical.

The ETags are used to compute the config id (it must depend on the data files being used).
Instead of using the ETags, we could simply use the commit hash of the dataset repository on the hub, as well and the glob pattern used to resolve the files (here it's `*` by default, to load all the files of the repository)