adding progress bar / ETA for `load_dataset`
NONE
Is this done now? I see progress bars when using `load_dataset`.
2023-03-24T19:15:43Z
https://github.com/huggingface/datasets/issues/2649#issuecomment-1483294253
https://api.github.com/repos/huggingface/datasets/issues/2649
IC_kwDODunzps5YaUYt
https://api.github.com/repos/huggingface/datasets/issues/comments/1483294253/reactions
2023-03-24T19:15:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1483294253
https://avatars.githubusercontent.com/u/95326326?v=4
https://api.github.com/users/jasongrout-db/events{/privacy}
https://api.github.com/users/jasongrout-db/followers
https://api.github.com/users/jasongrout-db/following{/other_user}
https://api.github.com/users/jasongrout-db/gists{/gist_id}
https://github.com/jasongrout-db
jasongrout-db
U_kgDOBa6Qdg
https://api.github.com/users/jasongrout-db/orgs
https://api.github.com/users/jasongrout-db/received_events
https://api.github.com/users/jasongrout-db/repos
https://api.github.com/users/jasongrout-db/starred{/owner}{/repo}
https://api.github.com/users/jasongrout-db/subscriptions
User
https://api.github.com/users/jasongrout-db
public
MEMBER
There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
2023-03-27T10:22:09Z
https://github.com/huggingface/datasets/issues/2649#issuecomment-1484890195
https://api.github.com/repos/huggingface/datasets/issues/2649
IC_kwDODunzps5YgaBT
https://api.github.com/repos/huggingface/datasets/issues/comments/1484890195/reactions
2023-03-27T10:32:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1484890195
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq 
