Retry HfApi call inside push_to_hub when 504 error
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3886). All of your documentation changes will be reflected on that endpoint.
2022-03-10T13:30:37Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1064067083
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_bGAL
https://api.github.com/repos/huggingface/datasets/issues/comments/1064067083/reactions
2022-03-10T13:30:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1064067083
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
I made it more robust by increasing the wait time, and I also added some logs when a request is retried. Let me know if it's ok for you
2022-03-15T11:06:20Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1067854424
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_pipY
https://api.github.com/repos/huggingface/datasets/issues/comments/1067854424/reactions
2022-03-15T11:06:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067854424
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
At the end you did not set the agreed max value of 60s. 

Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of `base_wait_time` and `max_wait_time`.
2022-03-15T11:19:20Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1067867147
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_plwL
https://api.github.com/repos/huggingface/datasets/issues/comments/1067867147/reactions
2022-03-15T11:22:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067867147
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yea I thought that in total we could wait 1min, but if we have a max_wait_time of 20sec between each request it's fine IMO

> Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of base_wait_time and max_wait_time.

What makes you think this ? If the exponential wait time becomes bigger than `max_wait_time` then it still does the retry, but after a wait time of `max_wait_time`
2022-03-15T11:29:42Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1067882442
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_ppfK
https://api.github.com/repos/huggingface/datasets/issues/comments/1067882442/reactions
2022-03-15T11:29:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067882442
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Sorry, I meant 4 retries **with exponential backoff**; the fifth one is with constant backoff.
2022-03-15T11:40:27Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1067890439
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_prcH
https://api.github.com/repos/huggingface/datasets/issues/comments/1067890439/reactions
2022-03-15T11:40:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067890439
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
OK, and one question: do you think that the retries do not affect the time the server needs to be operational again and able to process the request? I guess that if does not affect, then the cause are other users' requests, or others; not our specific request.

Just to be sure: 
- Then 20s at most between consecutive requests do not impact the server.
- And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.
2022-03-15T11:44:57Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1067893731
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_psPj
https://api.github.com/repos/huggingface/datasets/issues/comments/1067893731/reactions
2022-03-15T13:44:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067893731
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
> do you think that the retries do not affect the time the server needs for being able to process the request (I guess in this case the cause are other users' requests, or other causes; not our specific request).

Yes I don't think the retries would affect the server, I think the cause of the 504 errors is elsewhere

> Just to be sure:
>
> Then 20s at most between consecutive requests do not impact the server.
> And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

Yes I think it's fine for now, we can still adapt this later if needed
2022-03-15T13:43:04Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1068004298
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_qHPK
https://api.github.com/repos/huggingface/datasets/issues/comments/1068004298/reactions
2022-03-15T13:43:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068004298
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Will be curious to see the impact of this in terms of upload reliability! Don't forget to let us know when you have more data. cc @huggingface/moon-landing-back 
2022-03-16T09:00:56Z
https://github.com/huggingface/datasets/pull/3886#issuecomment-1068886924
https://api.github.com/repos/huggingface/datasets/issues/3886
IC_kwDODunzps4_teuM
https://api.github.com/repos/huggingface/datasets/issues/comments/1068886924/reactions
2022-03-16T09:00:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068886924
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.