HfHubHTTPError - exceeded our hourly quotas for action: commit
MEMBER
how is your dataset structured? (file types, how many commits and files are you trying to push, etc)
2023-09-25T10:00:32Z
https://github.com/huggingface/datasets/issues/6257#issuecomment-1733348075
https://api.github.com/repos/huggingface/datasets/issues/6257
IC_kwDODunzps5nUMrr
https://api.github.com/repos/huggingface/datasets/issues/comments/1733348075/reactions
2023-09-25T10:00:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733348075
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
NONE
I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.
2023-09-25T12:50:23Z
https://github.com/huggingface/datasets/issues/6257#issuecomment-1733648809
https://api.github.com/repos/huggingface/datasets/issues/6257
IC_kwDODunzps5nVWGp
https://api.github.com/repos/huggingface/datasets/issues/comments/1733648809/reactions
2023-09-25T12:52:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733648809
https://avatars.githubusercontent.com/u/57996478?v=4
https://api.github.com/users/yuvalkirstain/events{/privacy}
https://api.github.com/users/yuvalkirstain/followers
https://api.github.com/users/yuvalkirstain/following{/other_user}
https://api.github.com/users/yuvalkirstain/gists{/gist_id}
https://github.com/yuvalkirstain
yuvalkirstain
MDQ6VXNlcjU3OTk2NDc4
https://api.github.com/users/yuvalkirstain/orgs
https://api.github.com/users/yuvalkirstain/received_events
https://api.github.com/users/yuvalkirstain/repos
https://api.github.com/users/yuvalkirstain/starred{/owner}{/repo}
https://api.github.com/users/yuvalkirstain/subscriptions
User
https://api.github.com/users/yuvalkirstain
public
COLLABORATOR
We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do. 

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?
2023-09-26T15:56:03Z
https://github.com/huggingface/datasets/issues/6257#issuecomment-1735831786
https://api.github.com/repos/huggingface/datasets/issues/6257
IC_kwDODunzps5ndrDq
https://api.github.com/repos/huggingface/datasets/issues/comments/1735831786/reactions
2023-09-26T15:56:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1735831786
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.
2023-09-27T07:04:59Z
https://github.com/huggingface/datasets/issues/6257#issuecomment-1736822023
https://api.github.com/repos/huggingface/datasets/issues/6257
IC_kwDODunzps5nhc0H
https://api.github.com/repos/huggingface/datasets/issues/comments/1736822023/reactions
2023-09-27T07:04:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1736822023
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3