Failing to upload dataset to the hub
MEMBER
Hi ! Feel free to re-run your code later, it will resume automatically where you left
2023-06-14T12:18:30Z
https://github.com/huggingface/datasets/issues/5945#issuecomment-1591082689
https://api.github.com/repos/huggingface/datasets/issues/5945
IC_kwDODunzps5e1f7B
https://api.github.com/repos/huggingface/datasets/issues/comments/1591082689/reactions
2023-06-14T12:18:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1591082689
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Tried many times in the last 2 weeks, problem remains.
2023-06-18T16:00:13Z
https://github.com/huggingface/datasets/issues/5945#issuecomment-1596191510
https://api.github.com/repos/huggingface/datasets/issues/5945
IC_kwDODunzps5fI_MW
https://api.github.com/repos/huggingface/datasets/issues/comments/1596191510/reactions
2023-06-18T16:00:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1596191510
https://avatars.githubusercontent.com/u/77382661?v=4
https://api.github.com/users/Ar770/events{/privacy}
https://api.github.com/users/Ar770/followers
https://api.github.com/users/Ar770/following{/other_user}
https://api.github.com/users/Ar770/gists{/gist_id}
https://github.com/Ar770
Ar770
MDQ6VXNlcjc3MzgyNjYx
https://api.github.com/users/Ar770/orgs
https://api.github.com/users/Ar770/received_events
https://api.github.com/users/Ar770/repos
https://api.github.com/users/Ar770/starred{/owner}{/repo}
https://api.github.com/users/Ar770/subscriptions
User
https://api.github.com/users/Ar770
public
MEMBER
Alternatively you can save your dataset in parquet files locally and upload them to the hub manually

```python
from tqdm import tqdm
num_shards = 60
for index in tqdm(range(num_shards)):
    ds.shard(num_shards=num_shards, index=index, contiguous=True).to_parquet(f"{index:05d}.parquet")
````
2023-06-19T10:54:54Z
https://github.com/huggingface/datasets/issues/5945#issuecomment-1596964827
https://api.github.com/repos/huggingface/datasets/issues/5945
IC_kwDODunzps5fL7_b
https://api.github.com/repos/huggingface/datasets/issues/comments/1596964827/reactions
2023-06-19T10:54:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1596964827
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

Trying to upload a dataset of hundreds of thousands of audio samples (the total volume is not very large, 60 gb) to the hub with push_to_hub, it doesn't work.
From time to time one piece of the data (parquet) gets pushed and then I get RemoteDisconnected even though my internet is stable.
Please help.
I'm trying to upload the dataset for almost a week.
Thanks

### Steps to reproduce the bug

not relevant 

### Expected behavior

Be able to upload thedataset

### Environment info

python: 3.9