Automatically revert to last successful commit to hub when a push_to_hub is interrupted
MEMBER
Could you share the error you got please ? Maybe the full stack trace if you have it ?

Maybe `push_to_hub` be implemented as a single commit @Wauplin ? This way if it fails, the repo is still at the previous (valid) state instead of ending-up in an invalid/incimplete state.
2022-09-30T15:25:55Z
https://github.com/huggingface/datasets/issues/5045#issuecomment-1263715857
https://api.github.com/repos/huggingface/datasets/issues/5045
IC_kwDODunzps5LUsYR
https://api.github.com/repos/huggingface/datasets/issues/comments/1263715857/reactions
2022-09-30T15:26:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263715857
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Maybe push_to_hub be implemented as a single commit ? 

I think that would definitely be the way to go. Do you know the reasons why not implementing it like this in the first place ? I guess it is because of not been able to upload all at once with `huggingface_hub` but if there was another reason, please let me know.
About pushing all at once, it seems to be a more and more requested feature. I have created this issue https://github.com/huggingface/huggingface_hub/issues/1085 recently but other discussions already happened in the past. The `moon-landing` team is working on it (cc @coyotte508). The `huggingface_hub` integration will come afterwards.

For now, maybe it's best to wait for a proper implementation instead of creating a temporary workaround :)

2022-09-30T15:36:41Z
https://github.com/huggingface/datasets/issues/5045#issuecomment-1263727261
https://api.github.com/repos/huggingface/datasets/issues/5045
IC_kwDODunzps5LUvKd
https://api.github.com/repos/huggingface/datasets/issues/comments/1263727261/reactions
2022-09-30T15:36:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263727261
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
MEMBER
> I think that would definitely be the way to go. Do you know the reasons why not implementing it like this in the first place ? I guess it is because of not been able to upload all at once with huggingface_hub but if there was another reason, please let me know.

Ideally we would want to upload the files iteratively - and then once everything is uploaded we proceed to commit. When we implemented `push_to_hub`, using `upload_file` for each shard was the only option.

For more context: for each shard to upload we do:
1. load the arrow shard in memory
2. convert to parquet
3. upload

So to avoid OOM we need to upload the files iteratively.

> For now, maybe it's best to wait for a proper implementation instead of creating a temporary workaround :)

Let us know if we can help !
2022-09-30T15:43:06Z
https://github.com/huggingface/datasets/issues/5045#issuecomment-1263733695
https://api.github.com/repos/huggingface/datasets/issues/5045
IC_kwDODunzps5LUwu_
https://api.github.com/repos/huggingface/datasets/issues/comments/1263733695/reactions
2022-09-30T15:44:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263733695
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Ideally we would want to upload the files iteratively - and then once everything is uploaded we proceed to commit. 

Oh I see. So maybe this has to be done in an implementation specific to `datasets/` as it is not a very common case (upload a bunch of files on the fly).

You can maybe have a look at how `huggingface_hub` is implemented for LFS files (arrow shards are LFS anyway, right?).
In [`upload_lfs_files`](https://github.com/huggingface/huggingface_hub/blob/e28646c977fc9304a4c3576ce61ff07f9778950b/src/huggingface_hub/_commit_api.py#L164) LFS files are uploaded 1 by 1 (multithreaded) and then [the commit is pushed](https://github.com/huggingface/huggingface_hub/blob/e28646c977fc9304a4c3576ce61ff07f9778950b/src/huggingface_hub/hf_api.py#L1926) to the Hub once all files have been uploaded. This is pretty much what you need, right ?

I can help you if you have questions how to do it in `datasets`. If that makes sense we could then move the implementation from `datasets` to `huggingface_hub` once it's mature. Next week I'm on holidays but feel free to start without my input.

(also cc @coyotte508 and @SBrandeis who implemented LFS upload in `hfh`)
2022-09-30T15:53:16Z
https://github.com/huggingface/datasets/issues/5045#issuecomment-1263743802
https://api.github.com/repos/huggingface/datasets/issues/5045
IC_kwDODunzps5LUzM6
https://api.github.com/repos/huggingface/datasets/issues/comments/1263743802/reactions
2022-09-30T15:53:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263743802
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
NONE
> Could you share the error you got please ? Maybe the full stack trace if you have it ?

Here’s part of the stack trace, that I can reproduce at the moment from a photo I took (potential typos from OCR):
```
ValueError
Traceback (most recent call last)
<ipython-input-4-274613b7d3f5> in <module>
from datasets import load dataset
ds = load_dataset('jrahn/chessv6', use_auth_token-True)

/us/local/1ib/python3.7/dist-packages/datasets/table.py in cast_table _to_schema (table, schema)
Line 2005 raise ValueError()

ValueError: Couldn't cast 
fen: string 
move: string 
res: string 
eco: string 
move_id: int64
res_num: int64 to
{ 'fen': Value(dtype='string', id=None), 
'move': Value(dtype=' string', id=None),
'res': Value(dtype='string', id=None),
'eco': Value(dtype='string', id=None), 
'hc': Value(dtype='string', id=None), 
'move_ id': Value(dtype='int64', id=None),
'res_num': Value(dtype= 'int64' , id=None) }
because column names don't match 
```

The column 'hc' was removed before the interrupted push_to_hub(). It appears in the column list in curly brackets but not in the column list above.

Let me know, if I can be of any help.
2022-09-30T16:49:21Z
https://github.com/huggingface/datasets/issues/5045#issuecomment-1263798770
https://api.github.com/repos/huggingface/datasets/issues/5045
IC_kwDODunzps5LVAny
https://api.github.com/repos/huggingface/datasets/issues/comments/1263798770/reactions
2022-09-30T16:49:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263798770
https://avatars.githubusercontent.com/u/13120204?v=4
https://api.github.com/users/jorahn/events{/privacy}
https://api.github.com/users/jorahn/followers
https://api.github.com/users/jorahn/following{/other_user}
https://api.github.com/users/jorahn/gists{/gist_id}
https://github.com/jorahn
jorahn
MDQ6VXNlcjEzMTIwMjA0
https://api.github.com/users/jorahn/orgs
https://api.github.com/users/jorahn/received_events
https://api.github.com/users/jorahn/repos
https://api.github.com/users/jorahn/starred{/owner}{/repo}
https://api.github.com/users/jorahn/subscriptions
User
https://api.github.com/users/jorahn
public
**Is your feature request related to a problem? Please describe.**
I pushed a modification of a large dataset (remove a column) to the hub. The push was interrupted after some files were committed to the repo. This left the dataset to raise an error on load_dataset() (ValueError couldn’t cast … because column names don’t match). Only by specifying the previous (complete) commit as revision=commit_hash in load_data(), I was able to repair this and after a successful, complete push, the dataset loads without error again.

**Describe the solution you'd like**
Would it make sense to detect an incomplete push_to_hub() and automatically revert to the previous commit/revision?

**Describe alternatives you've considered**
Leave everything as is, the revision parameter in load_dataset() allows to manually fix this problem.

**Additional context**
Provide useful defaults
